Elias was born on 23 May 1617 in Breadmarket Street only a few metres away from the birthplace of Samuel Johnson.

His mother, Anne, was the daughter of a wealthy Coventry draper, Anthony Bowyer. His father, Simon Ashmole (1589–1634), was a saddler, who had served as a soldier in Ireland and Europe. Elias Ashmole attended Lichfield Grammar School, which now houses the chamber of Lichfield District Council, and became a chorister at the Cathedral in the city.

Elias left Lichfield in 1633 to live and study in London but returned many times during his life and tried unsuccessfully to become an MP for the area.  A sandstone statue immortalizing Elias Ashmole can be seen on the south-east corner of Lichfield Cathedral.