Although some archaeological remains suggest that the building dates back to Roman times, the appearance of the Torreón de Lozoya took shape between the 15th and 16th centuries, during two distinct periods that correspond stylistically to the late Gothic—with its Mudejar influences—and the Renaissance.

The medieval construction was a fortified house, most likely belonging to the Cuéllar family, featuring the large tower that gives the building its name and serves as its entrance, as well as a smaller tower; the original courtyard would have been situated between the two. All of this must have formed a veritable miniature castle, reflecting the turbulent times Segovia experienced during the Late Middle Ages.

The second decisive phase in the history of this monument began in 1563, with the acquisition of the property by D. Francisco de Eraso, Chamber Secretary and Councillor of State to Charles I and Councillor of the Treasury and the Indies to Philip II, who transformed the medieval mansion into an impressive Renaissance palace, the most significant remains of which are concentrated in the courtyard and the gallery built to overlook the garden.

The building currently houses the exhibition rooms and museum of the Torreón de Lozoya Foundation.