The Duke of Arcos, Antonio Ponce de León Spínola, one of the gentlemen closest to Carlos III, ordered this building to be built. The design and direction of the building work was by the architect Manuel Machaca Vargas.
It was commissioned by the Duke of Arcos to lodge his family and servants during the Royal Visits. His daughter, the 13th Duchess of Alba, immortalized by Goya in his paintings, became the proprietor in 1784.
Its architectural design coincides with the style that Juan de Villanueva set down in the Royal Estate, of two floors and an attic, and it was one of the few 18th century buildings that did not have a third floor added.
With the discontinuance of the Royal Visits, the house became a permanent residence, being divided into three parts.