As is often the case with large properties in this part of Tuscany, the building that now forms Ca' di Pesa developed and changed throughout the centuries. Records dating back to the XI century show that it was a part of the ancient Roffiano castle, a military structure tied to the monks of Passignano. At the end of the XII century, Roffiano was the object of an armed dispute between Passignano and the Bishop of Fiesole. In 1261 it was conquered and occupied by the Ghibellines, who destroyed a portion of it - including the top of its tower, and a house.

It soon became evident that, with the resolution of military conflict, it would be more profitable to convert old military structures into working farms which operated under a feudal system and, from the XIV century onwards, under a form of sharecropping which lasted almost until the present day, marking centuries of constant agricutural production serving the city of Florence and beyond.

Ca' di Pesa's fate is very well documented from the XV century onwards, showing its ownership by the hospital of Santa Maria Nuova in Florence, with records of lease agreements pointing to the farm's rental by Antonmaria Gherardini - whose daughter Lisa was made famous by the painter Leonardo da Vinci!