As conquerors occupied the island of Rhodes, they imposed their architectural and religious preferences on the buildings. Originally constructed in the late 11th century in the area of the medieval city called Kollakion, the majestic church called Our Lady of the Castle (Panagia tou Kastrou) (Virgin of the Fort) has had many reincarnations. It was first built with a dome in the Byzantine "cross-in-square" style of architecture.
When the Knights of the Order of St. John occupied the island in 1309, one of their Grand Masters, Hélion de Villeneuve (1319-1346) along with Pope John XXII (whose pontificate lasted from 1316 to 1334), converted the church into a Gothic three-aisled basilica with a transept. The Byzantine dome was replaced by a barrel-vault and ribbed cross-vaults. It became the Catholic cathedral of Rhodes. Inscriptions on the coats of arms of the Grand Master and the Pope are embedded on one of the walls. Historians have found references to it in a Papal bull (Pope's written public decree) of 1322.
In 1522, Rhodes fell to the Ottoman Turks. This resulted in further changes to the church. The Turks whitewashed the interior walls, destroying the wall paintings, and added a minaret and a mihrab (recess for prayer). The building then functioned as a mosque, called the Ederum or Kadouri Tzami. The additions of the Ottoman era were carefully removed during the Italian occupation (1912-1947). The building has been subsequently conserved by the Greek Archaeological Service.
In 1826, the Belgian traveler Rottiers wrote that in the church were many tombstones of Catholic residents of the city of Rhodes and foreigners who died there, and that the sanctuary had stained-glass windows, which he copied down before they were destroyed.
Today, the decoration of the interior is no longer preserved, with the exception of some fragments of the wall-paintings, representing the Holy Mother of God and four other saints. The best preserved among them is the figure of St. Lucia dating to the 14th century. Frescoes combine both Byzantine and Western European elements.
Since 1988 the church of Panagia has housed an exhibition of Byzantine and post-Byzantine painting, including portable icons of the 17th and 18th centuries. Also exhibited are wall-paintings taken from the church of St. Zacharias on the island of Chalki (14th century) and from the monastery of the Archangel Michael at Tharri of Rhodes (17th century). The collection includes various architectural pieces and some Early Christian mosaics.