In this paper we report on our experience in creating photo-realistic virtual environments of frescoed surfaces to allow scientists, conservationists, historians, and tourists to closely study important frescoes at high level of details. In addition to allowing unlimited virtual access, this will also protect the frescoes from the damage that can be caused by opening the site to a large number of visitors. We address the main challenges associated with photo-realism or texture quality, and real-time interactive visualization and discuss available solutions. Where no effective solutions existed, we had to develop our own techniques. As case studies, we present results of modeling and visualization of two sites, both located at Buonconsiglio castle, one of the most important castles in Trentino in northern Italy. The first is the frescoed walls “Cycle of the Months”, a major masterpieces of international Gothic art that was painted around 1400. The second is the Gothic-Venetian Romanino loggia, named after the artist Girolamo Romanino (1484-1562) who decorated the ceiling and walls of the loggia, as well as the adjacent corridors, with remarkable frescoes in 1531.