Few-Shot Housing Element Detection with Vision Transformers: Improving Energy Retrofit-To-Address Matching

Abstract: Traditional computer vision struggles with domain-specific few-shot building element detection in street-view imagery, creating a bottleneck for extracting the building information needed for data-driven retrofit matching. This paper proposes a few-shot framework that combines a pre-trained Vision Transformer with a Random Forest patch classifier and adaptive cropping to address domain shift and limited annotations. With only 21 labeled images per element, the method achieves mean detection accuracy greater than 0.95 across seven elements and enriches building stock data at address level, enabling more accurate matching of retrofit measures.