VLM-Augmented RAG Framework with Ontology-Constrained LLM Reasoning for Existing Building Material Documentation

Abstract: Documenting information about materials in existing buildings is essential for circular construction and the reuse of elements. While multimodal Vision-Language Models (VLMs) can classify materials from images, they often lack the contextual semantic depth required for technical documentation. This study introduces a VLM-augmented Retrieval-Augmented Generation (RAG) framework that utilises ontology-constrained LLM reasoning to bridge this gap. By integrating domain-specific text retrieval and formal ontologies, the method enables the identification of building element layers and material types. The method is validated using a reference dataset of Dutch residential buildings and demonstrated by enriching semantic graphs with predicted material data.