Enhancing Visual-LLM through Prompt Engineering and Hybrid Retrieval-augmented Generation for Site Safety Compliance Checking

Abstract: The increasing prevalence of safety incidents on construction sites states the urgent need for enhanced monitoring. This study proposes an innovative hybrid Retrieval-Augmented Generation (RAG) algorithm to compliance check accuracy for site images. By integrating the Visual Language Model (VLM), we developed an algorithm capable of mastering domain knowledge without fine-tuning and addressing the limitation of interpreting RAG technology with visual information. A three-phased prompting framework was designed to enhance the VLM’s compliance analysis abilities. Experiments based on actual construction site in Hong Kong demonstrated 21.89% increase in retrieval accuracy.