A Mixed Reality Human Robot Interaction Interface Using Hand Tracking and Stereo Vision for Construction Applications

Abstract: The modern day construction site is able to complete complex tasks using highly skilled on-site workers and robust machinery. On-site workers face severe physical injuries from unforeseen accidents due to construction site dynamics. This project proposes and implements a mixed reality human robot interaction system, utilizing stereo vision systems and hand tracking algorithms, to teleoperate a robot arm within a simulation. Results demonstrate enhanced depth and situational awareness in the robot workspace versus traditional teleoperation. This research allows operators to safely perform complex construction tasks remotely with a robot arm using a head-mounted display.