Real-time upper body motion tracking using computer vision for improved human-robot interaction and teleoperation

Abstract

Upper body motion tracking mapping is crucial for robot control because it gives the machine a better understanding of how a human operator moves, allowing it to react instinctively and naturally. Most current research has focused on using wearable sensors and remote controls to enhance communication between robots and humans. However, this research aims to address the issue by embracing a nonwearable sensor-based strategy to promote more natural and spontaneous interactions between humans and robots. Moreover, A 3-DoF manipulator was also designed and developed utilizing robotics technologies. The vision system captured a human operator's upper body movements in realtime video footage. Computer vision approaches were used to extract positional and orientation information from the upper body in this setting. The system combines the MediaPipe pose model with kinematics theories to estimate the hands' position and movement in real-time. According to the experiment results, the system's overall accuracy is 94.1 (±1.2) %, and the motion tracking system's accuracy is 96.5 (±2.0) %.