MediaPipe and OpenCV Application for Manipulator Robot Control in Object Retrieval Using Finger-Count Gestures

Abstract

The development of robotics and computer vision technologies has opened up opportunities for developing user-friendly Human-Robot Interaction (HRI) systems. This research aims to design and implement a control system for a 4-degree-of-freedom (4-DOF) manipulator robot using finger-count gestures based on MediaPipe and OpenCV. The system utilizes a webcam to capture real-time hand images, which are then processed by MediaPipe Hands to detect 21 hand landmark points to calculate the number of extended fingers. The detected number of fingers is translated into robot control commands, including moving toward the target, picking up objects, moving objects, releasing objects, and returning to the initial position. The manipulator robot is controlled by an Arduino Uno, which regulates the movement of the servo motors via serial communication. The testing results show that the finger-count gesture detection system achieved a success rate of 88% out of 50 trials. Meanwhile, the robot testing for picking up and moving objects obtained a success rate of 80% out of 10 trials. The results of this study demonstrate that the integration of MediaPipe and OpenCV can be effectively used as a real-time manual control medium for manipulator robots based on finger-count gestures.