Abstract:Aiming at the low precision and efficiency of the fruit identification and harvesting motion of existing harvesting robots, the hand-eye coordination planning with deep visual servo for harvesting robot was carried out. A small lifting harvesting robot with deep visual servo of RealSense-in-hand was developed, which was composed of the autonomous vehicle, lifting bin, electric fork lifter, grip-cut integrated end-effector, and 3-freedom manipulator. The workspace and posture analysis of fruit picking and placing was performed, and the coordinate transformation model of hand-eye coordination was established for the eye-in-hand mode. Based on the depth visual servo of RealSense-in-hand, the far-to-close hand-eye coordination strategy was proposed for the harvesting robot. The canopy detection from a distance, sub-region division and location, close-range fruit accurate identification and positioning were effectively combined, so that the step-by-step visual guidance of the manipulator was realized. According to the parameters of RealSense and the manipulator, the segmented motion planning between far-to-close key points based on depth vision was completed. It was shown in the hand-eye coordinated harvesting test that, the average positioning accuracy of the end-effector in the X, Y and Z directions was 3.51mm, 2.79mm and 3.35mm, respectively. The average time consuming was 19.24s, which included the motion time of the manipulator from the initial position to picking position (12.04s), the fruit recognition and computing time (3.82s), and the fruit placing time (7.2s). The time of manipulator motion accounted for 80.2% of the whole cycle. Both the robot structure and the hand-eye coordination strategy based on depth visual servo of RealSense-in-hand can meet the needs of fruit harvesting operation.