Robot Arm Pen Grab Challenge

Use the RealSense to measure the 3D location of my purple pen. I use the interbotix_xs_toolbox to control the robot. Finally, get the robot capturing the pen. The physical setup for this project requires the Trossen PincherX 100 and the Intel RealSense D435i. The field of view of the RealSense should substantially overlap with part of the PincherX's workspace, and I decided to offset the D435i 90 degrees from the front of the PincherX.

The approach is to use classical computer vision techniques on the RGB image to locate the pen in 2D space. Then align the Depth map to the RGB image and use the pen location as a mask to get the 3D information. Finally, draw the contour of the pen and find the centroid of the pen. This information will be fed into a controller that will enable the robot to grab the pen.

I will use the interbotix_xs_toolbox to control the robot. There are a total of four steps:

• Measure the pen location.
• Move forward until the pen is inside the grippers.
• Close the gripper.
• Return to home position.

Source Code