1. General
The NAO robot is 58cm tall and weighs 5.4kg. the main hardware includes CPU, motherboard, speaker, microphone, infrared, camera, ultrasonic (sonar), sensors, motors, speech synthesizer, gyroscope, and so on. The model of NAO robot in the lab is NAO
H25, which is constructed as follows:
The operating system of NAO Robotics is Gentoo Linux, which supports remote control of Windows, Linux, Mac OS and other operating systems, and can be programmed to control NAO on these platforms. the head CPU runs the Linux kernel and supports the Aldebaran Robotics kernel NAOqi, which provides a set of application Programming Interface (API) for operating the robot, such as controlling robot motion, shooting, sound recognition, reading sensor values, etc. These APIs can be called using languages such as C++, Python, .Net, Java, Matlab, and others.
This project uses a NAO robot to recognize a ball and pick it up, then throw it to a specified location. It mainly involves image acquisition, filtering, target object localization and motion control of NAO robot.
The whole process is roughly as follows: obtain the image from the robot's head camera, perform a series of processing on the image, and then find the coordinates of the ball in the image according to the Hough circle detection function. In this way, we cycle through the ball coordinates, first moving the robot left and right so that the ball is in the middle of the image to prevent it from going out of the range of the camera, and then moving the robot back and forth so that the ball is at the bottom of the image so that it can be switched to the mouth camera to detect the ball next. Next, the robot moves forward a fixed distance and then switches to the mouth camera for proximity detection, similar to the head camera detection, cyclically detecting and adjusting the coordinates of the ball to a predetermined pixel range, then the recorded ball grabbing action can be called to pick up the ball. Finally, the robot detects the contour of the trash can, filters irrelevant contours and moves to the specified position according to the center of gravity and area of the contour, and then throws the ball into the trash can.
Issues to be addressed include:
(1) OpenCV: Image Specified Color Extraction and Binarization, Noise Filtering, Circle and Trash Can Contour Detection and Localization
(1) NAO Robot: Acquisition of camera images, recording of specified motion frames in Animation mode, motion control based on detected contour coordinates
2. Relevant technologies
The main techniques used in this project are motion control of NAO robot and OpenCV image processing.
2.1 NAO Robot
(1) Start-up and connection
NAO can be activated by pressing the button on the chest of the NAO robot, which can be connected to a computer via Ethernet or Wi-Fi. The first time you use the robot, connect it to your computer with an Ethernet cable and press the button on its chest after the robot has started, it will report the IP address of the robot, and then you can open the IP address in your browser to enter the NAO robot's setup interface. This interface allows you to set the robot's speaker volume, Wi-Fi connection, language, and other functions. With this IP address and fixed port number 9559, you can use C++, Python and other programming languages to connect to NAOqi via TCP/IP protocol, and call NAOqi's API to realize the operation of NAO robot.
In addition, NAO is actually a computer with enough computing power to provide remote login (Telnet) and file transfer (FTP) services with SSH (Secure Shell Protocol). With the remote login and file transfer services, NAO can be used as if it were a computer with a Linux operating system.
（2）NAOqi
Execution of NAOqi on NAO is done through a proxy program (Broker). When starting the robot, the agent program automatically loads the/etc/naoqi/file, which specifies which NAOqi libraries need to be loaded, and which library files are located in the/usr/lib/naoqidirectory. A library contains one or more modules, each of which defines multiple methods. For example, the motion functions of NAO are all placed in the ALMotion module, so that the robot to complete the movement, turn the head, open the hand, etc., respectively, to call the ALMotion module in themoveto()、setAngles()、OpenHand()and other methods.
When you use NAOqi module, you don't need to import all modules with import statement like normal Python program, the module notifies the methods it provides through Broker. Through the Broker, any module can find all the modules and methods that have been advertised.
The Broker consists of two main roles: direct service, which looks up modules and methods, and network access, which calls module methods from outside the Broker process.The Broker is both an executable program and a server that listens for remote commands to a specified IP and port. Programs with calls to NAOqi modules over IPs and ports can be run either on the Broker or directly on remote computers. Generally the Broker is transparent, and in most cases can be programmed without regard to the Broker, and the code that calls the local module is the same as the code that calls the remote module.
2.2 Choregraphe
Choregraphe is a programming environment provided by NAO that allows the creation of behavioral modules for NAO robots and their uploading to connected robots for testing.Choregraphe uses graphical programming and does not require the user to write a single line of code for the creation of complex behavioral modules.Choregraphe also provides user-defined features that allow custom modules to be written in Python. Choregraphe also provides user customization features, allowing custom modules to be written in the Python language. In this ball picking program the main use of the software is the timeline command box and Animation mode to record the actions that the user wants the robot to perform.
2.3 OpenCV
OpenCV is a cross-platform computer vision library, initiated and co-developed by Intel Corporation, licensed under the BSD license and freely available for commercial and research use.OpenCV can be used to develop real-time image processing, computer vision, and pattern recognition programs.OpenCV can be used to solve problems in the following areas: augmented reality, face recognition, gesture recognition, human-computer interaction, motion recognition, motion tracking, object recognition, image segmentation, robotics. OpenCV can be used to solve problems in the following areas: augmented reality, face recognition, gesture recognition, human-computer interaction, action recognition, motion tracking, object recognition, image segmentation, and robotics.
OpenCV is written in C++ and its main interface is also in C++, but still retains a large number of C interfaces. The library also has a large number of interfaces for Python, Java and MATLAB. In this project, we mainly use the OpenCV interface in Python.
3. General and detailed design
The system is divided into three main modules, namely: pose control, motion control, and OpenCV vision processing.
3.1 System Module Division
I. Postural control
Since the ball needs to be recognized based on the image from the camera and the different poses of the robot determine the visual range of the robot, the robot needs to be specified to maintain different poses in different situations.
II. Motion control
The robot needs to grab and throw the ball, and the whole action sequence by filling in the angle value of each joint in different frames is not only difficult to achieve the ideal effect, but also will take a lot of energy. So here we use Choregraphe's Animation mode to record the key frames of the action.
III. OpenCV Visual Processing and Robot Movement
The core work of grabbing and dropping the ball is to process the image acquired by the camera and control the movement of the robot. The image processing is mainly performed by denoising, blurring, color extraction, and contour detection of the image through the API provided by OpenCV, and the movement of the robot is mainly controlled by the pixel information of the contour acquired to control the robot to move to the specified position.
3.2 Main Functional Modules
I. Postural control
As shown in the figure below, the NAO robot uses the rotational sets roll, pitch, and yaw to represent the kinematic postures, which correspond to rotations in the direction around the X, Y, and Z axes, respectively. When describing the range of motion of a joint, clockwise rotation along the axis of rotation is negative and counterclockwise rotation is positive.
The name of each joint of NAO robot consists of part name + pose name, such as HeadYaw indicates the joints that rotate the robot's head left and right. After testing, the initialized motions in this project are ShoulderPitch is 80 degrees, ShoulderRoll is 20 degrees, HipPitch is -25 degrees, KneePitch is 40 degrees, AnklePitch is -20 degrees, and all other joints are 0 degrees. Write the above joints and angles into a list, and then call ALMotion'sangleInterpolationWithSpeedThe function adjusts the robot's posture.
II. Motion control
Open Choregraphe-right click to create command box-timeline, double click on the created timeline to edit the robot's movements, you can store the angle information of a joint or all joints of NAO body at different frame moments to complete the recording of movements. In normal mode, the stiffness of all joints of NAO robot is 1, and it can't move its joints. In Animation mode, if the color of the LED of the robot's eyes and feet is orange, it means the stiffness is 1. You can touch the color of the LED of its hands and feet to make the color of the LED change to green, so you can adjust the different joints of the robot at will, and then you can record the keyframe in the software after adjusting it.
The action between two keyframes can be simulated by a Bezier curve to generate a smooth action curve to ensure the normal execution of the robot action. After adjusting the motion, the corresponding C++/Python code can be generated.
III. OpenCV Visual Processing and Robot Movement
(1) NAO Camera Capture
The NAO head has two cameras for recognizing objects in the field of view, with the forehead camera used primarily for distant images and the mouth camera used primarily for images below. There are six optional resolutions for the cameras, up to 1280x960px, here set to 640x480px.
Get image: first connect to ALVideoDevice module, then call subscribe function to subscribe image, this function can set the resolution of the image, get the color space of the image and other parameters. Finally, callgetImageRemoteYou can get an array of images from the camera, where elements 0 and 1 are the width and height of the image and element 6 is the image data. After getting the image convert the image to numpy format and return it.
Compared with RGB, HSV color space will not change drastically with the change of light intensity, and the color value of the target object will not have a large deviation, which to a certain extent attenuates the influence of lighting conditions on the robot vision system and enhances the adaptive ability of the robot vision system. Therefore, after acquiring the image in RGB color space, it is usually converted into HSV color space for detection.
def getImage(IP, PORT, flag):
camProxy = ALProxy("ALVideoDevice", IP, PORT)
resolution = 2 # 640x480
colorSpace = 11 # RGB
# Head/mouth camera
(18, flag)
videoClient = ("python_client", resolution, colorSpace, 5)
naoImage = (videoClient)
(videoClient)
imageWidth = naoImage[0]
imageHeight = naoImage[1]
array = naoImage[6]
im = ("RGB", (imageWidth, imageHeight), array)
ci = (im) # Convert to numpy format image
r, g, b = (ci) # opencv's images are BGR so they need to be converted
ci = ([b, g, r])
return ci
(2) Ball detection
The red ball is mainly detected by Hough's circle transform as follows:
1. Acquire an image from the robot's camera and convert it to an image in HSV color space.
2, then according to the HSV color range table to extract the specified color range, the green ball, for example, the range of its HSV [35,43,46] to [77,255,255], call the inRange function to extract the green color space. However, the actual test results are not good, the surrounding environment will be extracted out of the irrelevant content, and the experiments used in the green ball is more vivid, the test will be the lower limit of the HSV S (saturation) up to 128 can be better extracted from the green color space.
3, Gaussian blurring of the image to remove noise from the picture, kernel takes a 7x7 matrix.
4. Carry out an open operation, i.e., corrosion followed by expansion to further eliminate noise.
5, observe the image at this time, in addition to the detected ball, there are many positions with intermittent white. So first take the inverse and then three closed operations (first expansion and then corrosion), kernel take 15x15 matrix.
6, the binary map obtained at this time is more ideal, the results of the image processing process is shown in the figure below:
7. Call HoughCircles for Hough circle detection with the following parameters:
(np.uint8(dilated),cv2.HOUGH_GRADIENT,1,100,param1=100,param2=7,minRadius=2, maxRadius=60)
Among them:
The first parameter is the input image
The second parameter detection method was chosen to be the Hough gradient method
The third parameter is the size of the resolution of the cumulative surface, which is set to 1 to indicate that the resolution is the same as that of the original image.
The fourth parameter is param1, the high threshold for Canny edge detection
The fifth parameter is param2, the smaller its value, the lower the limit of the detection circle is
The sixth parameter is the minimum circle radius to be detected
The seventh parameter is the maximum radius of the circle to be detected
8, if the filtering effect is good, it can detect the coordinates and radius of all the circles in the figure, where the assumption that there is only a green ball, so the final judgment if the results of the detection of the results of only one circle, then return to the center of the circle coordinates.
Above is the step of detecting the center of the ball circle, the specific call, 10 times in a row to obtain the camera image and detect the center of the ball circle, after sorting, if the middle element is (0,0) means that there is no detection of the ball, otherwise it will be returned.
(3) Grabbing the ball
As the NAO robot crouches down to grasp the ball on the ground when the posture is maintained for too long is prone to heat up resulting in the robot needing to dissipate heat before it can continue to be used, due to the fact that this requires a lot of time and experience, and the time for hands-on training is limited, so the ball is placed here on a 21cm high platform for the robot to grasp.
First turn on Animation mode in Choregraphe and record 3 keyframes:
(1) Open the palm of your hand to prevent your hand from touching the platform section
(2) Move the right hand of the robot to the height of a point above the ball
(3) Next move your right hand from right to left to ensure that the ball stays in the palm of your hand.
After generating the code for the above three keyframes and testing it in Python, you need to call ALMotion'sopenHandfunction opens the robot's hand, and after the above actions are complete then call thecloseHandGrab the ball.
(4) Ball positioning
Place the robot in a fixed position and then test that the above maneuvers can successfully grasp the ball. The relative position of the robot to the ball was recorded: the front foot was 9 cm from the platform, and the center of the cap circle was parallel to the right side of the knee of the right leg. Then call the previous ball detection function at this position, when the robot uses the mouth camera and the HipPitch and KneePitch of the hip and knee are -25 and +40, the coordinates of the ball are approximately at (523.5,426.5), since the robot is very close to the ball at this time, the X and Y coordinates are allowed to have a 25-pixel deviation, and it is tested that the ball can be grasped within the range of the deviation. Grab the ball.
The above is the parameterization of the mouth camera and at the beginning as the robot is at a distance from the ball, it needs to shoot the ball with the forehead camera. So at first the robot is kept in a fixed motion and then the ball is detected with the forehead camera and the (x,y) is adjusted according to the coordinates of the detected ball. First want the ball to be located to the very bottom of the camera, where the ball is detected in real time and the robot is allowed to move forward, increasing y to 430 or more. Next, we want the ball to be located at the very center of the camera, so we detect the ball in real time and adjust its x coordinate to between 290 and 350.
The robot was then allowed to advance until it could detect the green ball from the mouth camera. Based on the previous tests, it is sufficient to stabilize the ball coordinates at (523.5,426.5) and allow a 25px deviation. So the same steps as for the forehead camera, adjust the y-coordinate first and then the x-coordinate.
It is important to note that, due to the ALMotion module robot forward or backward function execution process will be affected by some objective conditions, the robot will often deviate to the left or right by a certain angle. So after the robot moves left and right or back and forth, determine the deviation of its x or y coordinate, such as adjusting the x coordinate, the deviation of the y coordinate exceeds a certain threshold, then the robot is considered to be moving away from the robot, so rotate the robot to the left or right by a certain angle to compensate.
Finally, you can just call the previously recorded ball grabbing action to grab the ball.
(5) Positioning of trash cans
Here it is desired to throw the grasped ball into the red trash can, again the image is acquired by the robot's camera, then mean offset filtering and median filtering are performed, and then the image is transferred to HSV space to extract the HSV range of the red color. Then the result is binarized, and then the findContour function in OpenCV is called to detect the contour. Here, according to the test of the distance between the trash can and the robot and other parameters, we can get the range of the area of the trash can within a certain distance, and according to this range, a contour is selected, and then the coordinates of the center of gravity of the contour can be calculated by the result of the contour.
The above function for detecting the center of gravity of the trash can is encapsulated into a function whose return value is the coordinates of the center of gravity of the contour. Here the trash can is placed on the left side of the robot. After the robot grabs the ball, it first steps back about 25cm, this is done to prevent the robot from being blocked by the platform when it rotates. Then the robot is rotated by 60° to ensure that the trash can is within the range of the robot's camera. The subsequent steps are similar to those for detecting the ball, first adjusting the Y coordinate of the robot to about 340, then adjusting the X coordinate in the range of [300,335], and finally the robot advances a certain distance to walk in front of the trash can.
(6) Throwing the ball
Again the movements were recorded through Choregraphe's Animation mode, simply recording the robot's hand movements for a total of two frames: opening the right arm and moving the right arm to the center. Since the diameter of the trash can is much larger compared to the ball, the range of deviations allowed is much larger and the success rate is higher.
Generate Python code that performs the above actions without error and calls the ALMotion module'sopenHandPut the ball in the trash.
4 Video Presentation
NAO robot recognizes the ball and grabs it to throw it in the trash.
The slow recognition is the result of a bad servo drive module inside this NAO robot, which calls its forward function to go crooked. I then slowed down the stride and recognized it slowly, along with a function to adjust for left and right crookedness. Replace it with an unbroken NAO robot and you won't have this problem.
To this article on the OpenCV NAO robot assisted pick up the ball to throw the ball of the article is introduced to this, more related OpenCV robot to pick up the ball to throw the ball content, please search for my previous articles or continue to browse the following related articles I hope that you will support me in the future more!