NAO_Gesture_Generation

This repository contains the system used for rendering gestures on the NAO robot. The gestures are generated by a deep learning-based generative model using the Diffusion model.

Installation & Preparation

1. Git clone the project.
2. Install the qi SDK https://support.unitedrobotics.group/en/support/solutions/articles/80001018812-nao-6-downloads and python 2.7 on your laptop.
3. Obtain a V6 NAO robot and ensure that both your NAO robot and laptop are connected to the same network.

How to run the code

1. The main.py is the entry point of the system.
   1. The Elbow.py, Hand.py, Head.py, Shoulder.py, and Wrist.py are utilized to post-process rotation angles based on directional vectors generated by the generative models, ensuring they conform to the joint constraints of the NAO robot.
2. Please modify the IP address on line 171 of main.py according to the network to which the NAO is connected.
3. In main.py, you can modify the Python variable file_name on line 180 to access the generated audio and pkl files for testing from the generation_results folder. Note that the testing samples typically range from 5 seconds to 12 seconds in length. Additionally, the Long_sample subfolder contains longer samples compared to other testing files.
   1. The audio files should be uploaded to the NAO robot, you can use FileZilla do this thing.
      1. Use the SFTP protocol
      2. Host Number should be the IP address of the NAO robot
      3. By default, user name: nao, password: nao
4. The utils.py file contains various functions used for smoothing angle lists and performing normalizations, among other tasks.
5. Finally, you can compare the *.mp4 files with the robot rendering results, enjoy!

Postprocessing Table

joint name	Motion	Min Range (deg)	Max Range (deg)	post-processing (deg)
HeadYaw	Head joint twist (Z)	-119.5	119.5	scale to [-15, 15]
HeadPitch	Head joint front and back (Y)	-38.5	29.5	scale to [-15, 0]
LShoulderPitch	Left shoulder joint front and back (Y)	-119.5	119.5	x
LShoulderRoll	Left shoulder joint right and left (Z)	-18.0	76.0	x
LElbowYaw	Left shoulder joint twist (X)	-119.5	119.5	x - 45
LElbowRoll	Left elbow joint (Z)	-88.5	-2.0	- abs(x) - 45
LWristYaw	Left wrist joint (X)	-104.5	104.5	- x
LHand	Left hand	Open and Close	Open and Close	normalization
RShoulderPitch	Right shoulder joint front and back (Y)	-119.5	119.5	180 - x
RShoulderRoll	Right shoulder joint right and left (Z)	-76.0	18.0	x
RElbowYaw	Right shoulder joint twist (X)	-119.5	119.5	x + 135
RElbowRoll	Right elbow joint (Z)	2.0	88.5	abs(x)
RWristYaw	Right wrist joint (X)	-104.5	104.5	abs(x)
RHand	Right hand	Open and Close	Open and Close	normalization

Dirctional_Vec_Info in the .pkl file:

dir_vec_pairs = [
    (0, 1, 0.26), # 0, spine-neck
    (1, 2, 0.22), # 1, neck-left shoulder
    (1, 3, 0.22), # 2, neck-right shoulder
    (2, 4, 0.36), # 3, left shoulder-elbow
    (4, 6, 0.33), # 4, left elbow-wrist
    (6, 8, 0.137), # 5 wrist-left index 1
    (8, 9, 0.044), # 6
    (9, 10, 0.031), # 7

    (6, 11, 0.144), # 8 wrist-left middle 1
    (11, 12, 0.042), # 9
    (12, 13, 0.033), # 10

    (6, 14, 0.127), # 11 wrist-left pinky 1
    (14, 15, 0.027), # 12
    (15, 16, 0.026), # 13

    (6, 17, 0.134), # 14 wrist-left ring 1
    (17, 18, 0.039), # 15
    (18, 19, 0.033), # 16

    (6, 20, 0.068), # 17 wrist-left thumb 1
    (20, 21, 0.042), # 18
    (21, 22, 0.036), # 19

    (3, 5, 0.36), # 20, right shoulder-elbow
    (5, 7, 0.33), # 21, right elbow-wrist

    (7, 23, 0.137), # 22 wrist-right index 1
    (23, 24, 0.044), # 23
    (24, 25, 0.031), # 24

    (7, 26, 0.144), # 25 wrist-right middle 1
    (26, 27, 0.042), # 26
    (27, 28, 0.033), # 27

    (7, 29, 0.127), # 28 wrist-right pinky 1
    (29, 30, 0.027), # 29
    (30, 31, 0.026), # 30

    (7, 32, 0.134), # 31 wrist-right ring 1
    (32, 33, 0.039), # 32
    (33, 34, 0.033), # 33

    (7, 35, 0.068), # 34 wrist-right thumb 1
    (35, 36, 0.042), # 35
    (36, 37, 0.036), # 36

    (1, 38, 0.18), # 37, neck-nose
    (38, 39, 0.14), # 38, nose-right eye
    (38, 40, 0.14), # 39, nose-left eye
    (39, 41, 0.15), # 40, right eye-right ear
    (40, 42, 0.15), # 41, left eye-left ear
]