DESIGN OF OBJECT SPELLING RECOGNITION SOCIAL ASSISTIVE ROBOT
FOR DEAF PEOPLE USING GESTURE RECOGNITION
Thesis project to Build a robot that can check sign language spelling using CNN
Introduction
Education is a right for all people in Indonesia has also become an obligation of the government of its citizens to educate the nation’s life. Along with the rapid development of the times many people are looking for effective and efficient learning methods and are not focused on teaching and learning hours by combining learning methods with technological advances. Learning methods that are commonly applied at this time are done by reading. In this study, the authors conducted a study by designing social robot assistants to assist in the learning of letter recognition and spelling for deaf children where these abilities are important provisions in reading skills. Broadly speaking, this study utilizes the concept of gestures recognition using machine learning as the main function. The robot interacts with the user through the user’s hand movements and the robot movement itself. The results of the research that have been done are the creation of a social assistant robot for the learning process to recognize letters and how to spell 4 alphabetical order, 5 alphabetical order, and 6 alphabetical order with 100% detection accuracy and utilize finite state machine methodology
Scope of Problem
- The spelling input still comes from the keyboard so that in every spelling
have to type in any letters that will be spelled by the user - Gesture Recognition is only performed on the wrist and finger scale
hands, can’t completely one body - The method used is for gesture recognition only
using the Convolutional Neural Network (CNN) method.
How The System Works
This system is activated via keyboard input entered by the user, so the robot can tell when he started his job, when the input is wrong then the robot will remain in an idle state until the keyboard input is correctly entered. When the system already active, the first step that must be taken is provides input from the keyboard in the form of alphabets of the object to be spelled, after input entered then the robot will change to move and the movement indicates the system is ready to process the spelling of the user, the spelling performed will be checked one by one. When there is a spelling error, the robot will immediately do it a gesture indicating the student’s misspelling and will retest with a maximum of three times, when If the error has passed the maximum limit, the robot will stay at idle condition with the aim of the teacher concerned can explain the material again regarding the introduction of the alphabet. If there is no mistake then the robot will do gesture indicating correct spelling is up to the object successfully spelled completely into a word,
The Data
the data that i use are photo of my hand while doing the alphabet movement from A-Z. Each alphabet had 1200 photo then i flip it, so eazh alphabet had 2400 photo.Image collected is converted to numpy arrays to make this suitable for tensor processing in training.
To make a good data set like the image above, it is recommended to use gloves that have a contrast color with background.
Training Process
All the classes has been trained using CNN method with the architecture as shown below:
after the dataset being processed with those architecture, the matrix is converted into a linear array so that to input it into the nodes of my neural network. After that those linear outputs are converted to number of output classes.
Next step is training the model:
after the training process is done , then the model will save the alphabet movements that we have made in the dataset.
here is my alphabet movement which will be used in the testing process.
Testing Process
the testing process is the process where we try the models we have created to recognize sign language movements if there are defects or inconsistencies in the recognition of sign language movements, there is something wrong when we do the training process.
Make the Spelling mode
spelling mode is created using the finite state machine method, therefore the spelling mode in this system can only perform spell check with variations in the same number of alphabets. On this occasion I made 3 variations on the spelling mode, that is variations with 4 alphabetical order, 5 alphabetical order and 6 alphabetical order. the finite state machine of 3 various mode :
To clarify the conditions START condition is a condition where the system has not been entered on spelling mode, IDLE condition is a condition when keyboard input has been pressed to enter spelling mode, but alphabet input has not entered and the robot condition is in a condition that represents that the robot is ready to check the spelling of the user. Next conditions T1 to T6
is a spelling condition in alphabetical order on the object, for example if the user give alphabetical input with object B-U-K-U, then T1 is the process the user must spell the letter B, while the user process T2 must spell the letter U and so on, whereas the finished process is the process by which the user spells successfully the whole alphabet that is inputted to the system and the condition of the robot will do clapping movement while saying “you, ve been finished your spell”.
Robot Prototype
the robot that i used is a humanoid robot by Robotis with type Robotis Bioloid Premium . The robot is built by integrating 3 systems, namely a computer system where the computer system on this robot uses the INTEL NUC mini pc, webcam, and the humanoid robot itself. The power source that used to turn on this system is power adapter, there are 2 power adapters used to power this system wich is 12 volt power adapter for the CM-530 to turn on the humanoid robot and 19 volt power adapter to power up the INTEL NUC. Communication between mini PC with CM-530 uses a USB to TTL cable, because of its communication used between the 2 components is serial communication. Communication between 2 This component is done with the aim that the mini pc can send commands movements on the CM-530 correspond to the workflow of the system being built.
Robot Movement
Robot movement in this system is done by sending commands movement in the form of hexa data written in python programming to CM-530. sending data is done serially and will be received by CM-530.Then will be the command to move the servo on the robot. Basically hexa data sent is replacement data from remote controller is the innate robot controller from Robotis, but its communication is originally radio in the system this time changed to serial with the aim that the robot can move automatically without having to be controlled by the remote.
NOTE !!
mostly, the programming scripts from hand gesture recognition i got on this YouTube videos and github which has been uploaded Evilport in his youtube channel Programming by EvilPort. but I’m not completely plagiarize, i did some modification for example in epoch, Batchsize, and learning rate parameter, and also for the dataset i made it myself. and for the spelling mode and robot motion was made by me.
Hope you guys enjoy it . Feel-free to ask me at ega.guntara394@gmail.com :)