SignNinja

Video Demo

About this project

For almost every spoken language, there are comprehensive tools available for people to learn to speak the language. However, these tools are not easily available for American sign language. This is a major problem for people like me, as my father is going deaf, and my family needs an easy way to learn ASL. Drawing inspiration from childhood games such as TypeRacer, our project is a game that prompts letters to the user, who must sign the letter back to the camera in order to proceed. The player starts with 3 lives, after which the game ends. If a player does not make the correct sign in 10 seconds, a life is lost. This site is designed to build muscle memory in ASL for the user, while being a fun alternative to traditional learning methods. We used the MediaPipe Holistic open source library to read and interpret 3d poses for the structure of the human body, face, and hands. While there are databases of pictures of ASL readily available online, we noticed that any program attempting to interpret ASL through these images would come up short, as ASL is a heavily movement based language. In order to circumvent this, we produced a neural network capable of interpreting common ASL letters and words by analyzing videos created by us, as individual videos can be spliced and analyzed in order to interpret a movement. To give a brief example, the words “good” and “bad” rely on the same arm and hand shape, but with a reversed movement. We also found that even with an extremely limited vocabulary, such as the alphabet, pictures could not alone suffice for letters such as “j” and “z”. In the future, we plan to expand the available in-game dictionary to include words, and eventually sentences. While we had a program in place to interpret signed ASL words, translate to any spoken language, and speak it out loud, we ultimately decided that using our program as a learning assistant rather than a translator tool was more aligned with our vision.

Gallery