DeepSign

Video Demo

About this project

DeepSign is a real-time communication bridge built for the Deaf and mute community. The idea came from a simple observation sign language interpreters cost over a hundred dollars an hour and aren't always available. A Deaf person sitting in a meeting, visiting a doctor, or just having a conversation with a stranger is constantly dependent on someone else being there to translate. We wanted to remove that dependency entirely. The app runs in two directions seemlessly, the first direction is voice to sign. When someone nearby speaks, DeepSign picks up their audio through the microphone, sends it to Deepgram for real-time transcription, then a signing stick figure appears on screen performing each word in ASL overlaid on a live camera feed like an AR display. It looks like looking through smart glasses. The second direction is sign to voice. This is the reverse for someone who is mute and communicates through signing. The camera watches their hands, MediaPipe extracts the skeletal landmarks of each hand pose in real time, and a machine learning model trained on the WLASL dataset identifies which ASL word is being signed. Those words are buffered into a sentence, the Claude API converts the raw sign sequence into natural grammatical English, and ElevenLabs speaks it aloud through the speaker. The mute person signs and the hearing person hears a natural voice.

Gallery