SignSense

1. Project Overview

SignSense is a self-directed project exploring on-device machine learning for accessibility. It reads American Sign Language from a live webcam feed and turns it into text in real time, and it also lets you draw in mid-air with hand gestures — all processed locally in the browser, with nothing sent to a server.

2. Problem and Objectives

Most sign-language recognition tools depend on cloud services, which introduces latency, running costs, and a privacy trade-off: your camera feed has to leave your machine. I wanted to see how far I could push a fully client-side approach — fast enough to feel real-time, private by design, and with no backend to pay for or maintain.

3. My Role and Contributions

I built SignSense solo, end to end: the landmark-extraction pipeline, the machine-learning models and their training, the real-time inference loop, and the React front end with the air-drawing canvas.

4. Technology Stack

React and the Canvas API provide the interface, MediaPipe extracts hand and pose landmarks, and TensorFlow / Keras models handle static fingerspelling and GRU-based motion sequences. The inference pipeline runs on the user's device.

5. Key Features

• Fingerspelling recognition: a neural network reads ASL alphabet handshapes and transcribes them into text
• Whole-word sign recognition: a GRU sequence model recognizes complete signs from body and hand motion over time
• Air-drawing: hand gestures let you draw in mid-air on a canvas, no mouse or touch needed
• Fully on-device pipeline: MediaPipe extracts hand and pose landmarks and the models run in the browser, so the webcam feed never leaves the machine

6. Technical Implementation

• MediaPipe: fast, reliable hand and pose landmark extraction directly in the browser, the foundation for both recognition and drawing
• TensorFlow / Keras + GRU: a frame-based classifier for static fingerspelling and a recurrent GRU model for motion-based whole-word signs
• On-device inference: keeps the camera feed private, removes server latency, and means zero hosting cost for the ML workload
• React + Canvas API: a responsive interface and a performant surface for the real-time air-drawing feature

7. Challenges and Solutions

Real-time recognition needed to avoid the latency and privacy cost of uploading camera frames. I used MediaPipe landmarks as compact model inputs and kept inference in the browser, allowing both the recognition and air-drawing interactions to work without sending the webcam feed to a backend.

8. Testing and Results

The live application is available for hands-on review. The current portfolio material does not document a formal accuracy study or performance benchmark, so no unsupported metric is presented.

9. What I Learned

This project developed my understanding of browser-based machine-learning pipelines, time-sequence modeling, camera interaction, and the trade-offs involved in keeping computer-vision features private and responsive on the client.