Silent Bridge: Real-Time Bidirectional Indian Sign Language and Kannada Translation with 3D Avatar

Uncategorized

Authors: H S Thanush, Kiran M K, Manoj G U, Tejaswini K

Abstract: Deaf and hard-of-hearing citizens in Karnataka communicate primarily through Indian Sign Language (ISL), a complete visual-spatial natural language, whereas public service counters, hospitals, banks, and schools operate in spoken and written Kannada. Almost no assistive software bridges ISL and Kannada bidirectionally on commodity hardware without privacy-invasive video uploads. This paper presents Silent Bridge, a real-time, privacy-preserving edge-AI assistive platform that (i) recognizes isolated ISL signs and continuous ISL sentences from a standard webcam and outputs grammatical Kannada text and speech, (ii) supports user-extensible few-shot custom sign enrollment in the browser via ONNX WebAssembly, and (iii) translates Kannada text into continuous ISL signing on a procedural 3D avatar. Hand and facial landmarks (147-D schema hands-face-v2) are extracted entirely in the browser via MediaPipe Tasks (v0.10.21) so raw camera frames never leave the client device. Trained exclusively on checksum-verified public ISL corpora (Amrita SLR, INCLUDE, and ISL-CSLTR) and evaluated strictly on unseen signers and unseen recording rounds, a Bidirectional GRU with 3-view Test-Time Augmentation (TTA) and selective HTTP-range INCLUDE expansion improved isolated-sign accuracy on 268 identical held-out videos from 59.7% (logistic regression baseline) to 79.9% (81.5% across all 314 held-out test videos, macro-F1 0.800), and on 213 videos of two completely unseen signers from 54.5% to 76.5% (exact paired McNemar p = 0.003). For continuous signing, a log-likelihood fusion of a 2-layer BiGRU-CTC decoder and a sentence classifier achieves 29.2% exact-sentence accuracy (GER 0.685) across 101 corpus sentences on unseen rounds. In the reverse direction, a multi-tier Kannada Subject–Object–Verb (SOV) grammar engine and an analytic two-bone Inverse Kinematics (IK) 3D avatar pipeline with Gaussian trajectory smoothing eliminated all 367 mid-sign arm drops across 144 dataset-derived clips and reduced wrist acceleration jitter by 36.0%.

DOI: https://doi.org/10.5281/zenodo.24392677

× How can I help you?