Authors: Dr. Nandini C,, Jayant Rajendra Habbu, Midhun Manoj,, Moulya R Hegde, Prerana A J
Abstract: In recent years, assistive technologies for visually and hearing-impaired individuals have predominantly focused on single-modality solutions, limiting their effectiveness for users with dual sensory impairments. This paper proposes EchoSense, a novel mobile-based assistive system integrating real-time visual scene interpretation and personalized sound detection enhanced with emotion recognition. Through a comprehensive survey of recent literature encompassing scene captioning, sound classification, emotion detection, and explainable AI, key limitations such as lack of multi-modality, minimal personalization, and dependence on cloud-based processing are identified. EchoSense addresses these gaps by offering an on-device, privacy-preserving, and offline-capable platform that delivers adaptive multimodal feedback tailored to blind, deaf, and deaf-blind users. The paper details the system’s conceptual framework, design considerations, and methodology, laying the groundwork for future implementation and evaluation.
DOI: http://doi.org/
