EBITH: A Smartphone-Based Multimodal Accessibility System for Blind and Deaf Users Using Computer Vision and Sign Language Recognition
- 1FAST National University of Computer and Emerging Sciences
Abstract
Assistive apps usually serve one disability at a time, so a deaf person and a blind person have no common channel. We present EBITH, a smartphone system that connects deaf users of Pakistani Sign Language (PSL) with blind and hearing users in one conversation, and helps blind users move through indoor spaces in Pakistani institutions. It has three components. (1) A PSL sentence recogniser built on signer-normalised pose and hand landmarks: evaluated leave-one-signer-out on 15 everyday phrases, it reaches 96.7% top-1 accuracy (95% CI 93.3–99.2) and 100% top-3, and a confidence threshold calibrated on unseen signers answers 97.5% of clips with 99.1% precision. An appearance-based CNN–LSTM scores 82.7% on its original split but only 10.1% on unseen signers, which motivates the landmark design and signer-independent evaluation. (2) A 40-class indoor detector (YOLO11s) for classrooms, offices and corridors, including 19 navigation-relevant classes absent from COCO, such as doors, stairs and door handles. Non-exhaustive training labels made it ignore most chairs in real classrooms; completing them with a COCO teacher raises mAP50 on exhaustively labelled local photos from 0.250 to 0.441, and fine-tuning on a few dozen local photos raises it from 0.473 to 0.723 in 5-fold cross-validation. (3) A hazard-first scene description that speaks warnings such as stairs or a close obstacle first, with a distinct vibration; a photo is described in about 0.5 s. We will release the data-processing, training and evaluation code.
Related
- Smartphone-Native Bidirectional Tactile Multimodal Large Language Models: A Unified Framework for Tactile Perception, Reasoning, Generation, and Physical Feedback
- Smartphone-Native Bidirectional Gustatory Multimodal Large Language Models: A Unified Framework for Taste Sensing, Reasoning, Generation, and Physical Taste Feedback
- Smartphone-Native Bidirectional Olfactory Multimodal Large Language Models: A Unified Framework for Smell Sensing, Semantic Reasoning, Scent Generation, and Closed-Loop Olfactory Interaction
- Smartphone-Native Bidirectional Pain-Related Multimodal Large Language Models: A Unified Framework for Pain Sensing, Contextual Reasoning, Safe Sensory Feedback, and Adaptive Interaction
- Embodied multimodal human-computer interaction for English language learning: integrating speech, gestures, emotion, and adaptive feedback.
BibTeX
@misc{farooqui2026ebith,
title = {{EBITH: A Smartphone-Based Multimodal Accessibility System for Blind and Deaf Users Using Computer Vision and Sign Language Recognition}},
author = {Muhammad Zohaib Farooqui and Abeer Saad and Moeed Uddin},
howpublished = {Zenodo},
year = {2026},
month = oct,
doi = {10.5281/zenodo.23465943},
url = {https://zenodo.org/records/23465943},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - EBITH: A Smartphone-Based Multimodal Accessibility System for Blind and Deaf Users Using Computer Vision and Sign Language Recognition
AU - Muhammad Zohaib Farooqui
AU - Abeer Saad
AU - Moeed Uddin
PY - 2026
DA - 2026/10/19
PB - Zenodo
DO - 10.5281/zenodo.23465943
UR - https://zenodo.org/records/23465943
AB - Assistive apps usually serve one disability at a time, so a deaf person and a blind person have no common channel. We present EBITH, a smartphone system that connects deaf users of Pakistani Sign Language (PSL) with blind and hearing users in one conversation, and helps blind users move through indoor spaces in Pakistani institutions. It has three components. (1) A PSL sentence recogniser built on signer-normalised pose and hand landmarks: evaluated leave-one-signer-out on 15 everyday phrases, it reaches 96.7% top-1 accuracy (95% CI 93.3–99.2) and 100% top-3, and a confidence threshold calibrated on unseen signers answers 97.5% of clips with 99.1% precision. An appearance-based CNN–LSTM scores 82.7% on its original split but only 10.1% on unseen signers, which motivates the landmark design and signer-independent evaluation. (2) A 40-class indoor detector (YOLO11s) for classrooms, offices and corridors, including 19 navigation-relevant classes absent from COCO, such as doors, stairs and door handles. Non-exhaustive training labels made it ignore most chairs in real classrooms; completing them with a COCO teacher raises mAP50 on exhaustively labelled local photos from 0.250 to 0.441, and fine-tuning on a few dozen local photos raises it from 0.473 to 0.723 in 5-fold cross-validation. (3) A hazard-first scene description that speaks warnings such as stairs or a close obstacle first, with a distinct vibration; a photo is described in about 0.5 s. We will release the data-processing, training and evaluation code.
ER -