Sign language teaching protocol with an interactive robotic arm and real-time vision feedback
4 affiliations
- 1Facultad de Ingenierías, Centro de Investigación en Mecatrónica y Sistemas Interactivos (MIST), Universidad Tecnológica Indoamérica, Quito, Ecuador.
- 2Centro de Investigación de Ciencias Humanas y de la Educación (CICHE), Universidad Tecnológica Indoamérica, Quito, Ecuador.
- 3Facultad de Salud y Bienestar, Carrera de Psicología, Pontificia Universidad Católica del Ecuador, Quito, Ecuador.
- 4Centro de Traducciones de la PUCE, Pontificia Universidad Católica del Ecuador, Quito, Ecuador.
Abstract
Background: Sign Language (SL) is essential for Deaf communities, yet limited SL proficiency sustains communication barriers in education and services. We propose an interactive tutoring system for Ecuadorian Sign Language (EcSL) that combines real-time, vision-based gesture assessment with an embodied “reward” delivered by a robotic arm, using fruit vocabulary as an initial learning domain. Methods: We will conduct a two-arm randomized controlled experimental design, pretest–posttest control-group study with individual random assignment. About 70 university students without formal EcSL training will learn EcSL fruit signs through video and sign recognition feedback module. The system uses a webcam, MediaPipe landmark extraction, and a landmark-based fusion classifier that combines a local bidirectional LSTM with a transferred ST-GCN motion encoder; modules communicate via Robot Operating System (ROS). Outcomes. A total of six dependent variables will be assessed in both the experimental and control groups: (a) handshape, (b) palm orientation, (c) location, (d) movement, (e) non-manual signals and expressions, and (f) global coordination. Analyses will use descriptive statistics, reliability checks when applicable, and a mixed repeated-measures MANOVA. Discussion: By coupling immediate feedback with embodied reinforcement, delivered through a robotic arm, the platform aims to support early sign accuracy and provide a motivating practice environment for learners. The robotic arm responds to participants’ performance with physical rewards, thereby reinforcing correct execution through tangible, real-time stimuli. Results will guide improvements in robustness and inform extensions to broader vocabularies and deployment in educational contexts.
Related
- Practice-based initial teacher education in alternative teaching certification programs: A scoping review protocol
- Teaching and learning strategies for the derivative in engineering education: A scoping review protocol
- Answering clinicians' questions over trial evidence tables with verifiable, feedback-driven language models
- Predictive and feedback signals differently shape the formation of language representations
- The Use of Causal Language in Case Reports and Case Series Reporting Ischemic Stroke Following Cervical Manual Therapy: A Qualitative Study Protocol
BibTeX
@article{zapata2026sign,
title = {{Sign language teaching protocol with an interactive robotic arm and real-time vision feedback}},
author = {Mireya Zapata and Andrès Rubio Proaño and Hugo Arias-Flores and Alejandro Camino and Carlos Ramos-Galarza and Jorge Banet},
journal = {PLOS One},
year = {2026},
month = oct,
doi = {10.1371/journal.pone.0345934},
url = {https://doi.org/10.1371/journal.pone.0345934},
note = {CC BY},
}
RIS
TY - JOUR
TI - Sign language teaching protocol with an interactive robotic arm and real-time vision feedback
AU - Mireya Zapata
AU - Andrès Rubio Proaño
AU - Hugo Arias-Flores
AU - Alejandro Camino
AU - Carlos Ramos-Galarza
AU - Jorge Banet
PY - 2026
DA - 2026/10/28
JO - PLOS One
DO - 10.1371/journal.pone.0345934
UR - https://doi.org/10.1371/journal.pone.0345934
AB - Background: Sign Language (SL) is essential for Deaf communities, yet limited SL proficiency sustains communication barriers in education and services. We propose an interactive tutoring system for Ecuadorian Sign Language (EcSL) that combines real-time, vision-based gesture assessment with an embodied “reward” delivered by a robotic arm, using fruit vocabulary as an initial learning domain. Methods: We will conduct a two-arm randomized controlled experimental design, pretest–posttest control-group study with individual random assignment. About 70 university students without formal EcSL training will learn EcSL fruit signs through video and sign recognition feedback module. The system uses a webcam, MediaPipe landmark extraction, and a landmark-based fusion classifier that combines a local bidirectional LSTM with a transferred ST-GCN motion encoder; modules communicate via Robot Operating System (ROS). Outcomes. A total of six dependent variables will be assessed in both the experimental and control groups: (a) handshape, (b) palm orientation, (c) location, (d) movement, (e) non-manual signals and expressions, and (f) global coordination. Analyses will use descriptive statistics, reliability checks when applicable, and a mixed repeated-measures MANOVA. Discussion: By coupling immediate feedback with embodied reinforcement, delivered through a robotic arm, the platform aims to support early sign accuracy and provide a motivating practice environment for learners. The robotic arm responds to participants’ performance with physical rewards, thereby reinforcing correct execution through tangible, real-time stimuli. Results will guide improvements in robustness and inform extensions to broader vocabularies and deployment in educational contexts.
SN - 1932-6203
ER -