QAMQOR: a benchmark dataset for engagement recognition in robot-assisted autism therapy
- 1Department of Science and Innovation, Astana IT University, Astana, Kazakhstan
- 2School of Computing and AI, Nazarbayev University, Astana, Kazakhstan
Abstract
Reliable engagement recognition is important for adaptive robot-assisted autism therapy (RAAT). However, progress remains constrained by the scarcity of publicly available benchmarks evaluated under realistic deployment conditions. Existing studies commonly adopt random train–test splits, which can inflate reported performance because recordings from the same child or therapy session may appear in both the training and test sets. To address this gap, we introduce QAMQOR, a benchmark dataset for engagement recognition collected during RAAT sessions with 34 children with Autism Spectrum Disorder (ASD), aged 3−12 years. The dataset includes visual engagement annotations derived from therapy session videos and feature representations extracted using two pose-estimation frameworks. The QAMQOR dataset defines four clinically relevant evaluation protocols based on Random, Child-Independent, Session-Independent, and Activity-Independent splits. These evaluation protocols assess model generalization to unseen children, therapy sessions and activity tasks. We benchmark classical machine learning classifiers together with temporal and sequence-learning baselines for both binary and multi-class engagement recognition. Experimental results show that the evaluation protocol influenced reported performance to a degree comparable to, and for fine-grained engagement classification exceeding, the influence of classifier selection: mean macro-F1 varied by 0.12 across the four protocols in the multi-class task, against 0.07 across discriminative classifiers. The Child-Independent split proved substantially more challenging than the Random split. Under the same conditions (windows, protocols, training schedule and untuned hyperparameters) a Temporal Convolutional Network (TCN) did not outperform the classical baselines. Applying identical imbalance handling to every model improved all imbalance-sensitive metrics but left this ordering unchanged, indicating that the result is not an artifact of missing class weights. Across the models evaluated, reported performance was shifted more by evaluation design and by imbalance handling than by architectural choice. Therefore, QAMQOR serves as a reproducible reference point for this task. It also shows why fair evaluation matters: keeping the same child or session out of both the training and test sets is needed to compare future methods on equal terms and to build engagement models that transfer to new children, sessions and activities in RAAT systems.
Related
- A Multicamera Classroom Dataset for Deep Learning-Based Student Action Recognition and Engagement Analysis
- A document classification benchmark dataset on cybersecurity policy domain
- The impact of reference data quality on building segmentation: a nationwide comparison of 15 deep learning models and a new benchmark dataset
- Artificial Intelligence-Based Quality Control of Pharmaceutical Tablets: A Visual Dataset and Benchmark for Automated Defect Detection
- MultiCaRe-Thorax: A Curated Benchmark Dataset and Reproducible Labeling Framework for Multi-Label Thoracic Disease Classification in Chest X-Ray Imaging
BibTeX
@article{rakhymbayeva2026qamqor,
title = {{QAMQOR: a benchmark dataset for engagement recognition in robot-assisted autism therapy}},
author = {Nazerke Rakhymbayeva and Aida Tungatarova and Zhansaule Telisheva and Anara Sandygulova},
journal = {Frontiers in Robotics and AI},
year = {2026},
month = oct,
doi = {10.3389/frobt.2026.1868211},
url = {https://doi.org/10.3389/frobt.2026.1868211},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - QAMQOR: a benchmark dataset for engagement recognition in robot-assisted autism therapy
AU - Nazerke Rakhymbayeva
AU - Aida Tungatarova
AU - Zhansaule Telisheva
AU - Anara Sandygulova
PY - 2026
DA - 2026/10/13
JO - Frontiers in Robotics and AI
DO - 10.3389/frobt.2026.1868211
UR - https://doi.org/10.3389/frobt.2026.1868211
AB - Reliable engagement recognition is important for adaptive robot-assisted autism therapy (RAAT). However, progress remains constrained by the scarcity of publicly available benchmarks evaluated under realistic deployment conditions. Existing studies commonly adopt random train–test splits, which can inflate reported performance because recordings from the same child or therapy session may appear in both the training and test sets. To address this gap, we introduce QAMQOR, a benchmark dataset for engagement recognition collected during RAAT sessions with 34 children with Autism Spectrum Disorder (ASD), aged 3−12 years. The dataset includes visual engagement annotations derived from therapy session videos and feature representations extracted using two pose-estimation frameworks. The QAMQOR dataset defines four clinically relevant evaluation protocols based on Random, Child-Independent, Session-Independent, and Activity-Independent splits. These evaluation protocols assess model generalization to unseen children, therapy sessions and activity tasks. We benchmark classical machine learning classifiers together with temporal and sequence-learning baselines for both binary and multi-class engagement recognition. Experimental results show that the evaluation protocol influenced reported performance to a degree comparable to, and for fine-grained engagement classification exceeding, the influence of classifier selection: mean macro-F1 varied by 0.12 across the four protocols in the multi-class task, against 0.07 across discriminative classifiers. The Child-Independent split proved substantially more challenging than the Random split. Under the same conditions (windows, protocols, training schedule and untuned hyperparameters) a Temporal Convolutional Network (TCN) did not outperform the classical baselines. Applying identical imbalance handling to every model improved all imbalance-sensitive metrics but left this ordering unchanged, indicating that the result is not an artifact of missing class weights. Across the models evaluated, reported performance was shifted more by evaluation design and by imbalance handling than by architectural choice. Therefore, QAMQOR serves as a reproducible reference point for this task. It also shows why fair evaluation matters: keeping the same child or session out of both the training and test sets is needed to compare future methods on equal terms and to build engagement models that transfer to new children, sessions and activities in RAAT systems.
SN - 2296-9144
ER -