Designing Temporally Grounded Multimodal Learning Interfaces for Verifiable Interaction
- 1University of Reading
Abstract
Textbooks, diagrams, and recorded lectures often provide complementary accounts of the same concept. Conversational learning assistants can retrieve relevant material, yet learners still need a way to inspect the original source of a generated statement.
This preprint presents the system architecture and interaction design of a multimodal retrieval-augmented learning prototype that indexes textbook passages, descriptions of textbook figures, and timestamped descriptions of sampled lecture-video frames. Its central contribution is a provenance-to-viewer mapping that connects retrieved evidence with inspectable document pages and video timestamps through an explicit source registry.
The accompanying Python and Streamlit research prototype incorporates dense semantic retrieval, an evidence-first split-pane interface, structured evidence controls, and handling for missing or invalid source references. The paper distinguishes implemented mechanisms from design intentions and identifies limitations concerning speech transcription, claim-level attribution, and end-to-end navigation verification.
We describe this approach as verifiable interaction: enabling learners to move from AI-generated explanations or retrieved references toward original materials for independent inspection. This is a system and interaction-design contribution, not a controlled evaluation of retrieval accuracy, usability, or learning outcomes.
Research context
A preliminary version of this work was included in Spotlight Presentations II at the AIED 2026 Workshop on Designing and Evaluating Next-Generation Learning Interfaces. This October 2026 revision is an independent system and interaction-design preprint, not a separately peer-reviewed proceedings publication.
Code repository: https://github.com/William-Turing/multimodal-rag-verifiable-tutor
Workshop programme: https://extendedhorizon.com/aied2026ws-ai-hci-ls/program/
Related
- Learning Protein-Protein Interaction Interfaces from Multimodal Representations
- Multimodal in vitro biocompatibility assessment of functionalized PEDOT neural interfaces
- Modeling user attention continuity and interaction mechanisms in continuous interfaces of digital platforms
- Designing robot-to-robot interaction to support early-stage engagement in a teleoperated Robot Avatar system with multiple robots
- A study on kindergarten teacher–child interaction quality based on a multimodal emotion recognition model integrating speech and text
BibTeX
@misc{wang2026designing,
title = {{Designing Temporally Grounded Multimodal Learning Interfaces for Verifiable Interaction}},
author = {Rongyu Wang},
howpublished = {Zenodo},
year = {2026},
month = oct,
doi = {10.5281/zenodo.23418230},
url = {https://zenodo.org/records/23418230},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Designing Temporally Grounded Multimodal Learning Interfaces for Verifiable Interaction
AU - Rongyu Wang
PY - 2026
DA - 2026/10/16
PB - Zenodo
DO - 10.5281/zenodo.23418230
UR - https://zenodo.org/records/23418230
AB - Textbooks, diagrams, and recorded lectures often provide complementary accounts of the same concept. Conversational learning assistants can retrieve relevant material, yet learners still need a way to inspect the original source of a generated statement. This preprint presents the system architecture and interaction design of a multimodal retrieval-augmented learning prototype that indexes textbook passages, descriptions of textbook figures, and timestamped descriptions of sampled lecture-video frames. Its central contribution is a provenance-to-viewer mapping that connects retrieved evidence with inspectable document pages and video timestamps through an explicit source registry. The accompanying Python and Streamlit research prototype incorporates dense semantic retrieval, an evidence-first split-pane interface, structured evidence controls, and handling for missing or invalid source references. The paper distinguishes implemented mechanisms from design intentions and identifies limitations concerning speech transcription, claim-level attribution, and end-to-end navigation verification. We describe this approach as verifiable interaction: enabling learners to move from AI-generated explanations or retrieved references toward original materials for independent inspection. This is a system and interaction-design contribution, not a controlled evaluation of retrieval accuracy, usability, or learning outcomes. Research context A preliminary version of this work was included in Spotlight Presentations II at the AIED 2026 Workshop on Designing and Evaluating Next-Generation Learning Interfaces. This October 2026 revision is an independent system and interaction-design preprint, not a separately peer-reviewed proceedings publication. Code repository: https://github.com/William-Turing/multimodal-rag-verifiable-tutor Workshop programme: https://extendedhorizon.com/aied2026ws-ai-hci-ls/program/
ER -