LOONA: An LLM Assistant for Spanish Language User Story Quality Assurance in Agile Development
5 affiliations
- 1Department of Electrical and Electronics, Sonora Institute of Technology, Cd. Obregon 85000, Mexico
- 2Navojoa Unit, Sonora Institute of Technology, Navojoa 85860, Mexico
- 3Department of Computing and Design, Sonora Institute of Technology, Cd. Obregon 85000, Mexico
- 4Research Department, Technological Institute of Nogales, Nogales 84065, Mexico
- 5Facultad de Ingeniería, Universidad Panamericana, Alvaro del Portillo 49, Zapopan 45010, Mexico
Abstract
User stories are central to agile requirements engineering, but in practice they often exhibit ambiguity, missing information, and weak or absent acceptance criteria, thereby increasing rework and reducing verifiability. This paper presents LOONA, a mathematically formulated NLP/LLM decision-support pipeline for Spanish user story quality assurance. We model the task as supervised binary classification followed by conditional text generation: a fine-tuned Spanish BERT classifier estimates whether a story requires improvement, and a generative model is triggered only when the estimated class exceeds a decision threshold. The study uses an anonymized reconstruction of 100 Spanish-language user stories from an administrative software backlog, labeled by three agile requirements experts through majority vote. The reconstructed annotation table yields Fleiss’ κ=0.76, indicating substantial agreement. We report stratified 5-fold classification results, lexical baselines, rule-based baselines, automatic rewrite-similarity metrics, and paired expert review of held-out outputs. LOONA achieved Accuracy =0.91, Precision =0.90, Recall =0.92, F1-score =0.91, and MCC =0.82, outperforming TF-IDF logistic regression, TF-IDF support vector machines, and a rule-based baseline. In paired expert review (n=30), 70% of generated rewrites improved over the original story, 20% were unchanged, and 10% degraded; generated acceptance criteria were judged correct in 86.7% of cases and free of hallucinated information in 93.3%. The contribution is framed as a formal and empirical proof of concept for model-level LLM-assisted requirements quality assurance, with explicit limitations regarding dataset size, confidentiality, and human oversight.
Related
- Can Large Language Model-Generated Responses Support Assessment Development? A Human-Calibrated Rasch Benchmark
- EvoRiri: Interpretable Evolutionary Search for Diagnosing Reward Design Failures in a Bangla Mental Health Assistant
- Quadrant classification and development path of gymnastics-related research topics in Chinese high-quality journals: Based on SCI, PKU Core and CSSCI source journals
- Le système intermédiaire d'assurance
- Development and preliminary validation of the Arabic Quality of Life During War Scale (AQOL-War)
BibTeX
@article{mejiadominguez2026loona,
title = {{LOONA: An LLM Assistant for Spanish Language User Story Quality Assurance in Agile Development}},
author = {Francisco Antonio Mejía-Domínguez and Ramón René Palacio and Gilberto Borrego and Samuel González-López and José A. Del-Puerto-Flores},
journal = {Mathematics},
year = {2026},
month = oct,
doi = {10.3390/math14193637},
url = {https://www.mdpi.com/2227-7390/14/19/3637},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - LOONA: An LLM Assistant for Spanish Language User Story Quality Assurance in Agile Development
AU - Francisco Antonio Mejía-Domínguez
AU - Ramón René Palacio
AU - Gilberto Borrego
AU - Samuel González-López
AU - José A. Del-Puerto-Flores
PY - 2026
DA - 2026/10/08
JO - Mathematics
DO - 10.3390/math14193637
UR - https://www.mdpi.com/2227-7390/14/19/3637
AB - User stories are central to agile requirements engineering, but in practice they often exhibit ambiguity, missing information, and weak or absent acceptance criteria, thereby increasing rework and reducing verifiability. This paper presents LOONA, a mathematically formulated NLP/LLM decision-support pipeline for Spanish user story quality assurance. We model the task as supervised binary classification followed by conditional text generation: a fine-tuned Spanish BERT classifier estimates whether a story requires improvement, and a generative model is triggered only when the estimated class exceeds a decision threshold. The study uses an anonymized reconstruction of 100 Spanish-language user stories from an administrative software backlog, labeled by three agile requirements experts through majority vote. The reconstructed annotation table yields Fleiss’ κ=0.76, indicating substantial agreement. We report stratified 5-fold classification results, lexical baselines, rule-based baselines, automatic rewrite-similarity metrics, and paired expert review of held-out outputs. LOONA achieved Accuracy =0.91, Precision =0.90, Recall =0.92, F1-score =0.91, and MCC =0.82, outperforming TF-IDF logistic regression, TF-IDF support vector machines, and a rule-based baseline. In paired expert review (n=30), 70% of generated rewrites improved over the original story, 20% were unchanged, and 10% degraded; generated acceptance criteria were judged correct in 86.7% of cases and free of hallucinated information in 93.3%. The contribution is framed as a formal and empirical proof of concept for model-level LLM-assisted requirements quality assurance, with explicit limitations regarding dataset size, confidentiality, and human oversight.
SN - 2227-7390
ER -