Incorporating natural language processing-derived features from pathology reports into machine learning models to predict differentiated thyroid cancer recurrence
6 affiliations
- 1Department of Surgery, Icahn School of Medicine at Mount Sinai
- 2Windreich Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai
- 3Department of Statistics and Data Science, Cornell University
- 4Department of Statistics and Actuarial Science, University of Iowa
- 5Center for Biostatistics, Department of Population Health Science and Policy, Icahn School of Medicine at Mount Sinai
- 6Mayo Clinic
Abstract
Objective: To evaluate the performance of machine learning (ML) models tasked with predicting thyroid cancer recurrence using natural language processing (NLP) and large language model (LLM) derived features.
Patients and methods: This was a retrospective study of adult thyroid cancer patients treated at an academic medical center between January 1, 2010 through December 31, 2022. Patients were classified as having cancer recurrence or no recurrence. Novel NLP features were extracted from cytopathology and surgical pathology reports using Term Frequency–Inverse Document Frequency (TF-IDF), Latent Dirichlet Allocation (LDA), and a zero-shot large language model (LLM) classification. Five learning algorithms (Decision Tree, Gradient Boosting, Logistic Regression, Random Forest, and XGBoost) were trained to predict cancer recurrence utilizing a combination of NLP-derived features and structured electronic medical record (EMR) variables. All models were evaluated using stratified 5-fold cross-validation. Model performance was assessed using F1 score and area under the receiver operating characteristic curve (ROC-AUC). The top performing model was hyperparameter-tuned, and feature importance was evaluated using SHapley Additive exPlanations (SHAP) values and gain-based importance.
Results: 310 patients with differentiated thyroid cancer diagnosed on final surgical pathology were included in this study. The optimized XGBoost model incorporating EMR variables and LDA-derived features achieved the best performance, with a cross-validated F1 score of 0.647 on the training set and a held-out test F1-score of 0.631 (95% CI 0.509–0.758) and ROC-AUC of 0.741 (95% CI 0.572-0.892). SHAP analysis and gain-based feature importance identified surgical pathology topic—cervical and lateral neck lymph node dissection—as the dominant predictor of cancer recurrence.
Conclusion: Various classifiers including machine learning models utilizing novel NLP features derived from unstructured pathology reports may enable improved prediction of recurrence in patients with differentiated thyroid cancer.
Related
- Comparison and calibration of the 2025 and 2015 American Thyroid Association recurrence-risk classifications in differentiated thyroid cancer
- Transformer-Based Multimodal Fusion Integrating Computed Tomography, Pathology, and Clinical Features for Oral Squamous Cell Carcinoma Recurrence Prediction
- Emotion Recognition, Face Processing and Crystallized Intelligence in Personality Pathology: A Multimethod Study
- Predicting One-Year Psychiatric Rehospitalization Using Natural Language Processing and Machine Learning Methods Based on Clinical Texts
- Do thoughts actually drive symptoms? Testing cognitive theories of posttraumatic stress with natural language processing in daily diaries
BibTeX
@article{lee2026incorporating,
title = {{Incorporating natural language processing-derived features from pathology reports into machine learning models to predict differentiated thyroid cancer recurrence}},
author = {Denise T. Lee and Steven Yen and Grace Deng and Yehia Elkersh and Andrew M. Thomas and Usha Govindarajulu and Michael L. Marin and Girish N. Nadkarni and David S. Matteson},
journal = {Frontiers in Endocrinology},
year = {2026},
month = oct,
doi = {10.3389/fendo.2026.1939077},
url = {https://www.frontiersin.org/articles/10.3389/fendo.2026.1939077/full},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Incorporating natural language processing-derived features from pathology reports into machine learning models to predict differentiated thyroid cancer recurrence
AU - Denise T. Lee
AU - Steven Yen
AU - Grace Deng
AU - Yehia Elkersh
AU - Andrew M. Thomas
AU - Usha Govindarajulu
AU - Michael L. Marin
AU - Girish N. Nadkarni
AU - David S. Matteson
PY - 2026
DA - 2026/10/21
JO - Frontiers in Endocrinology
DO - 10.3389/fendo.2026.1939077
UR - https://www.frontiersin.org/articles/10.3389/fendo.2026.1939077/full
AB - Objective: To evaluate the performance of machine learning (ML) models tasked with predicting thyroid cancer recurrence using natural language processing (NLP) and large language model (LLM) derived features. Patients and methods: This was a retrospective study of adult thyroid cancer patients treated at an academic medical center between January 1, 2010 through December 31, 2022. Patients were classified as having cancer recurrence or no recurrence. Novel NLP features were extracted from cytopathology and surgical pathology reports using Term Frequency–Inverse Document Frequency (TF-IDF), Latent Dirichlet Allocation (LDA), and a zero-shot large language model (LLM) classification. Five learning algorithms (Decision Tree, Gradient Boosting, Logistic Regression, Random Forest, and XGBoost) were trained to predict cancer recurrence utilizing a combination of NLP-derived features and structured electronic medical record (EMR) variables. All models were evaluated using stratified 5-fold cross-validation. Model performance was assessed using F1 score and area under the receiver operating characteristic curve (ROC-AUC). The top performing model was hyperparameter-tuned, and feature importance was evaluated using SHapley Additive exPlanations (SHAP) values and gain-based importance. Results: 310 patients with differentiated thyroid cancer diagnosed on final surgical pathology were included in this study. The optimized XGBoost model incorporating EMR variables and LDA-derived features achieved the best performance, with a cross-validated F1 score of 0.647 on the training set and a held-out test F1-score of 0.631 (95% CI 0.509–0.758) and ROC-AUC of 0.741 (95% CI 0.572-0.892). SHAP analysis and gain-based feature importance identified surgical pathology topic—cervical and lateral neck lymph node dissection—as the dominant predictor of cancer recurrence. Conclusion: Various classifiers including machine learning models utilizing novel NLP features derived from unstructured pathology reports may enable improved prediction of recurrence in patients with differentiated thyroid cancer.
SN - 1664-2392
ER -