When Does Text Inform? Benchmarking Information-Theoretic Metrics for Multimodal Time-Series Forecasting
Abstract
Multimodal forecasting models that combine time series with text annotations promise richer prediction through textual context, but how do we know whether a text annotation meaningfully contributes to the forecasters prediction? This is an information-theoretic question, but to evaluate whether information-theoretic metrics can reliably measure the predictive value an annotation provides, a ground truth benchmark is needed, and none currently exist. We create a synthetic time series signal with annotations in three categories: semantically correct, incorrect, and irrelevant. Because the data generation process is fully controlled, ground-truth information content is known exactly, enabling principled evaluation of six complementary mutual information estimators (KSG, MINE, InfoNCE, CCA, PID and V-information). We show that all six estimators identify correct annotations as most informative, and are able to audit the quality of mixed text corpora, choosing the annotations that result in the best downstream forecasting results without the need for model training. Our benchmark identifies limitations of each estimator, and these are validated on seven real-world datasets, which show how estimator performance differs on weak signals. Finally, we establish practical rules for implementing these metrics for annotation auditing and fusion selection.
Related
- Dependency, Compression, and Synergy: A Unified Information-Theoretic View of Multimodal Learning
- Game-Theoretic Foundations for Undiscounted Bisimulation Metrics
- Measuring Learned Equivariance: An Information Theoretic Approach
- A Sample-Based Approach for Hierarchical Information-Theoretic Compression of Probabilistic Occupancy Grids
- Conditional Fisher-Information Central Limit Theorems under Log-Concavity with Information-Theoretic Consequences
BibTeX
@misc{andrews2026when,
title = {{When Does Text Inform? Benchmarking Information-Theoretic Metrics for Multimodal Time-Series Forecasting}},
author = {Emma Andrews and Gianmarco Mengaldo},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.11282},
eprint = {2609.11282},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.11282},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - When Does Text Inform? Benchmarking Information-Theoretic Metrics for Multimodal Time-Series Forecasting
AU - Emma Andrews
AU - Gianmarco Mengaldo
PY - 2026
DA - 2026/09/10
PB - arXiv
DO - 10.48550/arxiv.2609.11282
UR - https://arxiv.org/abs/2609.11282
AB - Multimodal forecasting models that combine time series with text annotations promise richer prediction through textual context, but how do we know whether a text annotation meaningfully contributes to the forecasters prediction? This is an information-theoretic question, but to evaluate whether information-theoretic metrics can reliably measure the predictive value an annotation provides, a ground truth benchmark is needed, and none currently exist. We create a synthetic time series signal with annotations in three categories: semantically correct, incorrect, and irrelevant. Because the data generation process is fully controlled, ground-truth information content is known exactly, enabling principled evaluation of six complementary mutual information estimators (KSG, MINE, InfoNCE, CCA, PID and V-information). We show that all six estimators identify correct annotations as most informative, and are able to audit the quality of mixed text corpora, choosing the annotations that result in the best downstream forecasting results without the need for model training. Our benchmark identifies limitations of each estimator, and these are validated on seven real-world datasets, which show how estimator performance differs on weak signals. Finally, we establish practical rules for implementing these metrics for annotation auditing and fusion selection.
ER -