Temporal dependency modeling of data inconsistencies in AI-driven ETL systems using graph-based embeddings
5 affiliations
- 1Department of Data Science and Business Systems, SRM Institute of Science and Technology, Kattangulathur Campus
- 2Department of Computational Intelligence, Faculty of Engineering and Technology, SRM Institute of Science and Technology, Kattankulathur Campus
- 3Department of Information Technology, SRM Institute of Science and Technology, Ramapuram Campus
- 4Department of Computer Science and Engineering, SRM Institute of Science and Technology, Ramapuram Campus
- 5Department of Computer Science and Engineering, Vel Tech High Tech Dr. Rangarajan Dr. Sakunthala Engineering College
Abstract
The emergence of heterogeneous data sources, high-velocity data, and changing inconsistency patterns, including missing values, duplication, schema conflicts, and delayed updates, has made the provision of data consistency in AI-driven Extract, Transform, Load (ETL) systems more difficult. Conventional rule-based validation frameworks are not powerful enough to encode complex relationships and time dependencies that are present in modern data pipelines. To overcome these shortcomings, a Temporal Graph-Based Embedding Model (TGBEM) is proposed in this paper that combines Graph Neural Networks (GNNs) in learning structural relationships with Temporal Convolutional Networks (TCNs) in learning time-varying variations. The suggested framework converts ETL data into graphical features to show the dependence between data entities and processes time series to detect changing inconsistencies over time steps. The model is tested on a mix of real-world benchmark datasets and a synthetically created ETL dataset that renders realistic conditions on the pipeline, such as multi-source conflicts and temporal drift. According to the results of the experiments, it is clear that the proposed TGBEM greatly surpasses the traditional and existing versions of deep learning, reaching an accuracy of 95.80%, precision of 94.95%, recall of 93.70%, F1-score of 94.32%, and AUC of 96.10%. Moreover, the model also exhibits lower detection latency, which renders this model applicable in near-real-time validation of ETL. The comparative and ablation studies also support the fact that the combination of graph-based and temporal learning components is vital in improving the detection performance. These results indicate that hybrid graph-temporal models can be successfully used to face the challenging problem of inconsistency in next-generation AI-guided ETL systems.
Related
- Psychosis-associated disruptions to semantic foraging in narrative recollections: Integrating language model embeddings and topic modeling
- Mapping the Landscape of Adolescent Trauma and Posttraumatic Stress Research: A Topic Modeling Study of Research Pathways and Temporal Trends
- A context-aware temporal modeling through unified multi-scale temporal encoding and hierarchical sequence learning for single-channel EEG sleep staging
- Timing Is Everything: Modeling the Temporal Dynamics of Suicidal Ideation
- Medical-oriented multi-label diagnosis of twelve-lead ECG signals via disease-aware relational modeling and pathology-aware temporal evidence aggregation
BibTeX
@article{bhavani2026temporal,
title = {{Temporal dependency modeling of data inconsistencies in AI-driven ETL systems using graph-based embeddings}},
author = {M. Monica Bhavani and G. Sumathy and R. Deeptha and G. Indumathi and P. Selvarani},
journal = {Scientific Reports},
year = {2026},
month = jul,
doi = {10.1038/s41598-026-58928-7},
url = {https://doi.org/10.1038/s41598-026-58928-7},
note = {CC BY},
}
RIS
TY - JOUR
TI - Temporal dependency modeling of data inconsistencies in AI-driven ETL systems using graph-based embeddings
AU - M. Monica Bhavani
AU - G. Sumathy
AU - R. Deeptha
AU - G. Indumathi
AU - P. Selvarani
PY - 2026
DA - 2026/07/01
JO - Scientific Reports
DO - 10.1038/s41598-026-58928-7
UR - https://doi.org/10.1038/s41598-026-58928-7
AB - The emergence of heterogeneous data sources, high-velocity data, and changing inconsistency patterns, including missing values, duplication, schema conflicts, and delayed updates, has made the provision of data consistency in AI-driven Extract, Transform, Load (ETL) systems more difficult. Conventional rule-based validation frameworks are not powerful enough to encode complex relationships and time dependencies that are present in modern data pipelines. To overcome these shortcomings, a Temporal Graph-Based Embedding Model (TGBEM) is proposed in this paper that combines Graph Neural Networks (GNNs) in learning structural relationships with Temporal Convolutional Networks (TCNs) in learning time-varying variations. The suggested framework converts ETL data into graphical features to show the dependence between data entities and processes time series to detect changing inconsistencies over time steps. The model is tested on a mix of real-world benchmark datasets and a synthetically created ETL dataset that renders realistic conditions on the pipeline, such as multi-source conflicts and temporal drift. According to the results of the experiments, it is clear that the proposed TGBEM greatly surpasses the traditional and existing versions of deep learning, reaching an accuracy of 95.80%, precision of 94.95%, recall of 93.70%, F1-score of 94.32%, and AUC of 96.10%. Moreover, the model also exhibits lower detection latency, which renders this model applicable in near-real-time validation of ETL. The comparative and ablation studies also support the fact that the combination of graph-based and temporal learning components is vital in improving the detection performance. These results indicate that hybrid graph-temporal models can be successfully used to face the challenging problem of inconsistency in next-generation AI-guided ETL systems.
SN - 2045-2322
ER -