On the variability of generative artificial intelligence methods in conceptual modeling: an experimental evaluation on combining causal maps
- 1Old Dominion University (ODU) Virginia Modeling, Analysis, and Simulation Center (VMASC), Suffolk, Virginia, United States of America
- 2School of Mines in Alès (IMT Mines Alès), Alès, France
Abstract
Large Language Models (LLMs) increasingly support time-consuming conceptual modeling tasks such as building or combining models from text. In this context, causal maps are labeled, directed graphs that represent causal relationships (edges) between selected factors (nodes) in a problem domain. Modelers control the output through the LLM’s prompt and decoding parameters such as temperature, and recent studies either use a default temperature or minimize it by expecting that it supports replicability. However, evidence about this expectation remains limited, as does our understanding of how different LLMs and decoding or prompting strategies shape results. We assess the extent to which findings are an artifact of the LLM or its variability rather than a reflection of the modeler’s controls. We focus on the common conceptual modeling task of combining models, exemplified through three algorithms that reflect a variety of prompting techniques. We use five LLMs (DeepSeek, Gemini, GPT, Qwen, Ministral) and different versions to account for the potential impact of the architectures. Using the Design of Experiments approach of factorial analyses (supplemented by paired tests and confidence interval checks), our results show that outcomes differ significantly across LLMs and versions. Depending on the algorithm, randomness from the LLM can be the dominating factor, responsible for up to 96% of the variation in results. This happens even if we directly control the decoding, as evidenced by our comparison of greedy, beam-search, and contrastive decoding. We find that prompt effects are systematic but metric-dependent, while decoding affects performance but there is no universally best strategy. Since our findings suggest that replicability of conceptual modeling studies using LLMs may be limited, future studies may report the distribution of performance measures alongside exact model, decoding, replication, and baseline-normalization settings.
Related
- Generative artificial intelligence creates delicious, sustainable, and nutritious burgers
- How Generative Artificial Intelligence (GAI) reshapes students’ creativity in higher education: A chain path modeling Study from “Exploration–Exploitation” to “Co-Creation–Reflection”
- Perceived GenAI competency and university teachers’ intention to continue using generative artificial intelligence: Extending the technology acceptance model
- A generative artificial intelligence approach for peptide antibiotic optimization
- Generative artificial intelligence in animal genomics for smart agriculture: Applications, challenges, and future prospects
BibTeX
@article{flandre2026variability,
title = {{On the variability of generative artificial intelligence methods in conceptual modeling: an experimental evaluation on combining causal maps}},
author = {Noé Y. Flandre and Cédric Daumas and Philippe J. Giabbanelli},
journal = {PLOS One},
year = {2026},
month = oct,
doi = {10.1371/journal.pone.0361281},
url = {https://doi.org/10.1371/journal.pone.0361281},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - On the variability of generative artificial intelligence methods in conceptual modeling: an experimental evaluation on combining causal maps
AU - Noé Y. Flandre
AU - Cédric Daumas
AU - Philippe J. Giabbanelli
PY - 2026
DA - 2026/10/27
JO - PLOS One
DO - 10.1371/journal.pone.0361281
UR - https://doi.org/10.1371/journal.pone.0361281
AB - Large Language Models (LLMs) increasingly support time-consuming conceptual modeling tasks such as building or combining models from text. In this context, causal maps are labeled, directed graphs that represent causal relationships (edges) between selected factors (nodes) in a problem domain. Modelers control the output through the LLM’s prompt and decoding parameters such as temperature, and recent studies either use a default temperature or minimize it by expecting that it supports replicability. However, evidence about this expectation remains limited, as does our understanding of how different LLMs and decoding or prompting strategies shape results. We assess the extent to which findings are an artifact of the LLM or its variability rather than a reflection of the modeler’s controls. We focus on the common conceptual modeling task of combining models, exemplified through three algorithms that reflect a variety of prompting techniques. We use five LLMs (DeepSeek, Gemini, GPT, Qwen, Ministral) and different versions to account for the potential impact of the architectures. Using the Design of Experiments approach of factorial analyses (supplemented by paired tests and confidence interval checks), our results show that outcomes differ significantly across LLMs and versions. Depending on the algorithm, randomness from the LLM can be the dominating factor, responsible for up to 96% of the variation in results. This happens even if we directly control the decoding, as evidenced by our comparison of greedy, beam-search, and contrastive decoding. We find that prompt effects are systematic but metric-dependent, while decoding affects performance but there is no universally best strategy. Since our findings suggest that replicability of conceptual modeling studies using LLMs may be limited, future studies may report the distribution of performance measures alongside exact model, decoding, replication, and baseline-normalization settings.
SN - 1932-6203
ER -