GTDD: Generative Test-Driven Development for AI Coding Agents with Adversarial Testing
Abstract
Test-driven development gives AI coding agents executable requirements for implementing software. Because these agents can adapt their implementations to the examples they observe, passing a predetermined collection of tests can leave substantial parts of the intended behavior unimplemented. We propose Generative Test-Driven Development (GTDD), a formulation of test-driven development in which a separate testing agent generates new inputs after each candidate implementation is fixed, using a human-specified behavioral contract and the feedback from earlier rounds. A trusted evaluator checks these inputs, returns reduced counterexamples to the coding agent, and saves them for regression testing, so development continually confronts failures beyond the initial examples. We characterize the evidence that this process provides through a finite-population analysis of false acceptance under adaptive candidate selection. The resulting bounds quantify how test visibility and repeated feedback affect acceptance, and show that fresh random audits after candidate commitment control false acceptance across development rounds. In a paired experiment on a stateful key-value store, both policies that regenerated tests during development ended with lower mean failure rates than the policy whose tests were generated once by the same language model, and giving the tester the candidate's source produced no detectable additional improvement. Further conditions requesting equal numbers of tests did not isolate any single feature of the policies as the source of this difference. GTDD combines this adaptive development feedback with established regression tests and an independent acceptance rule.
Related
- Optimal Adversarial Testing: Extracting Honest Test Results from Dishonest Test Takers
- Prediction model for compressive strength of alkali-activated multi-source solid-waste grouting materials based on generative adversarial networks and machine learning
- Testing Conditional Distribution Equality Using Generative Models
- AntennaFlow: A Generative Flow Model for Offset Correction in Phaseless Antenna Testing
- Stress-Testing Dynamical and Generative Downscaling Using Subseasonal Extreme Precipitation Forecasts
BibTeX
@misc{kato2026gtdd,
title = {{GTDD: Generative Test-Driven Development for AI Coding Agents with Adversarial Testing}},
author = {Masahiro Kato},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.02952},
eprint = {2610.02952},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.02952},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - UNPB
TI - GTDD: Generative Test-Driven Development for AI Coding Agents with Adversarial Testing
AU - Masahiro Kato
PY - 2026
DA - 2026/10/02
PB - arXiv
DO - 10.48550/arxiv.2610.02952
UR - https://arxiv.org/abs/2610.02952
AB - Test-driven development gives AI coding agents executable requirements for implementing software. Because these agents can adapt their implementations to the examples they observe, passing a predetermined collection of tests can leave substantial parts of the intended behavior unimplemented. We propose Generative Test-Driven Development (GTDD), a formulation of test-driven development in which a separate testing agent generates new inputs after each candidate implementation is fixed, using a human-specified behavioral contract and the feedback from earlier rounds. A trusted evaluator checks these inputs, returns reduced counterexamples to the coding agent, and saves them for regression testing, so development continually confronts failures beyond the initial examples. We characterize the evidence that this process provides through a finite-population analysis of false acceptance under adaptive candidate selection. The resulting bounds quantify how test visibility and repeated feedback affect acceptance, and show that fresh random audits after candidate commitment control false acceptance across development rounds. In a paired experiment on a stateful key-value store, both policies that regenerated tests during development ended with lower mean failure rates than the policy whose tests were generated once by the same language model, and giving the tester the candidate's source produced no detectable additional improvement. Further conditions requesting equal numbers of tests did not isolate any single feature of the policies as the source of this difference. GTDD combines this adaptive development feedback with established regression tests and an independent acceptance rule.
ER -