Structure-First Text Generation: A Unified Framework for Hierarchical Planning, Content Realization, Ordering, and Composition in Autoregressive Language Models
Abstract
Autoregressive language models typically generate text by predicting successive tokens conditioned on previously generated tokens. Although this approach can produce coherent language, it does not explicitly require the model to establish a document structure before realizing its linguistic content. Human writers, by contrast, often adopt a structure-first writing strategy: they identify a topic, construct an outline, define sections and paragraphs, and subsequently fill these structural units with content. This paper proposes Structure-First Text Generation (SFTG), a conceptual framework for incorporating this writing strategy into autoregressive language generation. The framework represents documents as hierarchical structures containing document-level sections, paragraphs, sentences, and words. It distinguishes structural planning, content-slot specification, linguistic realization, unit ordering, composition, and iterative revision. Importantly, the framework permits both fixed-order generation, in which structural units are realized according to a predetermined sequence, and flexible-order generation, in which units may be generated, revised, reordered, or connected according to their dependencies and estimated utility. A formal objective is introduced to balance structural completeness, semantic coherence, factual consistency, linguistic quality, redundancy, and computational cost. The paper also proposes a family of implementation strategies, training objectives, evaluation criteria, and experimental designs. Existing research on content planning, sentence ordering, hierarchical generation, and plan-conditioned realization establishes substantial foundations for this direction. Accordingly, the proposed contribution is not the invention of outline-based generation itself, but a unified framework for investigating structure-first generation across multiple linguistic granularities and alternative realization orders. The framework remains a research proposal requiring empirical validation.
Related
- PR-JEPA: Decoupling Latent Planning from Few-Step Discrete Realization for Non-Autoregressive Generation of Code, Legal Documents, and Control
- LinkerGen: Guided Autoregressive Generation of Organic Linkers for Reticular Chemistry
- Limitations on Autoregressive Token Generation via Sequential Sampling
- Meta Non-Autoregressive Models: A Framework for Generating, Selecting, and Adapting Non-Autoregressive Generation Mechanisms
- Scaling an Autoregressive Transformer for Single-Cell Generation
BibTeX
@misc{zhang2026structurefirst,
title = {{Structure-First Text Generation: A Unified Framework for Hierarchical Planning, Content Realization, Ordering, and Composition in Autoregressive Language Models}},
author = {Jincheng Zhang},
howpublished = {Zenodo},
year = {2026},
month = oct,
doi = {10.5281/zenodo.23614887},
url = {https://zenodo.org/records/23614887},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Structure-First Text Generation: A Unified Framework for Hierarchical Planning, Content Realization, Ordering, and Composition in Autoregressive Language Models
AU - Jincheng Zhang
PY - 2026
DA - 2026/10/24
PB - Zenodo
DO - 10.5281/zenodo.23614887
UR - https://zenodo.org/records/23614887
AB - Autoregressive language models typically generate text by predicting successive tokens conditioned on previously generated tokens. Although this approach can produce coherent language, it does not explicitly require the model to establish a document structure before realizing its linguistic content. Human writers, by contrast, often adopt a structure-first writing strategy: they identify a topic, construct an outline, define sections and paragraphs, and subsequently fill these structural units with content. This paper proposes Structure-First Text Generation (SFTG), a conceptual framework for incorporating this writing strategy into autoregressive language generation. The framework represents documents as hierarchical structures containing document-level sections, paragraphs, sentences, and words. It distinguishes structural planning, content-slot specification, linguistic realization, unit ordering, composition, and iterative revision. Importantly, the framework permits both fixed-order generation, in which structural units are realized according to a predetermined sequence, and flexible-order generation, in which units may be generated, revised, reordered, or connected according to their dependencies and estimated utility. A formal objective is introduced to balance structural completeness, semantic coherence, factual consistency, linguistic quality, redundancy, and computational cost. The paper also proposes a family of implementation strategies, training objectives, evaluation criteria, and experimental designs. Existing research on content planning, sentence ordering, hierarchical generation, and plan-conditioned realization establishes substantial foundations for this direction. Accordingly, the proposed contribution is not the invention of outline-based generation itself, but a unified framework for investigating structure-first generation across multiple linguistic granularities and alternative realization orders. The framework remains a research proposal requiring empirical validation.
ER -