TASTE: A Designer-Annotated Multi-Dimensional Preference Dataset for AI-Generated Graphic Design
Abstract
Text-to-image models now generate graphic design at production scale, yet their supervision still comes primarily from photo-style preference datasets with a single overall verdict per comparison. Designers evaluate designs along several distinct axes (e.g., typography, layout, color harmony) that a single preference label collapses. We release \emph{TASTE} \textit{(Typography, Aesthetics, Spatial, Tone, Etc.)}, a multi-dimensional preference dataset in which two disjoint cohorts of five professional designers each ranked outputs from four current text-to-image models across nine criteria along with per-image hallucination flags. We pair the dataset with two contributions. First, a criterion-agnostic signal-validation framework based on Kendall's τ, majority-vote probability, and Condorcet cycles against exact iid-uniform nulls; the analysis reveals significant but moderate designer agreement, with every TASTE criterion rejecting the random-rater null. Second, we benchmark preference models on TASTE and find that off-the-shelf VLM judges and dedicated T2I scorers fail to reach majority agreement with the designer panel, while a small MLP head trained directly on TASTE substantially narrows the gap to the single-rater ceiling, setting a baseline for future TASTE-trained preference models.
Related
- High-Dimensional Asymptotics and Dataset Selection for Private Transfer Learning
- Invitation to torso parameters of annotated graphs
- Simulated Annealing for Model‐Robust Partial Profile Choice Designs in Healthcare Preference Studies
- Robust Direct Preference Optimization under Preference Label-Flip Attacks
- Reliability-Weighted Direct Preference Optimization: Penalized Weights for Robust Preference Learning
BibTeX
@misc{zhu2026taste,
title = {{TASTE: A Designer-Annotated Multi-Dimensional Preference Dataset for AI-Generated Graphic Design}},
author = {Haonan Zhu and Elad Hirsch and Alexandria Minetti and Allison Nulty and Ben Huffman and Purvanshi Mehta},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2605.20731},
eprint = {2605.20731},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.20731},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - TASTE: A Designer-Annotated Multi-Dimensional Preference Dataset for AI-Generated Graphic Design
AU - Haonan Zhu
AU - Elad Hirsch
AU - Alexandria Minetti
AU - Allison Nulty
AU - Ben Huffman
AU - Purvanshi Mehta
PY - 2026
DA - 2026/09/21
PB - arXiv
DO - 10.48550/arxiv.2605.20731
UR - https://arxiv.org/abs/2605.20731
AB - Text-to-image models now generate graphic design at production scale, yet their supervision still comes primarily from photo-style preference datasets with a single overall verdict per comparison. Designers evaluate designs along several distinct axes (e.g., typography, layout, color harmony) that a single preference label collapses. We release \emph{TASTE} \textit{(Typography, Aesthetics, Spatial, Tone, Etc.)}, a multi-dimensional preference dataset in which two disjoint cohorts of five professional designers each ranked outputs from four current text-to-image models across nine criteria along with per-image hallucination flags. We pair the dataset with two contributions. First, a criterion-agnostic signal-validation framework based on Kendall's τ, majority-vote probability, and Condorcet cycles against exact iid-uniform nulls; the analysis reveals significant but moderate designer agreement, with every TASTE criterion rejecting the random-rater null. Second, we benchmark preference models on TASTE and find that off-the-shelf VLM judges and dedicated T2I scorers fail to reach majority agreement with the designer panel, while a small MLP head trained directly on TASTE substantially narrows the gap to the single-rater ceiling, setting a baseline for future TASTE-trained preference models.
ER -