Group-aware validation sharply reduces the apparent accuracy of extraction-category classification in public cannabis laboratory data
Abstract
Published models that classify cannabis products from cannabinoid and terpene profiles report near-perfect accuracy: above 95% for resin type, 100% for chemovar class. These figures come from random cross-validation or Kennard-Stone partitioning, which hold out no plant, producer or laboratory. We quantify the cost on a task with explicit group structure. On 33,229 public laboratory-result rows for cannabis concentrates that name a producer, nearly all from Nevada, each with a source-declared extraction category (non-solvent against solvent-based), a gradient-boosted classifier scores 78.5% balanced accuracy and a ROC AUC of 0.89 under random five-fold cross-validation. Holding out the producer leaves 54.0% and 0.62. A signal survives: the pooled held-out AUC is 0.61, with a producer-level bootstrap interval of 0.535 to 0.698. But the reported model's pooled balanced accuracy does not separate from chance (51.6%, interval 47.3 to 56.6); it keeps about 30% of its above-chance AUC, and an unweighted logistic regression 41 to 53%. A designed experiment shows the same direction: with the random split averaged over 200 seeds, 162 HPLC-assayed samples fall from 54.3% to 38.9% under leave-one-variety-out with gradient boosting and from 51.8% to 46.3% with logistic regression, against 33.3% chance. In the market data the same profiles identify the producer at 67.1% against a 14.4% baseline and the laboratory at 92.3% against 40.3%. We conclude that the declared extraction category is only weakly recoverable from these analyte panels for an unseen producer, and that a random split overstates it by 8 to 26 points of balanced accuracy depending on the model and subset. We did not rerun the published studies.
Related
- Validation Design Governs the Apparent Predictive Performance of Machine Learning for PLGA Drug Release
- When Raman Spectral Fidelity and Bacterial Classification Accuracy Diverge
- Development and Validation of Low-Temperature Partitioning Extraction Methods for Multiresidue Determination of Pesticides in Soil and Surface Water
- Hybrid QuEChERS–Micro-Solid Phase Extraction for the Determination of Chloramphenicol in Honey by HPLC-MS/MS: Method Validation and Matrix-Effect Assessment
- Excitation-Energy Accuracy Does Not Guarantee Observable Fidelity: Rethinking Validation of Excited-State Quantum Algorithms
BibTeX
@misc{breguez2026groupaware,
title = {{Group-aware validation sharply reduces the apparent accuracy of extraction-category classification in public cannabis laboratory data}},
author = {Arthur Gonçalves Breguez},
howpublished = {ChemRxiv},
year = {2026},
month = oct,
doi = {10.26434/chemrxiv.15010167/v1},
url = {https://chemrxiv.org/doi/full/10.26434/chemrxiv.15010167/v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Group-aware validation sharply reduces the apparent accuracy of extraction-category classification in public cannabis laboratory data
AU - Arthur Gonçalves Breguez
PY - 2026
DA - 2026/10/09
PB - ChemRxiv
DO - 10.26434/chemrxiv.15010167/v1
UR - https://chemrxiv.org/doi/full/10.26434/chemrxiv.15010167/v1
AB - Published models that classify cannabis products from cannabinoid and terpene profiles report near-perfect accuracy: above 95% for resin type, 100% for chemovar class. These figures come from random cross-validation or Kennard-Stone partitioning, which hold out no plant, producer or laboratory. We quantify the cost on a task with explicit group structure. On 33,229 public laboratory-result rows for cannabis concentrates that name a producer, nearly all from Nevada, each with a source-declared extraction category (non-solvent against solvent-based), a gradient-boosted classifier scores 78.5% balanced accuracy and a ROC AUC of 0.89 under random five-fold cross-validation. Holding out the producer leaves 54.0% and 0.62. A signal survives: the pooled held-out AUC is 0.61, with a producer-level bootstrap interval of 0.535 to 0.698. But the reported model's pooled balanced accuracy does not separate from chance (51.6%, interval 47.3 to 56.6); it keeps about 30% of its above-chance AUC, and an unweighted logistic regression 41 to 53%. A designed experiment shows the same direction: with the random split averaged over 200 seeds, 162 HPLC-assayed samples fall from 54.3% to 38.9% under leave-one-variety-out with gradient boosting and from 51.8% to 46.3% with logistic regression, against 33.3% chance. In the market data the same profiles identify the producer at 67.1% against a 14.4% baseline and the laboratory at 92.3% against 40.3%. We conclude that the declared extraction category is only weakly recoverable from these analyte panels for an unseen producer, and that a random split overstates it by 8 to 26 points of balanced accuracy depending on the model and subset. We did not rerun the published studies.
ER -