{
  "abstract": "Background: Social media offers valuable information for public health surveillance, but detecting valid symptom expressions in noisy and informal text remains challenging. Fine-tuned language models can achieve strong test accuracy while relying on spurious patterns, limiting their reliability in real-world applications.\n\nObjective: This study aimed to develop a more robust symptom detection model by incorporating surveillance-relevant attributes into the training objective and introducing a counterfactual-based evaluation framework to assess attribute sensitivity.\n\nMethods: We proposed a concept-aware learning framework (CALF), which combines multiattribute counterfactual data augmentation with multitask learning. We compared standard fine-tuning (baseline) with data-level interventions, a model-level intervention, hybrid strategies, and prompt-based methods across multiple encoders and LLMs for data augmentation. Performance was evaluated on the following 3 datasets: the MedWeb test using macro F1-score (m-F1), the MedWeb checklist using flip accuracy (FA), and a supplementary test set using F1-score for flu-or-fever detection (f-F1). FA measures sensitivity to surveillance-relevant attributes, and the supplementary test score serves as an indicator for out-of-distribution performance.\n\nResults: The baseline achieved an m-F1-score of 0.871 on the test set but a low overall FA on the checklist (FA=0.342). It also has limited sensitivity to recovery status, timing of occurrence, and uncertainty of expression. Compared with the baseline, CALF has a comparable MedWeb test score (m-F1=0.880 vs 0.871; P=.18), but a substantially higher FA (FA=0.830 vs 0.342; P=.001) and supplementary test score (f-F1=0.856 vs 0.797; P=.007). CALF also improved sensitivity to recovery, timing, and uncertainty across pretrained encoders. The gains were generally larger when GPT-4.1 (OpenAI) was used for data augmentation. Under GPT-4.1, CALF often achieved the highest overall FA among encoder-based methods. When Qwen2.5-7B-Instruct (Alibaba Cloud) was used, an alternative hybrid strategy may achieve comparable or slightly better results. Checklist performance was strongly associated with supplementary test performance (ρ=0.811), whereas MedWeb test performance showed weak association with supplementary test performance (ρ=–0.042). Prompting GPT-4.1 achieved the highest FA overall (FA=0.920) and the highest supplementary test score (f-F1=0.910), but a lower test set score than the baseline (m-F1=0.776 vs 0.871).\n\nConclusions: Standard held-out metrics alone do not adequately characterize robustness to dataset-specific or spuriously correlated features. Models may perform well on the test set while remaining insensitive to attributes that influence whether a symptom mention is relevant to surveillance. The MedWeb checklist complements conventional evaluation by measuring sensitivity to controlled attribute changes and may indicate out-of-distribution performance. CALF improved sensitivity to surveillance-relevant attributes while maintaining strong classification performance, supporting concept-aware learning as a practical strategy for developing more robust encoder-based models.",
  "abstract_shown": true,
  "affiliations": [
    "Nara Institute of Science and Technology Ikoma, Nara Japan",
    "University of the Philippines Diliman Quezon City, Metro Manila Philippines"
  ],
  "article_type": "research article",
  "arxiv": null,
  "author_total": 5,
  "authors": [
    {
      "affs": [
        0,
        1
      ],
      "name": "Michael Van Supranes",
      "orcid": "0009-0002-5776-2308"
    },
    {
      "affs": [
        0
      ],
      "name": "Shaowen Peng",
      "orcid": "0000-0003-4020-9100"
    },
    {
      "affs": [
        0
      ],
      "name": "Tomohiro Nishiyama",
      "orcid": "0000-0003-1538-8266"
    },
    {
      "affs": [
        0
      ],
      "name": "Shoko Wakamiya",
      "orcid": "0000-0002-9371-1340"
    },
    {
      "affs": [
        0
      ],
      "name": "Eiji Aramaki",
      "orcid": "0000-0003-0201-3609"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/d70802vp8j66t.bib",
  "cited_by": 0,
  "doi": "10.2196/94769",
  "fulltext_url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC13720407/",
  "id": "d70802vp8j66t",
  "issns": [
    "2291-9694"
  ],
  "landing_url": "https://doi.org/10.2196/94769",
  "language": "en",
  "licence": "CC-BY-4.0",
  "licence_label": "CC BY 4.0",
  "licence_source": "article",
  "licence_url": "https://creativecommons.org/licenses/by/4.0/",
  "metadata_licence": null,
  "native_subjects": [
    [
      "heading",
      "Original Paper"
    ],
    [
      "openalex",
      "2713|Data-Driven Disease Surveillance"
    ]
  ],
  "openalex": "W7134821972",
  "pdf_url": null,
  "pmcid": "PMC13720407",
  "pmid": "42879779",
  "published": "2026-10-15",
  "publisher": "JMIR Publications Inc.",
  "ris": "https://research.draconitic.com/paper/d70802vp8j66t.ris",
  "sources": [
    {
      "first_seen": "2026-11-03T12:21:17Z",
      "native_id": "PMC13720407",
      "source": "pmc",
      "url": "https://doi.org/10.2196/94769"
    }
  ],
  "subjects": [
    {
      "field": "medicine",
      "origin": "openalex:2713|Data-Driven Disease Surveillance",
      "subfield": "epidemiology-public-health"
    }
  ],
  "title": "Concept-Aware Learning Framework for Detecting Common Symptoms in Social Media Text: Model Development and Evaluation Study",
  "title_text": "Concept-Aware Learning Framework for Detecting Common Symptoms in Social Media Text: Model Development and Evaluation Study",
  "type": "peer-reviewed",
  "updated_at": "2026-11-03T12:21:17Z",
  "url": "https://research.draconitic.com/paper/d70802vp8j66t",
  "venue": "JMIR Medical Informatics",
  "versions": []
}