A Competency Assessment Tool for Liver Contrast-Enhanced Ultrasound Among Sonography Trainees: Development and Multicenter Validation Study
21 affiliations
- 1Department of Medical Ultrasonics, Third Affiliated Hospital of Sun Yat-sen University, 600 Tianhe Road, Guangzhou, 510630, China, 86 13925155583
- 2Beijing Tiantan Hospital, Capital Medical University, Beijing, China
- 3Department of Health Statistics, School of Preventive Medicine, Air Force Medical University, Xi'an, China
- 4Department of Ultrasound, China-Japan Union Hospital of Jilin University, Jilin, China
- 5Shanxi Provincial Cancer Hospital, Taiyuan, China
- 6Lanzhou University Second Hospital, Lanzhou, China
- 7The First Hospital Affiliated to AMU, Chongqing, China
- 8Chinese PLA General Hospital, Beijing, China
- 9Tianjin Third Central Hospital, Tianjin, China
- 10First Affiliated Hospital of Zhengzhou University, Zhengzhou, China
- 11Henan Provincial People's Hospital, Zhengzhou, China
- 12Shanghai First People's Hospital, Shanghai, China
- 13Yunnan Cancer Hospital, Kunming, China
- 14First Affiliated Hospital of Soochow University, Soochow, China
- 15West China Hospital of Sichuan University, Sihchuan, China
- 16Xiangya Hospital Central South University, Changsha, China
- 17Second Affiliated Hospital of Zhejiang University, Hangzhou, China
- 18Sichuan Academy of Medical Sciences & Sichuan Provincial People's Hospital, Chengdu, China
- 19Peking University People's Hospital, Beijing, China
- 20The Affiliated First Hospital of Ningbo University, Ningbo, China
- 21The First Affiliated Hospital, Sun Yat-sen University, Guangzhou, China
Abstract
Background: Despite the increasing use of liver contrast-enhanced ultrasound (CEUS), no standardized, organ-specific tool exists for assessing trainees’ cognitive competency. Existing instruments lack multidimensional evaluation and external validation, which limit training quality and patient safety.
Objective: This study aims to develop and validate the CEUS Competency Assessment for the Liver (CCAL), a tool to standardize the evaluation of theoretical and cognitive skills in liver CEUS.
Methods: This retrospective cohort study had four stages: (1) item generation from literature and guidelines, (2) 3 Delphi rounds (12/21, 57.1% female experts; mean age 51.0, SD 7.2 y) and cognitive testing (10 trainees) for content validity (>85% agreement) and clarity, (3) internal validation involving 100 sonography trainees (n=66, 66% assistant professors; n=27, 27% associate professors; n=7, 7% professors; n=70, 70% CEUS-naive) with 1 month or more of training at 4 hospitals (2018‐2020), and (4) external validation on 579 sonography trainees (n=299, 51.6% assistant professors; n=183, 31.6% associate professors; n=97, 16.8% professors; n=439, 75.8% CEUS-naive) with 1 or more months of training at 25 multicenter sites (2018‐2020). Primary outcomes were reliability (Cronbach α and Guttman split-half) and validity (exploratory factor analysis and item discrimination), while secondary outcomes included item difficulty distribution, pass rates, and between-group differences.
Results: The final CCAL comprised 74 items across 3 domains (demographics: 8, self-evaluation: 13, and objective questions: 53). Internal consistency was excellent (Cronbach α=0.916 and Guttman split-half=0.908). Factor analysis confirmed a 3-factor structure explaining 74.6% of the cumulative variance. Item analysis showed a balanced difficulty distribution (35/53, 66.0% easy; 17/53, 32.1% moderate; 1/53, 1.9% difficult items). Scores varied significantly by CEUS training duration (P<.001); trainees with 4 to 6 months or more than 6 months of training achieved higher self-evaluation (F2,576=34.751, partial η²=0.108, 95% CI 0.069-0.147) and objective question (F2,576=34.374, partial η²=0.107, 95% CI 0.068-0.146) scores than those with less than 3 months of training. Prior independent CEUS experience was also associated with higher scores on the self-evaluation (t577=6.736, Cohen d=0.561, 95% CI 0.393-0.728; P<.001) and objective question sections (t577=3.494, Cohen d=0.291, 95% CI 0.127-0.454; P=.001). Competency gaps were identified across specific competency domains: the pass rate ranged from 32.5% (188/579; image-guided intervention) to 81.3% (471/579; indications and contraindications).
Conclusions: The CCAL is the first liver-specific CEUS competency tool integrating objective knowledge testing with structured self-evaluation. Unlike generic or simulation-based tests, it is organ-specific and validated across a large multicenter cohort, providing a multidimensional framework that identifies specific competency gaps, most notably in advanced image-guided procedures, beyond global pass/fail outcomes. The tool supports competency-based medical education through rigorous data-driven assessment, targeted curricular improvements, cross-institutional benchmarking, and informed training standards. It evaluates cognitive competence exclusively and should not replace procedural assessment in high-stakes decisions. International validation and the establishment of criterion-related and predictive validity are essential next steps.
Related
- A lightweight portable ultrasound robot for autonomous liver sonography
- A competency-based assessment system for undergraduate oral and maxillofacial surgery internship: development and pilot validation.
- The Need for Device-Specific Training Before Handheld Ultrasound Use: Agreement and Repeatability of Simulated Aortic Aneurysm Diameter Measurements by Senior Sonography Students on a Familiar Conventional and an Unfamiliar Handheld System
- WFUMB-Recommended Versus Manufacturer-Optimized Protocols for Ultrasound-Guided Attenuation Parameter in Hepatic Steatosis Assessment: A Retrospective Single-Vendor Multicenter Study
- Assessment in competency-based medical education for anesthesiology: validity evidence, equity, and faculty readiness
BibTeX
@article{ma2026competency,
title = {{A Competency Assessment Tool for Liver Contrast-Enhanced Ultrasound Among Sonography Trainees: Development and Multicenter Validation Study}},
author = {Yanping Ma and Bowen Zheng and Jie Ren and Wen He and Lei Shang and Hui Wang and Gaiqin Xue and Fang Nie and Yanli Guo and Yukun Luo and Hongxia Zhang and Xuan Wang and Xiang Jing and Ruifang Zhang and Jianjun Yuan and Rong Wu and Xiaomao Luo and Fenglin Dong and Yan Luo and Qiang Lu and Jintang Liao and Pintong Huang and Qin Chen and Jia’an Zhu and Youfeng Xu and Xiaoyan Xie},
journal = {JMIR Medical Education},
year = {2026},
month = nov,
doi = {10.2196/86730},
url = {https://doi.org/10.2196/86730},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - A Competency Assessment Tool for Liver Contrast-Enhanced Ultrasound Among Sonography Trainees: Development and Multicenter Validation Study
AU - Yanping Ma
AU - Bowen Zheng
AU - Jie Ren
AU - Wen He
AU - Lei Shang
AU - Hui Wang
AU - Gaiqin Xue
AU - Fang Nie
AU - Yanli Guo
AU - Yukun Luo
AU - Hongxia Zhang
AU - Xuan Wang
AU - Xiang Jing
AU - Ruifang Zhang
AU - Jianjun Yuan
AU - Rong Wu
AU - Xiaomao Luo
AU - Fenglin Dong
AU - Yan Luo
AU - Qiang Lu
AU - Jintang Liao
AU - Pintong Huang
AU - Qin Chen
AU - Jia’an Zhu
AU - Youfeng Xu
AU - Xiaoyan Xie
PY - 2026
DA - 2026/11/09
JO - JMIR Medical Education
DO - 10.2196/86730
UR - https://doi.org/10.2196/86730
AB - Background: Despite the increasing use of liver contrast-enhanced ultrasound (CEUS), no standardized, organ-specific tool exists for assessing trainees’ cognitive competency. Existing instruments lack multidimensional evaluation and external validation, which limit training quality and patient safety. Objective: This study aims to develop and validate the CEUS Competency Assessment for the Liver (CCAL), a tool to standardize the evaluation of theoretical and cognitive skills in liver CEUS. Methods: This retrospective cohort study had four stages: (1) item generation from literature and guidelines, (2) 3 Delphi rounds (12/21, 57.1% female experts; mean age 51.0, SD 7.2 y) and cognitive testing (10 trainees) for content validity (>85% agreement) and clarity, (3) internal validation involving 100 sonography trainees (n=66, 66% assistant professors; n=27, 27% associate professors; n=7, 7% professors; n=70, 70% CEUS-naive) with 1 month or more of training at 4 hospitals (2018‐2020), and (4) external validation on 579 sonography trainees (n=299, 51.6% assistant professors; n=183, 31.6% associate professors; n=97, 16.8% professors; n=439, 75.8% CEUS-naive) with 1 or more months of training at 25 multicenter sites (2018‐2020). Primary outcomes were reliability (Cronbach α and Guttman split-half) and validity (exploratory factor analysis and item discrimination), while secondary outcomes included item difficulty distribution, pass rates, and between-group differences. Results: The final CCAL comprised 74 items across 3 domains (demographics: 8, self-evaluation: 13, and objective questions: 53). Internal consistency was excellent (Cronbach α=0.916 and Guttman split-half=0.908). Factor analysis confirmed a 3-factor structure explaining 74.6% of the cumulative variance. Item analysis showed a balanced difficulty distribution (35/53, 66.0% easy; 17/53, 32.1% moderate; 1/53, 1.9% difficult items). Scores varied significantly by CEUS training duration (P<.001); trainees with 4 to 6 months or more than 6 months of training achieved higher self-evaluation (F2,576=34.751, partial η²=0.108, 95% CI 0.069-0.147) and objective question (F2,576=34.374, partial η²=0.107, 95% CI 0.068-0.146) scores than those with less than 3 months of training. Prior independent CEUS experience was also associated with higher scores on the self-evaluation (t577=6.736, Cohen d=0.561, 95% CI 0.393-0.728; P<.001) and objective question sections (t577=3.494, Cohen d=0.291, 95% CI 0.127-0.454; P=.001). Competency gaps were identified across specific competency domains: the pass rate ranged from 32.5% (188/579; image-guided intervention) to 81.3% (471/579; indications and contraindications). Conclusions: The CCAL is the first liver-specific CEUS competency tool integrating objective knowledge testing with structured self-evaluation. Unlike generic or simulation-based tests, it is organ-specific and validated across a large multicenter cohort, providing a multidimensional framework that identifies specific competency gaps, most notably in advanced image-guided procedures, beyond global pass/fail outcomes. The tool supports competency-based medical education through rigorous data-driven assessment, targeted curricular improvements, cross-institutional benchmarking, and informed training standards. It evaluates cognitive competence exclusively and should not replace procedural assessment in high-stakes decisions. International validation and the establishment of criterion-related and predictive validity are essential next steps.
SN - 2369-3762
ER -