A segmentation-guided CNN–Vision transformer feature fusion framework for multi-class breast ultrasound image classification
- 1Department of Ultrasound, The Fifth People’s Hospital of Shanxi Province, Shanxi Provincial Geriatric Hospital, Taiyuan, Shanxi, China
- 2Department of Breast Surgery, Jinan Maternity and Child Care Hospital Affiliated to Shandong First Medical University, Jinan, ShanDong, China
Abstract
Breast ultrasound (BU) imaging is widely used for detecting breast abnormalities because it is cost-effective, non-invasive, and suitable for dense breast tissue. However, multi-class classification of BU images is considered a challenging task due to low contrast, speckle noise, and overlapping visual patterns between benign and malignant tumours. To address this issue, we develop a segmentation-guided CNN and Vision Transformer based feature fusion framework for efficient multi-class BU image classification. The framework first applies a lesion segmentation model to identify the region of interest by using Fusion-Enhanced Transformer (FET) Unet model. The FET Unet model uses CNNs and Swin Transformers integrated and constructs a UNet-like architecture. In the second step, CNN-based and Vision Transformer-based features are extracted from the segmented lesion regions. In the third step, the CNN and Vision Transformer features are fused to generate a robust feature representation. A deep neural network classifier comprising four dense layers with 1,024, 512, 256, and 128 neurons, respectively, followed by a softmax output layer, is then developed using the fused features to classify breast ultrasound images into benign, malignant, and normal categories. The proposed framework was evaluated using the publicly available Breast Ultrasound Images (BUSI) dataset, which contains 780 ultrasound images collected from women aged 25–75 years, including 437 benign, 210 malignant, and 133 normal cases. Numerical results show that the proposed framework showed 95.11% of accuracy, 95.66% of sensitivity and 97.63% of specificity and F1 score of 0.944644. Based on the classification accuracy, the effectiveness of the proposed hybrid framework is demonstrated in comparison with previously reported methods for multi-class breast ultrasound image classification.
Related
- Multi-task deep learning framework for brain tumor classification and segmentation using ensemble feature extraction
- ReViT-RF: A hybrid ResNet-50–Vision Transformer with a Radiomic Feature Fusion framework for lung tumor classification and TNM-based stage prediction using CT images
- MTF-Net: a brain tumor segmentation network with multi-scale transformer and feature fusion enhancement
- A privacy-preserving multicenter framework for lung nodule segmentation in low-dose CT using an F1-score-weighted federated 3D Transformer-UNet
- Boundary-interior ratio: a spatially decomposed evaluation framework for CNN and mamba-based breast ultrasound segmentation
BibTeX
@article{wu2026segmentationguided,
title = {{A segmentation-guided CNN–Vision transformer feature fusion framework for multi-class breast ultrasound image classification}},
author = {Meiru Wu and Jian Wang},
journal = {PLOS One},
year = {2026},
month = sep,
doi = {10.1371/journal.pone.0353628},
url = {https://doi.org/10.1371/journal.pone.0353628},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - A segmentation-guided CNN–Vision transformer feature fusion framework for multi-class breast ultrasound image classification
AU - Meiru Wu
AU - Jian Wang
PY - 2026
DA - 2026/09/17
JO - PLOS One
DO - 10.1371/journal.pone.0353628
UR - https://doi.org/10.1371/journal.pone.0353628
AB - Breast ultrasound (BU) imaging is widely used for detecting breast abnormalities because it is cost-effective, non-invasive, and suitable for dense breast tissue. However, multi-class classification of BU images is considered a challenging task due to low contrast, speckle noise, and overlapping visual patterns between benign and malignant tumours. To address this issue, we develop a segmentation-guided CNN and Vision Transformer based feature fusion framework for efficient multi-class BU image classification. The framework first applies a lesion segmentation model to identify the region of interest by using Fusion-Enhanced Transformer (FET) Unet model. The FET Unet model uses CNNs and Swin Transformers integrated and constructs a UNet-like architecture. In the second step, CNN-based and Vision Transformer-based features are extracted from the segmented lesion regions. In the third step, the CNN and Vision Transformer features are fused to generate a robust feature representation. A deep neural network classifier comprising four dense layers with 1,024, 512, 256, and 128 neurons, respectively, followed by a softmax output layer, is then developed using the fused features to classify breast ultrasound images into benign, malignant, and normal categories. The proposed framework was evaluated using the publicly available Breast Ultrasound Images (BUSI) dataset, which contains 780 ultrasound images collected from women aged 25–75 years, including 437 benign, 210 malignant, and 133 normal cases. Numerical results show that the proposed framework showed 95.11% of accuracy, 95.66% of sensitivity and 97.63% of specificity and F1 score of 0.944644. Based on the classification accuracy, the effectiveness of the proposed hybrid framework is demonstrated in comparison with previously reported methods for multi-class breast ultrasound image classification.
SN - 1932-6203
ER -