Hybrid rank fusion of a group-token transformer with gradient boosting improves detection of evasive and novel-family malware
Abstract
Static machine-learning malware detectors report near-perfect closed-set accuracy, but the malware that matters operationally is the malware that slips past detection and the malware that belongs to families never seen in training. We study both on three longitudinal benchmarks: EMBER2024 (3.2 M Windows, .NET, APK, ELF and PDF files, 52 training weeks and 12 test weeks, plus a challenge set of 6,315 files that evaded every antivirus engine at first sight), LAMDA (1.0 M Android applications, 2013–2025, with an IID/NEAR/FAR drift protocol) and BODMAS (134 K PE files, 2019–2020, monthly evaluation). We make five contributions. First, an operational evaluation protocol: true-positive rate at 1% and 0.1% false-positive rate, challenge-set detection at fixed FPR, novel-family, singleton and low-consensus strata, week-by-week and year-by-year drift, thresholds calibrated on time-aware validation data, and paired-bootstrap confidence intervals. Second, three new deep architectures for static feature vectors, trained jointly across formats: GT-Former, a Group-Token Transformer over the semantic feature groups, whose hyper-parameters are tuned with Optuna on a time-aware validation window and which is evaluated with five seeds on every benchmark; FR-MoE, a format-routed mixture of experts; and ProtoCon-Net, a prototype classifier with family-aware supervised-contrastive training. Third, three metadata-driven objectives that use antivirus consensus and family labels at training time only (a consensus-aware margin loss, a consensus-regression head and a family-adversarial gradient-reversal head), with full ablations. Fourth, explainability with SHAP and CLS attention. Fifth, benchmark-hygiene findings, including that the public EMBER2024 release lists every record twice. The results are sobering and useful: the EMBER2024 LightGBM benchmark remains the strongest single model at strict operating points (Win32: 90.0% TPR at 0.1% FPR versus 88.1 ± 0.4 % for the tuned GT-Former over five seeds), and the metadata-driven objectives do not pay for themselves: on LAMDA, with five seeds per variant, no objective activated on its own is distinguishable from the plain model, and only their combination degrades the strictest operating point (NEAR TPR at 0.1% FPR falls from 10.1 ± 2.3 % to 4.4 ± 3.7 %). The seed spread on this benchmark is large enough that single-seed ablations are unreliable, which we regard as a methodological finding in itself. The deep models are nevertheless complementary to gradient boosting: a rank average of the tuned GT-Former and LightGBM raises Win32 challenge-set detection from 63.2% to 68.8% at 1% FPR (95% CI +4.7 to +6.6 points), novel-family detection from 89.1% to 91.8% (CI +1.5 to +3.9 ) and test TPR at 1% FPR from 96.8% to 97.3%, with significant challenge-set gains also on .NET ( +1.6 ) and APK ( +3.9 ); on LAMDA a rank average of an MLP and LightGBM raises far-future (2018–2025) detection from 40.5% to 54.4% at 1% FPR (CI +13.7 to +14.0 ) and singleton-family detection from 34.6% to 53.3% when the threshold is calibrated on the benign files of the test region; under a threshold fixed on validation data and applied unchanged, twelve years of drift move both models far below their nominal budget and the advantage falls to 0.1 points (Section Retrospective and deployment operating points), which we regard as the honest deployment figure; and on BODMAS the tuned GT-Former/LightGBM fusion matches LightGBM at 1% FPR (99.85%) and improves detection at 0.1% FPR (98.9% to 99.5%) and novel-family detection at 0.1% FPR (94.7% to 97.1%). SHAP shows why the benchmark misses evasive files: its strongest evidence is structural sloppiness (suspicious section flags, missing signatures), and evasive malware is exactly the malware that avoids it. Code, de-duplication masks, models and all score files are released.
Related
- Dual-layer edge intelligence for secure real-time medical IoT: a Zero-Trust sequential framework integrating soft-voting intrusion detection and Histogram Gradient Boosting
- Construction of a Standardized Time-Lapse Imaging Database and a Gradient Boosting Ensemble Framework for Integrating Zygote Morphokinetic Parameters with Conventional Embryo Assessment
- Gradient boosting models for gallstone disease classification: A comparative study using demographic and clinical data
- Persistent atrial fibrillation in older adults : Comparison of seven models and development of a gradient boosting machine classifier in a single-center observational study
- Predicting documented prior intentional self-harm in emergency departments: an interpretable gradient boosting approach using European injury surveillance data.
BibTeX
@article{tawfik2026hybrid,
title = {{Hybrid rank fusion of a group-token transformer with gradient boosting improves detection of evasive and novel-family malware}},
author = {Mohammed Tawfik and Belal Al-sellami and Mohamed Heshmat and Islam S. Fathi and Warda M. Shaban},
journal = {Scientific Reports},
year = {2026},
month = oct,
doi = {10.1038/s41598-026-75034-w},
url = {https://www.nature.com/articles/s41598-026-75034-w},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - JOUR
TI - Hybrid rank fusion of a group-token transformer with gradient boosting improves detection of evasive and novel-family malware
AU - Mohammed Tawfik
AU - Belal Al-sellami
AU - Mohamed Heshmat
AU - Islam S. Fathi
AU - Warda M. Shaban
PY - 2026
DA - 2026/10/01
JO - Scientific Reports
DO - 10.1038/s41598-026-75034-w
UR - https://www.nature.com/articles/s41598-026-75034-w
AB - Static machine-learning malware detectors report near-perfect closed-set accuracy, but the malware that matters operationally is the malware that slips past detection and the malware that belongs to families never seen in training. We study both on three longitudinal benchmarks: EMBER2024 (3.2 M Windows, .NET, APK, ELF and PDF files, 52 training weeks and 12 test weeks, plus a challenge set of 6,315 files that evaded every antivirus engine at first sight), LAMDA (1.0 M Android applications, 2013–2025, with an IID/NEAR/FAR drift protocol) and BODMAS (134 K PE files, 2019–2020, monthly evaluation). We make five contributions. First, an operational evaluation protocol: true-positive rate at 1% and 0.1% false-positive rate, challenge-set detection at fixed FPR, novel-family, singleton and low-consensus strata, week-by-week and year-by-year drift, thresholds calibrated on time-aware validation data, and paired-bootstrap confidence intervals. Second, three new deep architectures for static feature vectors, trained jointly across formats: GT-Former, a Group-Token Transformer over the semantic feature groups, whose hyper-parameters are tuned with Optuna on a time-aware validation window and which is evaluated with five seeds on every benchmark; FR-MoE, a format-routed mixture of experts; and ProtoCon-Net, a prototype classifier with family-aware supervised-contrastive training. Third, three metadata-driven objectives that use antivirus consensus and family labels at training time only (a consensus-aware margin loss, a consensus-regression head and a family-adversarial gradient-reversal head), with full ablations. Fourth, explainability with SHAP and CLS attention. Fifth, benchmark-hygiene findings, including that the public EMBER2024 release lists every record twice. The results are sobering and useful: the EMBER2024 LightGBM benchmark remains the strongest single model at strict operating points (Win32: 90.0% TPR at 0.1% FPR versus 88.1 ± 0.4 % for the tuned GT-Former over five seeds), and the metadata-driven objectives do not pay for themselves: on LAMDA, with five seeds per variant, no objective activated on its own is distinguishable from the plain model, and only their combination degrades the strictest operating point (NEAR TPR at 0.1% FPR falls from 10.1 ± 2.3 % to 4.4 ± 3.7 %). The seed spread on this benchmark is large enough that single-seed ablations are unreliable, which we regard as a methodological finding in itself. The deep models are nevertheless complementary to gradient boosting: a rank average of the tuned GT-Former and LightGBM raises Win32 challenge-set detection from 63.2% to 68.8% at 1% FPR (95% CI +4.7 to +6.6 points), novel-family detection from 89.1% to 91.8% (CI +1.5 to +3.9 ) and test TPR at 1% FPR from 96.8% to 97.3%, with significant challenge-set gains also on .NET ( +1.6 ) and APK ( +3.9 ); on LAMDA a rank average of an MLP and LightGBM raises far-future (2018–2025) detection from 40.5% to 54.4% at 1% FPR (CI +13.7 to +14.0 ) and singleton-family detection from 34.6% to 53.3% when the threshold is calibrated on the benign files of the test region; under a threshold fixed on validation data and applied unchanged, twelve years of drift move both models far below their nominal budget and the advantage falls to 0.1 points (Section Retrospective and deployment operating points), which we regard as the honest deployment figure; and on BODMAS the tuned GT-Former/LightGBM fusion matches LightGBM at 1% FPR (99.85%) and improves detection at 0.1% FPR (98.9% to 99.5%) and novel-family detection at 0.1% FPR (94.7% to 97.1%). SHAP shows why the benchmark misses evasive files: its strongest evidence is structural sloppiness (suspicious section flags, missing signatures), and evasive malware is exactly the malware that avoids it. Code, de-duplication masks, models and all score files are released.
SN - 2045-2322
ER -