RabbitCD-HIT: Distributed High-Sensitivity Greedy Incremental Clustering for Billion-Scale Protein Sequences…

Abstract

We present RabbitCD-HIT, a scalable distributed high-sensitivity greedy incremental clustering tool for ultra-large protein sequence datasets. It fully exploits cluster resources to enable efficient billion-scale clustering. RabbitCD-HIT clustered the March 10, 2026 NCBI nr snapshot (1.02 billion sequences; 487 GB FASTA) at 90\% identity within 5 hours on 64 nodes; the resulting representatives and cluster assignments are released as a public database resource. On a 256-core server, RabbitCD-HIT achieved a 2.1×--6.6× speedup over CD-HIT with identical results under matched settings. Our approach addresses performance bottlenecks of traditional greedy incremental algorithms and provides an effective solution for large-scale protein sequence clustering.

Differential sensitivity of k-mer based identification tools using closely related database sequences affects relative abundance estimation
A. J. Forster, M. Payne, R. J. Rockett +2
bioRxiv15 Oct 2026PreprintCC BYFull textBioinformatics & computational biologybioRxiv
Clinical-Anchored Incremental Survival Learning: A Stability-Constrained Framework for Molecularly Enhanced Survival Prediction in Lung Adenocarcinoma
P. Shringarpure, S. Divakar, S. Khobragade
bioRxiv4 Nov 2026PreprintAll rights reservedFull textBioinformatics & computational biologybioRxiv
ACOR: incremental evidence accumulation towards real-time DNA storage reconstruction
Linxuan Han, Huaming Wu, Shi Jia +1
arXiv27 Oct 2026PreprintCC BY-NC-SA 4.0Full textGenetics & genomicsarXiv
CADS: Measuring the Incremental Contribution of LLM Proposers in Sequential Biological Screens
Guanxing Chen, Nihui Shao, Haohuai He +3
arXiv17 Oct 2026PreprintarXiv licenceFull textBioinformatics & computational biologyarXiv
External Evaluation of Multi-Omics Prognostic Models Across TCGA-BRCA and METABRIC: Transportability, Stability, and Incremental Performance
E. Krikun, A. Alkhateeb
bioRxiv18 Sep 2026PreprintCC BY-NC-NDFull textBioinformatics & computational biologybioRxiv

Cite

BibTeX RIS

BibTeX

@misc{gao2026rabbitcdhit,
  title = {{RabbitCD-HIT: Distributed High-Sensitivity Greedy Incremental Clustering for Billion-Scale Protein Sequences in Hours}},
  author = {Y. Gao and G. Ma and Z. Yin and L. Yan and Y. Yang and T. Zhang and T. Zhao and Z. Liu and X. Duan and B. Niu and B. Schmibt and W. Liu},
  howpublished = {bioRxiv},
  year = {2026},
  month = nov,
  doi = {10.64898/2026.11.05.764756},
  url = {https://www.biorxiv.org/content/10.64898/2026.11.05.764756v1},
  note = {CC BY-NC-ND},
}

RIS

TY  - UNPB
TI  - RabbitCD-HIT: Distributed High-Sensitivity Greedy Incremental Clustering for Billion-Scale Protein Sequences in Hours
AU  - Y. Gao
AU  - G. Ma
AU  - Z. Yin
AU  - L. Yan
AU  - Y. Yang
AU  - T. Zhang
AU  - T. Zhao
AU  - Z. Liu
AU  - X. Duan
AU  - B. Niu
AU  - B. Schmibt
AU  - W. Liu
PY  - 2026
DA  - 2026/11/05
PB  - bioRxiv
DO  - 10.64898/2026.11.05.764756
UR  - https://www.biorxiv.org/content/10.64898/2026.11.05.764756v1
AB  - We present RabbitCD-HIT, a scalable distributed high-sensitivity greedy incremental clustering tool for ultra-large protein sequence datasets. It fully exploits cluster resources to enable efficient billion-scale clustering. RabbitCD-HIT clustered the March 10, 2026 NCBI nr snapshot (1.02 billion sequences; 487 GB FASTA) at 90\% identity within 5 hours on 64 nodes; the resulting representatives and cluster assignments are released as a public database resource. On a 256-core server, RabbitCD-HIT achieved a 2.1×--6.6× speedup over CD-HIT with identical results under matched settings. Our approach addresses performance bottlenecks of traditional greedy incremental algorithms and provides an effective solution for large-scale protein sequence clustering.
ER  -

Abstract

Related

Cite