Democratizing planetary-scale analysis: an ultra-lightweight Earth embedding database for accurate and flexible global land monitoring
8 affiliations
- 1Department of Geography, The University of Hong Kong, Hong Kong, China
- 2University of Hong Kong (Pok Fu Lam, Hong Kong)
- 3Pengcheng Laboratory, Shenzhen 518000, China
- 4Ministry of Education Key Laboratory for Earth System Modeling, Department of Earth System Science, Tsinghua University, Beijing 100084, China
- 5Department of Land Surveying and Geo-Informatics, The Hong Kong Polytechnic University, Hong Kong, China
- 6Tsinghua Shenzhen International Graduate School, Shenzhen, China
- 7Department of Architecture, The University of Hong Kong, Hong Kong, China
- 8Institute for Climate and Carbon Neutrality, The University of Hong Kong, Hong Kong, China
Abstract
The rapid evolution of satellite-borne Earth Observation (EO) systems has fundamentally revolutionized terrestrial monitoring, yielding comprehensive petabyte-scale archives. However, the immense computational resources and storage volumes required for global-scale analysis often preclude widespread use by many research teams, hindering broader scientific adoption and the execution of planetary-scale studies. To address these barriers, we present the Embedded Seamless Data (ESD), an ultra-lightweight, 30-m global Earth embedding database spanning the 25-year period from 2000 to 2024. By transforming high-dimensional, multi-sensor observations from the Landsat series (5, 7, 8, and 9) and MODIS Terra into information-dense, quantized latent vectors, ESD distils essential geophysical and semantic features into a unified latent space. Utilizing the ESDNet architecture and Finite Scalar Quantization (FSQ), the dataset achieves a transformative ~340-fold reduction in data volume compared to raw daily archives. This compression allows the entire global land surface for a single year to be encapsulated within approximately 2.4 TB, enabling decadal-scale global analysis on standard local workstations. Rigorous validation demonstrates that ESD maintains high reconstructive fidelity to the original reflectance values across the spectral dimension, achieving a Mean Absolute Error (MAE) of 0.0130 (averaged over six spectral bands, including Blue, Green, Red, NIR, SWIR1, and SWIR2), a Root Mean Square Error (RMSE) of 0.0179, and a Correlation Coefficient (CC) of 0.8543. By condensing the annual phenological cycle into 12 temporal latent steps, the embeddings provide inherent denoising effects and a semantically organized latent space that outperforms raw reflectance data in downstream land-cover classification tasks, achieving a comparable and even higher overall accuracy of 79.74 % than the 76.92 % obtained using raw sensor fusion data on globally distributed land cover sample sets. With robust few-shot learning capabilities and longitudinal consistency across 25 years, the ESD product provides a versatile foundation for democratizing planetary-scale Earth system research and advancing next-generation geospatial artificial intelligence.
Related
- IoT-enabled individual poultry monitoring with lightweight step-counting for health assessment
- A lightweight deep-learning detector for the real-time monitoring of the invasive frogs Rhinella marina and Polypedates leucomystax
- Smartphone-Assisted Aerogel Colorimetric Sensor Array Coupled with Lightweight Deep Learning for Multi-Meat Freshness Monitoring.
- Democratizing Agentic Access to Bioinformatics and Biopharmaceutical Databases and Analyses with BioMCP-TS
- Democratizing three-dimensional surface phenotyping: an open structured-light platform reveals and removes the projection bias in biological imaging
BibTeX
@article{chen2026democratizing,
title = {{Democratizing planetary-scale analysis: an ultra-lightweight Earth embedding database for accurate and flexible global land monitoring}},
author = {Shuang Chen and Jie Wang and Shuai Yuan and Jiayang Li and Yu Xia and yuanhong Liao and Junbo Wei and Jincheng Yuan and Xiaoqing Xu and Xiaolin Zhu and Peng Zhu and Hailong Zhang and Y Zhou and Haohuan Fu and Huabing Huang and Бин Чэн and Fan Dai and Peng Gong},
journal = {Earth system science data},
year = {2026},
month = jul,
doi = {10.5194/essd-18-5375-2026},
url = {https://doi.org/10.5194/essd-18-5375-2026},
note = {CC BY},
}
RIS
TY - JOUR
TI - Democratizing planetary-scale analysis: an ultra-lightweight Earth embedding database for accurate and flexible global land monitoring
AU - Shuang Chen
AU - Jie Wang
AU - Shuai Yuan
AU - Jiayang Li
AU - Yu Xia
AU - yuanhong Liao
AU - Junbo Wei
AU - Jincheng Yuan
AU - Xiaoqing Xu
AU - Xiaolin Zhu
AU - Peng Zhu
AU - Hailong Zhang
AU - Y Zhou
AU - Haohuan Fu
AU - Huabing Huang
AU - Бин Чэн
AU - Fan Dai
AU - Peng Gong
PY - 2026
DA - 2026/07/24
JO - Earth system science data
DO - 10.5194/essd-18-5375-2026
UR - https://doi.org/10.5194/essd-18-5375-2026
AB - The rapid evolution of satellite-borne Earth Observation (EO) systems has fundamentally revolutionized terrestrial monitoring, yielding comprehensive petabyte-scale archives. However, the immense computational resources and storage volumes required for global-scale analysis often preclude widespread use by many research teams, hindering broader scientific adoption and the execution of planetary-scale studies. To address these barriers, we present the Embedded Seamless Data (ESD), an ultra-lightweight, 30-m global Earth embedding database spanning the 25-year period from 2000 to 2024. By transforming high-dimensional, multi-sensor observations from the Landsat series (5, 7, 8, and 9) and MODIS Terra into information-dense, quantized latent vectors, ESD distils essential geophysical and semantic features into a unified latent space. Utilizing the ESDNet architecture and Finite Scalar Quantization (FSQ), the dataset achieves a transformative ~340-fold reduction in data volume compared to raw daily archives. This compression allows the entire global land surface for a single year to be encapsulated within approximately 2.4 TB, enabling decadal-scale global analysis on standard local workstations. Rigorous validation demonstrates that ESD maintains high reconstructive fidelity to the original reflectance values across the spectral dimension, achieving a Mean Absolute Error (MAE) of 0.0130 (averaged over six spectral bands, including Blue, Green, Red, NIR, SWIR1, and SWIR2), a Root Mean Square Error (RMSE) of 0.0179, and a Correlation Coefficient (CC) of 0.8543. By condensing the annual phenological cycle into 12 temporal latent steps, the embeddings provide inherent denoising effects and a semantically organized latent space that outperforms raw reflectance data in downstream land-cover classification tasks, achieving a comparable and even higher overall accuracy of 79.74 % than the 76.92 % obtained using raw sensor fusion data on globally distributed land cover sample sets. With robust few-shot learning capabilities and longitudinal consistency across 25 years, the ESD product provides a versatile foundation for democratizing planetary-scale Earth system research and advancing next-generation geospatial artificial intelligence.
SN - 1866-3508
SN - 1866-3516
ER -