WebXR-based mobile point cloud rendering and semantic segmentation using depth data and voxel fusion
Abstract
A novel framework was developed for interactive, on-device visualization and user-triggered semantic segmentation of point clouds on mobile devices by integrating WebXR, THREE.js, and WebGL. Despite advances in mobile hardware, these tasks remain computationally intensive. The proposed framework leverages the WebXR API to acquire camera pose and depth data for efficient point cloud generation using BufferGeometry, followed by two-dimensional image recognition. A dynamic voxel fusion buffer was developed to construct voxel-based point clouds by aggregating semantic segmentation results from multiple frames. Segmentation results are further refined using a buffer voxel-counting algorithm and rendered in real time within a web browser using ShaderMaterial. Shaders are used to visualize public indoor datasets and convert them into depth maps, creating data compliant with WebXR constraints. The framework was trained and evaluated using the Stanford 3D Indoor Scene Dataset (S3DIS), and a lightweight 2D semantic segmentation model—trained on annotated real-world indoor scenes—achieved 89% accuracy and a mean Intersection over Union of 76%. The system was deployed and demonstrated in indoor scenes on a WebXR-capable mobile device. Under the tested configuration, the framework supports interactive visualization and user-triggered semantic segmentation; comprehensive latency, resource-use, cross-device, and environmental-robustness benchmarks remain subjects for future work.
Related
- Compressing Semantic Segmentation Models for Agricultural IoT: A Systematic Study of Pruning Trade-Offs for Crop-Weed Segmentation.
- Boundary-aware multi-scale residual learning with hybrid radiomic–deep fusion for lung tumor segmentation and classification
- AMF-U-Net: an adaptive multimodal fusion residual attention 3D U-Net for boundary-aware brain tumour segmentation
- Dual-stream 1D mobile-net with channel attention fusion for rolling element fault diagnosis
- UAV image segmentation in complex mining areas by integrating spatially adaptive feature modulation and multi-scale context fusion modules
BibTeX
@article{yang2026webxrbased,
title = {{WebXR-based mobile point cloud rendering and semantic segmentation using depth data and voxel fusion}},
author = {Ming-Der Yang and Chun-Wei Yeh and Hui Ping Tsai},
journal = {Scientific Reports},
year = {2026},
month = oct,
doi = {10.1038/s41598-026-74609-x},
url = {https://www.nature.com/articles/s41598-026-74609-x},
note = {CC BY-NC-ND 4.0},
}
RIS
TY - JOUR
TI - WebXR-based mobile point cloud rendering and semantic segmentation using depth data and voxel fusion
AU - Ming-Der Yang
AU - Chun-Wei Yeh
AU - Hui Ping Tsai
PY - 2026
DA - 2026/10/13
JO - Scientific Reports
DO - 10.1038/s41598-026-74609-x
UR - https://www.nature.com/articles/s41598-026-74609-x
AB - A novel framework was developed for interactive, on-device visualization and user-triggered semantic segmentation of point clouds on mobile devices by integrating WebXR, THREE.js, and WebGL. Despite advances in mobile hardware, these tasks remain computationally intensive. The proposed framework leverages the WebXR API to acquire camera pose and depth data for efficient point cloud generation using BufferGeometry, followed by two-dimensional image recognition. A dynamic voxel fusion buffer was developed to construct voxel-based point clouds by aggregating semantic segmentation results from multiple frames. Segmentation results are further refined using a buffer voxel-counting algorithm and rendered in real time within a web browser using ShaderMaterial. Shaders are used to visualize public indoor datasets and convert them into depth maps, creating data compliant with WebXR constraints. The framework was trained and evaluated using the Stanford 3D Indoor Scene Dataset (S3DIS), and a lightweight 2D semantic segmentation model—trained on annotated real-world indoor scenes—achieved 89% accuracy and a mean Intersection over Union of 76%. The system was deployed and demonstrated in indoor scenes on a WebXR-capable mobile device. Under the tested configuration, the framework supports interactive visualization and user-triggered semantic segmentation; comprehensive latency, resource-use, cross-device, and environmental-robustness benchmarks remain subjects for future work.
SN - 2045-2322
ER -