Open4DGS
An Open-Source, Production-Ready 4D Gaussian Splatting Pipeline for Volumetric Video and Advanced Spatial Storytelling
Democratizing Spatial Storytelling
Community-driven Innovation in spatial storytelling
For many years, volumetric capture and 4D playback have remained confined to expensive, proprietary studio pipelines. High equipment costs, closed formats, and restrictive licensing have strictly limited who can experiment, create, and innovate with spatial media.
Developed over the past year at Arizona State University’s Narrative and Emerging Media (NEM) Program, Open4DGS is released today as a complete, integrated, and production-ready open-source pipeline. It is engineered to make advanced spatial storytelling using real human performance accessible to universities, independent creators, immersive theatre groups, and research laboratories worldwide.
Importantly, Open4DGS eliminates the requirement for gen-locked, global-shutter camera hardware. Through automated pre-processing, our visual timing solver directly learns continuous sub-frame synchronization offsets and models rolling shutter readout. This enables commodity multi-camera arrays—such as GoPros, Sony action cameras, or mixed consumer setups—to be turned into high-fidelity volumetric capture stages.
Our Open Ecosystem Mission
"The goal is not simply to release software, but to help build an open ecosystem for spatial media. By sharing complete workflows and technical developments, we hope to accelerate research, encourage experimentation, and empower a wider community to build upon this work. Open-source development is essential to ensuring that the future of immersive storytelling and spatial entertainment isn't shaped exclusively by private companies, but through collaboration, shared knowledge, and collective innovation."
Volumetric Results & Performance Demos
High-fidelity free-viewpoint human performance reconstructions trained with Open4DGS
Trained with speed-aware sub-keyframes and weighted fast-frame visits. Preserves intricate limb extensions and fabric motion without temporal ghosting.
Coupled $\mathfrak{se}(3)$ rigid-body motion with Bézier residuals tracks rapid rotations while gauge-fixed anchors prevent translation-rotation float.
Captured with consumer Sony RX0 II units. The solver learned per-camera sub-millisecond offsets and calibrated the 15 ms line-by-line rolling shutter distortion.
Dense spatial RoMa v2 matches combined with multi-hop MEMFOF optical flow tracking yield razor-sharp geometry across all viewing angles.
Sub-Frame Multi-Camera Synchronization & Rolling Shutter Compensation
No gen-lock required: turning affordable consumer action cameras into precision volumetric capture stages.
The Hardware Barrier
Traditional volumetric stages rely on broadcast-grade global-shutter cameras locked to a hardware genlock generator. This single requirement raises entry costs into the tens of thousands of dollars, locking out independent studios, educational institutions, and creative labs.
The Open4DGS Solution
Our pre-processing pipeline introduces an automated visual phase solver (calculate-sync.py) that learns continuous camera timing offsets directly from scene dynamics:
Where $\Delta t_{\text{cam}}$ is the learned sub-frame phase offset, and the rolling shutter readout term accounts for scanline index $y$ across frame height $H$.
- Direct Epipolar Phase Estimation: Tracks spatial epipolar flow across calibrated camera pairs to resolve sub-millisecond inter-camera phase discrepancies.
- Rolling Shutter Modeling: Accounts for row-by-row sensor readout (e.g. 15.0 ms for Sony RX0 II at 60 fps), eliminating the severe warp artifacts caused by fast-moving limbs across rolling shutter scanlines.
- Cycle-Consistent Graph Validation: Performs strict closed-loop graph checks. When confidence criteria fail, the system safely falls back to a timecode-only baseline.
Multi-camera sub-millisecond phase alignment on fast dynamic athletic movement
Sub-millisecond alignment: razor-sharp paddle boundaries, fast ball motion, and solid player silhouettes without rolling shutter tear.
Mastering Fast-Moving Motion: Speed-Aware Sub-Keyframes & Weighted Sampling
Overcoming the fundamental breakdown of dynamic Gaussian splats during rapid human acceleration, dance moves, and acrobatics.
1. Speed-Aware Sub-Keyframes
In conventional 4DGS pipelines, keyframes are sampled sparsely (e.g., every 5 frames). When a performer moves rapidly—such as a swinging arm, kick, or pirouette—Gaussians are forced to extrapolate linearly across large temporal gaps, creating distorted "splat sprays" and smeared textures.
Our Solution: Open4DGS combines dense RoMa v2 spatial matching with MEMFOF optical flow to compute per-point velocities. Points with speed exceeding --subkeyframe-min-speed 4.0 spacings/frame trigger automatic intermediate 1-frame sub-keyframes with localized temporal support ($\sigma_t$). Fast parts receive tightly bounded lifetimes instead of lingering erroneously across neighbouring frames.
2. Adaptive Weighted Frame Sampling
Because sub-keyframe splats possess ultra-short lifetimes (1-frame temporal windows), they only receive optimization gradients when the training loop randomly selects their specific frame. Under uniform sampling, fast frames are severely starved of learning steps relative to static backgrounds.
Our Solution: The Open4DGS trainer implements an adaptive fast-frame visit top-up (--config.frame-sampling-fast-visits 387). Frames housing short-lived primitives are dynamically topped up from standard visits (~98 visits) to 387 visits. Slower frames retain their full training budget without distortion.
Frames with fast motion ($n_t$) are adaptively topped up from 98 standard visits toward 387 visits.
Empirical evaluation on fast dance sequence apt-dance-01 (Frames 66–126, 27 cameras at 60 fps)
Adaptive top-up result on fast dance take: crisp moving contours, articulated fingers, and solid silhouettes (+2.65 dB PSNR on fast hand).
Why Uniform Sampling Fails on Fast Motion:
Short-lived sub-keyframe splats (1-frame temporal width) only receive gradient updates on steps that randomly draw a camera inside their fleeting window. Under uniform sampling, fast frames are visited only ~98 times across 6,000 steps—starving Adam moments.
Our additive fast-frame top-up raises visits on fast frames to 387, allocating extra gradient steps precisely where velocity peaks without disturbing the static scene.
Benchmark Metrics: Uniform Baseline vs. Adaptive Fast-Frame Top-Up
| Motion Group | Evaluated Views | PSNR Uniform Baseline | PSNR Adaptive Top-Up | Δ PSNR | Win Rate | Δ SSIM (↑) | Δ LPIPS (↓) |
|---|---|---|---|---|---|---|---|
| Fast Frames (81–102) | 108 | 31.69 dB | 34.18 dB | +2.50 dB | 108/108 (100%, p=6.2e-33) | +0.0162 | -0.0354 |
| • Fast Hand Motion (86–96) | 66 | 31.59 dB | 34.23 dB | +2.65 dB | 66/66 (100%, p=2.7e-20) | +0.0174 | -0.0372 |
| • Tight Hand Box Crop (86–96) | 66 | 29.73 dB | 31.54 dB | +1.81 dB | 64/66 (97%, p=6.0e-17) | +0.0151 | -0.0318 |
| • Mid-Stride Hand (93–95) | 18 | 29.06 dB | 30.72 dB | +1.66 dB | 18/18 (100%, p=7.6e-06) | +0.0142 | -0.0295 |
| • Slowing Deceleration (97–98) | 12 | 32.21 dB | 34.68 dB | +2.47 dB | 12/12 (100%, p=4.9e-04) | +0.0168 | -0.0362 |
| Before Fast Motion (66–80) | 36 | 31.56 dB | 33.02 dB | +1.46 dB | 34/36 (94%) | +0.0103 | -0.0254 |
| After Fast Motion (103–126) | 42 | 31.68 dB | 33.39 dB | +1.71 dB | 40/42 (95%) | +0.0092 | -0.0260 |
* Evaluated across 6 held-out camera viewpoints on fast dance sequence apt-dance-01 (frames 66–126). Non-fast frames preserved 86% median splat retention relative to uniform baseline, verifying capacity relocation without background degradation.
High-Resolution Crop Sheets: Ground Truth vs. Uniform vs. Weighted Top-Up
Sheet Breakdown: Rows 1-3 (Cam 12), Rows 4-6 (Cam 14), Rows 7-9 (Cam 01) across fast frames 86–96. Each group displays Ground Truth Photo (top), Uniform Baseline (middle), and Adaptive Weighted Sampling (bottom). Notice how fingers and knuckles remain distinct instead of dissolving into blur.
Open Full Res (1350x1710)Integrated Pipeline Innovations
Built on solid mathematical foundations for scalable, high-fidelity dynamic splatting
Coupled $\mathfrak{se}(3)$ & Bézier
Per-primitive translation and rotation are optimized jointly as a single Lie-algebra twist $\zeta_i(t) = (\omega_i, \nu_i) \in \mathfrak{se}(3)$ and mapped via closed-form exponential map. Local quadratic Bézier control points model nonlinear paths without rotational ambiguity.
Gauge-Fixed Anchors
Each primitive carries a learnable local anchor $a_i$ acting as its rotation pivot. We canonicalize $a_i \perp \omega_i$ (orthogonal to the rotation axis) and constrain translation to $p_{\parallel}$, fixing the gauge ambiguity and ensuring rock-solid numerical stability.
Fixed Canonical Set
Instead of runaway densification that explodes VRAM, Open4DGS fixes the canonical primitive count. Low-opacity slots are recycled into active regions weighted by a 4-term normalized score: opacity, motion magnitude, 2D gradient, and temporal loss gradient.
The End-to-End Open4DGS Pipeline
From raw multi-camera video to interactive spatial playback in 7 transparent stages.
Extract Frames & Timecode Alignment prep/extract/
Extract synchronized video sequences across camera subfolders using common timecode references. Organizes takes automatically with orientation correction.
Multi-Camera Calibration & Sparse Alignment prep/align/
Establishes unified scene metric coordinate space, camera intrinsics, and extrinsics via automated COLMAP model generation. Target reprojection error below 1 pixel.
Foreground Mask Extraction (BGMv2 / SAM 2) prep/mask/
Generates high-precision alpha mattes with clean background plate subtraction (Background Matting V2) or promptable SAM 2 segmentation when clean plates are unavailable.
Sub-Frame Timing Offset & Rolling Shutter Solver prep/sync/
Learns continuous per-camera phase offsets and compensates for sensor line readout time (e.g. 15 ms for Sony RX0 II). Replaces costly hardware genlock.
RoMa v2 Spatial Matching + MEMFOF Temporal Tracking prep/init/
Computes dense cross-camera 3D points, tracks dynamic correspondences across frames with optical flow, detects high-velocity regions, and injects speed-aware sub-keyframes.
Fast-Motion Aware 4DGS Training train/trigs/
Trains continuous $\mathfrak{se}(3)$ trajectories with adaptive fast-frame visit top-up (387 visits). Supports temporal chunking for arbitrarily long captures.
Timeline Delivery & Interactive Viewer deliver/ & viewer.py
Exports lightweight streaming timeline files (output.trigsbin) and dense PLY sequences. Watch runs in real-time with zero-restart viewer snapshots.
Real-Time Interactive 4D Viewer
Open4DGS includes an interactive viewer (src/trigs/core/viewer.py) that watches training in real time. It streams lightweight parameter snapshots every 1,000 steps without restarting the training loop, allowing directors and researchers to inspect reconstructions live from any perspective.
- Live Snapshot Monitoring: Watch convergence progress in real-time across arbitrary camera orbits.
- Timeline Scrubbing: Fluid interactive playback with variable speed, forward/backward looping, and timeline scrub.
- Production Formats: Seamless export to
.trigsbinstreaming timelines and standard per-frame PLY packages for Blender, Unreal Engine, and WebGL viewers.
Live interactive orbit during a 60 FPS dance take reconstruction.
Additional Capture Results & Angles
Multi-view reconstructions across diverse motion speeds and camera layouts
Dance Take 01 (apt-dance-01): High-speed pirouettes reconstructed with sub-frame sync and 1-frame sub-keyframes.
Acrobatic Turning (maurine-4d_v01): Rapid body rotation with continuous SE(3) trajectory and zero ghosting.
Rapid Prop Juggling (jet-juggle-4d): High-velocity airborne props and hand coordination tracked cleanly via adaptive weighted visits.
Bubble Dynamics (bubbles-4d): Delicate translucent surfaces, floating particles, and specular reflections captured in full 4D.
Multi-Performer Interaction (two-spidermen-4d): Dual-subject acrobatics with mutual occlusions, rapid contact, and synchronized poses.
Athletic Pickleball (pickleball-4d): Explosive paddle swings, ball trajectories, and court footwork without motion smearing.
Free-Viewpoint Orbit: Continuous novel-view camera flight through time-varying Gaussian splats with full temporal coherence.
BibTeX & Citations
If you use Open4DGS in your research or spatial productions, please cite our open-source release alongside the foundational TRiGS and Gaussian Splatting works:
@misc{open4dgs2026asu,
title = {Open4DGS: Open-Source Production-Ready 4D Gaussian Splatting Pipeline for Volumetric Video},
author = {Narrative and Emerging Media Program, Arizona State University},
year = {2026},
url = {https://github.com/ASU-NEM-Labs},
note = {Open-source spatial storytelling initiative}
}
@article{yeom2026trigs,
title = {TRiGS: Temporal Rigid-Body Motion for Scalable 4D Gaussian Splatting},
author = {Yeom, Suwoong and Nam, Joonsik and Choi, Seunggyu and Lee, Lucas Yunkyu and Kim, Sangmin and Park, Jaesik and Kim, Joonsoo and Yun, Kugjin and Kong, Kyeongbo and Kang, Sukju},
journal = {arXiv preprint arXiv:2604.00538},
year = {2026},
url = {https://wwwjjn.github.io/TRiGS-project_page/}
}
@article{kerbl20233d,
title = {3D Gaussian Splatting for Real-Time Radiance Field Rendering},
author = {Kerbl, Bernhard and Kopanas, Georgios and Leimk{\"u}hler, Thomas and Drettakis, George},
journal = {ACM Transactions on Graphics},
volume = {42},
number = {4},
year = {2023}
}
Foundational Credits & Open-Source Lineage:
Open4DGS builds on the brilliant research of TriGSVanilla and TRiGS (Yeom et al.), RoMa v2 (Edstedt et al.), MEMFOF (MSU Video Group), SAM 2 (Meta AI), Background Matting V2 (Lin et al.), gsplat (Ye et al.), and COLMAP (Schönberger et al.).