Arizona State University · Narrative and Emerging Media (NEM) Program

Open4DGS

An Open-Source, Production-Ready 4D Gaussian Splatting Pipeline for Volumetric Video and Advanced Spatial Storytelling

Narrative and Emerging Media (NEM)
Arizona State University
Arvin Tehrani
Contrast Media Labs
Dynamic Human Performance · 60 FPS Free-Viewpoint 4D Gaussian Splatting

Open4DGS transforms standard multi-camera arrays into production-ready volumetric spatial video. Overcoming expensive proprietary systems with sub-frame motion synchronization, rolling shutter compensation, and speed-aware sub-keyframe training.

Open Source
No Proprietary Lock-In
Sub-Frame
Learned Visual Sync
+2.5 dB
Fast-Motion PSNR Gain
60 FPS
Real-Time Streaming

Democratizing Spatial Storytelling

Community-driven Innovation in spatial storytelling

For many years, volumetric capture and 4D playback have remained confined to expensive, proprietary studio pipelines. High equipment costs, closed formats, and restrictive licensing have strictly limited who can experiment, create, and innovate with spatial media.

Developed over the past year at Arizona State University’s Narrative and Emerging Media (NEM) Program, Open4DGS is released today as a complete, integrated, and production-ready open-source pipeline. It is engineered to make advanced spatial storytelling using real human performance accessible to universities, independent creators, immersive theatre groups, and research laboratories worldwide.

Importantly, Open4DGS eliminates the requirement for gen-locked, global-shutter camera hardware. Through automated pre-processing, our visual timing solver directly learns continuous sub-frame synchronization offsets and models rolling shutter readout. This enables commodity multi-camera arrays—such as GoPros, Sony action cameras, or mixed consumer setups—to be turned into high-fidelity volumetric capture stages.

Our Open Ecosystem Mission

"The goal is not simply to release software, but to help build an open ecosystem for spatial media. By sharing complete workflows and technical developments, we hope to accelerate research, encourage experimentation, and empower a wider community to build upon this work. Open-source development is essential to ensuring that the future of immersive storytelling and spatial entertainment isn't shaped exclusively by private companies, but through collaboration, shared knowledge, and collective innovation."

Volumetric Results & Performance Demos

High-fidelity free-viewpoint human performance reconstructions trained with Open4DGS

Fast Dynamic Dance Take 01 apt-dance-01
Free-Viewpoint Render

Trained with speed-aware sub-keyframes and weighted fast-frame visits. Preserves intricate limb extensions and fabric motion without temporal ghosting.

Acrobatic Movement & Turning maurine-4d_v01
Novel Trajectory (60 FPS)

Coupled $\mathfrak{se}(3)$ rigid-body motion with Bézier residuals tracks rapid rotations while gauge-fixed anchors prevent translation-rotation float.

Sony RX0 II Rig (15ms Rolling Shutter) Subframe Sync
Compensated Render

Captured with consumer Sony RX0 II units. The solver learned per-camera sub-millisecond offsets and calibrated the 15 ms line-by-line rolling shutter distortion.

Multi-View Consistency RoMa v2 + MEMFOF
Full Capture Orbit

Dense spatial RoMa v2 matches combined with multi-hop MEMFOF optical flow tracking yield razor-sharp geometry across all viewing angles.

HARDWARE DEMOCRATIZATION

Sub-Frame Multi-Camera Synchronization & Rolling Shutter Compensation

No gen-lock required: turning affordable consumer action cameras into precision volumetric capture stages.

The Hardware Barrier

Traditional volumetric stages rely on broadcast-grade global-shutter cameras locked to a hardware genlock generator. This single requirement raises entry costs into the tens of thousands of dollars, locking out independent studios, educational institutions, and creative labs.

The Open4DGS Solution

Our pre-processing pipeline introduces an automated visual phase solver (calculate-sync.py) that learns continuous camera timing offsets directly from scene dynamics:

$$t_{\text{cam}}(y) = t_{\text{nominal}} + \Delta t_{\text{cam}} + \frac{y}{H} \, T_{\text{readout}}$$

Where $\Delta t_{\text{cam}}$ is the learned sub-frame phase offset, and the rolling shutter readout term accounts for scanline index $y$ across frame height $H$.

  • Direct Epipolar Phase Estimation: Tracks spatial epipolar flow across calibrated camera pairs to resolve sub-millisecond inter-camera phase discrepancies.
  • Rolling Shutter Modeling: Accounts for row-by-row sensor readout (e.g. 15.0 ms for Sony RX0 II at 60 fps), eliminating the severe warp artifacts caused by fast-moving limbs across rolling shutter scanlines.
  • Cycle-Consistent Graph Validation: Performs strict closed-loop graph checks. When confidence criteria fail, the system safely falls back to a timecode-only baseline.
Sub-Frame Synchronization Result

Multi-camera sub-millisecond phase alignment on fast dynamic athletic movement

pickleball-4d
Sub-Frame Sync (60 FPS)

Sub-millisecond alignment: razor-sharp paddle boundaries, fast ball motion, and solid player silhouettes without rolling shutter tear.

python prep/sync/calculate-sync.py --fps 60 --rs-readout-ms 15.0 --output-sync sync.json
TRAINER REVOLUTION

Mastering Fast-Moving Motion: Speed-Aware Sub-Keyframes & Weighted Sampling

Overcoming the fundamental breakdown of dynamic Gaussian splats during rapid human acceleration, dance moves, and acrobatics.

1. Speed-Aware Sub-Keyframes

In conventional 4DGS pipelines, keyframes are sampled sparsely (e.g., every 5 frames). When a performer moves rapidly—such as a swinging arm, kick, or pirouette—Gaussians are forced to extrapolate linearly across large temporal gaps, creating distorted "splat sprays" and smeared textures.

Our Solution: Open4DGS combines dense RoMa v2 spatial matching with MEMFOF optical flow to compute per-point velocities. Points with speed exceeding --subkeyframe-min-speed 4.0 spacings/frame trigger automatic intermediate 1-frame sub-keyframes with localized temporal support ($\sigma_t$). Fast parts receive tightly bounded lifetimes instead of lingering erroneously across neighbouring frames.

$$\sigma_{t,i} = \mathrm{clamp}\left( \frac{K d_i}{\|v_i\| + \frac{1}{2}\|a_i\|\Delta t}, \; \sigma_{\min}, \; \sigma_{\max} \right)$$
Zero Splat Splatter on Rapid Movements

2. Adaptive Weighted Frame Sampling

Because sub-keyframe splats possess ultra-short lifetimes (1-frame temporal windows), they only receive optimization gradients when the training loop randomly selects their specific frame. Under uniform sampling, fast frames are severely starved of learning steps relative to static backgrounds.

Our Solution: The Open4DGS trainer implements an adaptive fast-frame visit top-up (--config.frame-sampling-fast-visits 387). Frames housing short-lived primitives are dynamically topped up from standard visits (~98 visits) to 387 visits. Slower frames retain their full training budget without distortion.

$$N_{\text{visits}}(t) = \max\left( 98, \; 387 \cdot \frac{n_t}{n_{\min}} \right)$$

Frames with fast motion ($n_t$) are adaptively topped up from 98 standard visits toward 387 visits.

+2.5 to +2.7 dB PSNR on Fast Frames (+1.7 dB on Still Frames)
Fast Motion Ablation: Uniform vs. Adaptive Weighted Sampling

Empirical evaluation on fast dance sequence apt-dance-01 (Frames 66–126, 27 cameras at 60 fps)

Validated Benchmark: Uniform vs. Adaptive Top-Up
Adaptive Fast-Frame Top-Up Result (387 Visits)

Adaptive top-up result on fast dance take: crisp moving contours, articulated fingers, and solid silhouettes (+2.65 dB PSNR on fast hand).

Why Uniform Sampling Fails on Fast Motion:

Short-lived sub-keyframe splats (1-frame temporal width) only receive gradient updates on steps that randomly draw a camera inside their fleeting window. Under uniform sampling, fast frames are visited only ~98 times across 6,000 steps—starving Adam moments.

Our additive fast-frame top-up raises visits on fast frames to 387, allocating extra gradient steps precisely where velocity peaks without disturbing the static scene.

trigs-train-colmap --config.frame-sampling-fast-visits 387

Benchmark Metrics: Uniform Baseline vs. Adaptive Fast-Frame Top-Up

Motion Group Evaluated Views PSNR Uniform Baseline PSNR Adaptive Top-Up Δ PSNR Win Rate Δ SSIM (↑) Δ LPIPS (↓)
Fast Frames (81–102) 108 31.69 dB 34.18 dB +2.50 dB 108/108 (100%, p=6.2e-33) +0.0162 -0.0354
  • Fast Hand Motion (86–96) 66 31.59 dB 34.23 dB +2.65 dB 66/66 (100%, p=2.7e-20) +0.0174 -0.0372
  • Tight Hand Box Crop (86–96) 66 29.73 dB 31.54 dB +1.81 dB 64/66 (97%, p=6.0e-17) +0.0151 -0.0318
  • Mid-Stride Hand (93–95) 18 29.06 dB 30.72 dB +1.66 dB 18/18 (100%, p=7.6e-06) +0.0142 -0.0295
  • Slowing Deceleration (97–98) 12 32.21 dB 34.68 dB +2.47 dB 12/12 (100%, p=4.9e-04) +0.0168 -0.0362
Before Fast Motion (66–80) 36 31.56 dB 33.02 dB +1.46 dB 34/36 (94%) +0.0103 -0.0254
After Fast Motion (103–126) 42 31.68 dB 33.39 dB +1.71 dB 40/42 (95%) +0.0092 -0.0260

* Evaluated across 6 held-out camera viewpoints on fast dance sequence apt-dance-01 (frames 66–126). Non-fast frames preserved 86% median splat retention relative to uniform baseline, verifying capacity relocation without background degradation.

High-Resolution Crop Sheets: Ground Truth vs. Uniform vs. Weighted Top-Up

Moving Hand Detail Comparison: Photo vs Uniform Sampling vs Adaptive Weighted Sampling

Sheet Breakdown: Rows 1-3 (Cam 12), Rows 4-6 (Cam 14), Rows 7-9 (Cam 01) across fast frames 86–96. Each group displays Ground Truth Photo (top), Uniform Baseline (middle), and Adaptive Weighted Sampling (bottom). Notice how fingers and knuckles remain distinct instead of dissolving into blur.

Open Full Res (1350x1710)

Integrated Pipeline Innovations

Built on solid mathematical foundations for scalable, high-fidelity dynamic splatting

Coupled $\mathfrak{se}(3)$ & Bézier

Per-primitive translation and rotation are optimized jointly as a single Lie-algebra twist $\zeta_i(t) = (\omega_i, \nu_i) \in \mathfrak{se}(3)$ and mapped via closed-form exponential map. Local quadratic Bézier control points model nonlinear paths without rotational ambiguity.

Closed-Form Exponential Map

Gauge-Fixed Anchors

Each primitive carries a learnable local anchor $a_i$ acting as its rotation pivot. We canonicalize $a_i \perp \omega_i$ (orthogonal to the rotation axis) and constrain translation to $p_{\parallel}$, fixing the gauge ambiguity and ensuring rock-solid numerical stability.

Elimination of Null Space Drift

Fixed Canonical Set

Instead of runaway densification that explodes VRAM, Open4DGS fixes the canonical primitive count. Low-opacity slots are recycled into active regions weighted by a 4-term normalized score: opacity, motion magnitude, 2D gradient, and temporal loss gradient.

Constant GPU Memory Footprint
PRODUCTION WORKFLOW

The End-to-End Open4DGS Pipeline

From raw multi-camera video to interactive spatial playback in 7 transparent stages.

1

Extract Frames & Timecode Alignment prep/extract/

Extract synchronized video sequences across camera subfolders using common timecode references. Organizes takes automatically with orientation correction.

python prep/extract/extract_synced_sequence.py --input-dir /path/to/videos --output-dir images --start-timecode HH:MM:SS:FF --num-frames 300 --take 7
2

Multi-Camera Calibration & Sparse Alignment prep/align/

Establishes unified scene metric coordinate space, camera intrinsics, and extrinsics via automated COLMAP model generation. Target reprojection error below 1 pixel.

python prep/align/build-colmap-ref-trigs.py --data-dir . --ref-frame 1
3

Foreground Mask Extraction (BGMv2 / SAM 2) prep/mask/

Generates high-precision alpha mattes with clean background plate subtraction (Background Matting V2) or promptable SAM 2 segmentation when clean plates are unavailable.

python prep/mask/generate_masks.py --data-dir . --method bgmv2 --refine-mode thresholding --foregrounds-output-dir foregrounds
4

Sub-Frame Timing Offset & Rolling Shutter Solver prep/sync/

Learns continuous per-camera phase offsets and compensates for sensor line readout time (e.g. 15 ms for Sony RX0 II). Replaces costly hardware genlock.

python prep/sync/calculate-sync.py --reference-json reference_export.json --images-root images --fps 60 --rs-readout-ms 15.0 --output-sync sync_run/sync.json
5

RoMa v2 Spatial Matching + MEMFOF Temporal Tracking prep/init/

Computes dense cross-camera 3D points, tracks dynamic correspondences across frames with optical flow, detects high-velocity regions, and injects speed-aware sub-keyframes.

python prep/init/prep-roma-trigs-memory.py --reference-json reference_export.json --images-root images --timing-json sync_run/timing_candidate.json --output-path keyframes.npz
6

Fast-Motion Aware 4DGS Training train/trigs/

Trains continuous $\mathfrak{se}(3)$ trajectories with adaptive fast-frame visit top-up (387 visits). Supports temporal chunking for arbitrarily long captures.

trigs-train-colmap --data-dir . --output-dir runs/dance --npz-path keyframes.npz --timing-json sync_run/timing_candidate.json --config.n-steps 30000
7

Timeline Delivery & Interactive Viewer deliver/ & viewer.py

Exports lightweight streaming timeline files (output.trigsbin) and dense PLY sequences. Watch runs in real-time with zero-restart viewer snapshots.

python deliver/make-trigs-timeline.py from-checkpoint --checkpoint runs/dance/trigs_ckpt.pt --output output.trigsbin --frame-rate 60
REAL-TIME INSPECTION

Real-Time Interactive 4D Viewer

Open4DGS includes an interactive viewer (src/trigs/core/viewer.py) that watches training in real time. It streams lightweight parameter snapshots every 1,000 steps without restarting the training loop, allowing directors and researchers to inspect reconstructions live from any perspective.

  • Live Snapshot Monitoring: Watch convergence progress in real-time across arbitrary camera orbits.
  • Timeline Scrubbing: Fluid interactive playback with variable speed, forward/backward looping, and timeline scrub.
  • Production Formats: Seamless export to .trigsbin streaming timelines and standard per-frame PLY packages for Blender, Unreal Engine, and WebGL viewers.
python src/trigs/core/viewer.py --watch runs/dance/viewer_snapshots --port 8080

Live interactive orbit during a 60 FPS dance take reconstruction.

Additional Capture Results & Angles

Multi-view reconstructions across diverse motion speeds and camera layouts

BibTeX & Citations

If you use Open4DGS in your research or spatial productions, please cite our open-source release alongside the foundational TRiGS and Gaussian Splatting works:

@misc{open4dgs2026asu,
  title   = {Open4DGS: Open-Source Production-Ready 4D Gaussian Splatting Pipeline for Volumetric Video},
  author  = {Narrative and Emerging Media Program, Arizona State University},
  year    = {2026},
  url     = {https://github.com/ASU-NEM-Labs},
  note    = {Open-source spatial storytelling initiative}
}

@article{yeom2026trigs,
  title   = {TRiGS: Temporal Rigid-Body Motion for Scalable 4D Gaussian Splatting},
  author  = {Yeom, Suwoong and Nam, Joonsik and Choi, Seunggyu and Lee, Lucas Yunkyu and Kim, Sangmin and Park, Jaesik and Kim, Joonsoo and Yun, Kugjin and Kong, Kyeongbo and Kang, Sukju},
  journal = {arXiv preprint arXiv:2604.00538},
  year    = {2026},
  url     = {https://wwwjjn.github.io/TRiGS-project_page/}
}

@article{kerbl20233d,
  title   = {3D Gaussian Splatting for Real-Time Radiance Field Rendering},
  author  = {Kerbl, Bernhard and Kopanas, Georgios and Leimk{\"u}hler, Thomas and Drettakis, George},
  journal = {ACM Transactions on Graphics},
  volume  = {42},
  number  = {4},
  year    = {2023}
}
Foundational Credits & Open-Source Lineage:

Open4DGS builds on the brilliant research of TriGSVanilla and TRiGS (Yeom et al.), RoMa v2 (Edstedt et al.), MEMFOF (MSU Video Group), SAM 2 (Meta AI), Background Matting V2 (Lin et al.), gsplat (Ye et al.), and COLMAP (Schönberger et al.).