3D Gaussian Splatting — Scene Reconstruction & Novel-View Synthesis

CV course project investigating 3D scene reconstruction and novel-view synthesis with 3D Gaussian Splatting (3DGS, Kerbl et al., SIGGRAPH 2023). We use the official implementation as a baseline and add lightweight extensions, a benchmark + phone-capture data pipeline, and an evaluation harness (PSNR / SSIM / LPIPS).

Team: Muhammad Shamoon Butt · Ahmad Hamza Warraich · Unnus Ahmad Usmani

Full development plan: see the approved plan and report/.

---

How this repo is organized

Path	Purpose
third_party/gaussian-splatting/	Official 3DGS code (cloned by setup_remote.sh) — the unmodified baseline.
scripts/	Orchestration around train.py / render.py / metrics.py / convert.py (train_scene.sh, train_all_360_v2.py, …).
data/	Datasets + phone captures (gitignored).
experiments/configs/	One YAML per run (reproducibility). experiments/runs/ holds outputs.
extensions/	Our three lightweight experiments (few-view, under-observed, smartphone).
eval/	aggregate_metrics.py → committed results tables in eval/results/.
viewer/	Stretch: web-based splat viewer.
report/	Figures, fly-through videos, write-up.

We deliberately do not reimplement 3DGS — the official repo already provides training, rendering, metrics, and the COLMAP wrapper.

---

Setup (run on the remote GPU box, not the laptop)

There is no local GPU; all training/rendering happens on a remote CUDA machine over SSH.

# on the GPU box, from the repo root:
bash scripts/setup_remote.sh        # clones official 3DGS, builds CUDA ext, makes conda env
conda activate gaussian_splatting

⚠️ #1 gotcha — CUDA mismatch. Check nvidia-smi (CUDA version) and edit environment.yml's pytorch-cuda= to match before running setup. The CUDA rasterizer won't build otherwise.

Use tmux/screen for long unattended training runs over SSH.

---

Smoke test (do this first — proves the toolchain)

# get NeRF-Synthetic 'lego' into data/benchmarks/nerf_synthetic/lego, then:
bash scripts/train_scene.sh data/benchmarks/nerf_synthetic/lego lego_baseline
bash scripts/render_eval.sh lego_baseline
python eval/aggregate_metrics.py

Expect PSNR ≈ 30+ on lego. If so, the pipeline is correct end-to-end.

---

Typical workflow

# 1. (custom data) phone video -> COLMAP scene
python scripts/prepare_video.py --video data/custom/livingroom.mp4 \
                                --out data/custom/livingroom --target-frames 150

# 2. train a scene  (run_name convention: <scene>_<variant>)
bash scripts/train_scene.sh data/benchmarks/mipnerf360/garden garden_baseline

# 3. render held-out views + compute PSNR/SSIM/LPIPS
bash scripts/render_eval.sh garden_baseline

# 4. aggregate every run into eval/results/summary.{csv,md}
python eval/aggregate_metrics.py

# 5. qualitative fly-through video (smooth orbit; ~15 s default)
python scripts/render_flythrough.py \
    --model experiments/runs/garden_baseline \
    --out report/garden_baseline_flythrough.mp4 \
    --duration 15 --fps 24

# all trained runs at once (2 rotations, 20 s default)
bash scripts/make_all_flythroughs.sh --duration 20 --fps 24 --rotations 2 --out-dir report

Train all Mip-NeRF 360 scenes (data/360_V2)

Place the official Mip-NeRF 360 download under data/360_V2/ (one folder per scene: bicycle, bonsai, counter, flowers, garden, kitchen, room, stump, treehill). Each scene should have COLMAP sparse/0/ and the downscaled images_4/ folder.

# Train all 9 scenes at 1/4 resolution (30k iterations each)
python scripts/train_all_360_v2.py

# Resume a long batch run — skip scenes that already finished
python scripts/train_all_360_v2.py --skip-existing

# Train + render + metrics in one pass
python scripts/train_all_360_v2.py --render-eval
python eval/aggregate_metrics.py

Models are saved to experiments/runs/<scene>_baseline/ (e.g. garden_baseline). The path contains 360, so --eval applies the official LLFF hold-out split automatically.

To train a single scene manually:

bash scripts/train_scene.sh data/360_V2/garden garden_baseline -i images_4

---

🎨 Interactive 3D Viewer

Explore trained scenes with an interactive arrow-key camera control system + smooth animation paths.

Launch the viewer

# On the GPU box (or any machine with trained models):
pip install fastapi uvicorn
bash scripts/launch_viewer.sh
# Opens at http://127.0.0.1:5000 in your browser

Controls

Action	Key
Move	↑ ↓ ← → or W A S D
Look around	Mouse drag
Up / Down	SPACE / CTRL
Play orbit	▶ Play button
Change model	Dropdown (10 pretrained scenes)
Upload phone video	Upload → auto-COLMAP → train pipeline

Features

• 10 pretrained models ready (Mip-NeRF 360: bicycle, bonsai, counter, flowers, garden, kitchen, room, stump, treehill + NeRF Synthetic: lego)
• Smooth camera orbits with pause/resume and manual scrubbing
• Phone video upload with automatic frame sampling & COLMAP
• Real-time FPS + triangle count in the HUD
• Adjustable field-of-view for different perspectives

The viewer loads trained .ply point clouds and renders them as interactive 3D point clouds. Each model's camera path is computed from COLMAP poses (or generated as a circular orbit).

---

Datasets

• Mip-NeRF 360 — full 9-scene set in data/360_V2/<scene>/ (uses images_4/).
• Tanks & Temples, NeRF Synthetic (subset of 2–3 scenes each).
• 2 custom phone scenes (1 indoor, 1 outdoor) via prepare_video.py + COLMAP.

Place benchmark scenes under data/benchmarks/<dataset>/<scene>, the Mip-NeRF 360 download under data/360_V2/<scene>, and phone captures under data/custom/<scene> (gitignored).

---

Extensions (breadth — see each folder's README)

• extensions/few_view/ — fewer input views, measure degradation + one mitigation.
• extensions/under_observed/ — depth/regularization prior for sparse-coverage regions.
• extensions/smartphone/ — robust pipeline + training tweaks for noisy phone data.

Every extension reuses the same eval harness and is compared against the *_baseline run.