InstantNuRec: Feed-Forward 3D Gaussian Reconstruction from Driving Logs

• Python 3.11
• NVIDIA driver and GPU VRAM — see the NuRec Hardware Setup and Requirements page; Instant-NuRec inherits the same minimums.
• uv — the Astral Python package manager. Install with curl -LsSf https://astral.sh/uv/install.sh | sh or pip install uv.

git clone https://github.com/NVIDIA/instant-nurec.git
cd instant-nurec
./setup.sh
source .venv/bin/activate

setup.sh runs uv sync --frozen, which installs the locked dependency tree from uv.lock into .venv/. In this default installation, the only CUDA dependency is whatever the pinned torch wheel ships with.

The optional calibrated sky-composited renders use gsplat, which is not installed by setup.sh. Install the render extra before using --render-preview or --render-video:

The render and training extras install nvdiffrast under NVIDIA's Source Code License (1-Way Commercial). Non-NVIDIA use is limited to non-commercial research or evaluation; read the complete terms in THIRD_PARTY_LICENSE.txt before installing or redistributing these extras.

uv sync --extra render

The first calibrated render JIT-compiles the pinned public CUDA kernels for the active PyTorch/CUDA/GPU target and can take several minutes; subsequent runs reuse the cache.

The PLY, sky sidecar, and cubemap quick-look image do not require this extra. Full-video export also requires ffmpeg with the libx264 encoder on PATH.

This repo is native-Python only — no Docker required. If you want a container, use the standard NuRec image as a generic CUDA environment.

Note: The checkpoint selected by --model is auto-downloaded into the Hugging Face hub cache on the first inference run. PA-front remains the default.

However, you can also manually download the model into a directory of your choice:

pip install huggingface_hub[cli]
hf auth login
hf download nvidia/instant-nurec --local-dir checkpoints

This places the following files in checkpoints/:

checkpoints/
└── pth/
    ├── instant_nurec_pa_front_1.1.0.pth
    ├── instant_nurec_pa_multiview_1.1.0.pth
    └── instant_nurec_pq_road_1.0.0.pth

Point the pipeline at this local copy by exporting:

export INSTANT_NUREC_FULL_PT="$(pwd)/checkpoints/pth/instant_nurec_pa_front_1.1.0.pth"

When using a local override, make sure the file matches the --model selection.

Note: The selected pretrained weights are fetched on first inference from the Hugging Face repo nvidia/instant-nurec and cached locally; subsequent runs read them from the cache. Set INSTANT_NUREC_FULL_PT to a matching local checkpoint to override the auto-download.

The following inference profiles are available:

The clip lives in a gated HF dataset. Accept the terms at nvidia/PhysicalAI-Autonomous-Vehicles-NCore while logged into Hugging Face, then hf auth login locally; the same auth covers the nvidia/instant-nurec model auto-download on first run.

# Download the clip (~2 GB)
hf download \
    nvidia/PhysicalAI-Autonomous-Vehicles-NCore --repo-type dataset \
    --include "clips/000da9de-0ee5-465a-9a2d-e7e91d3016bb/*" \
    --local-dir ./demo_clip

# Reconstruct it
python run_inference.py \
    --model pa-front \
    --ncore-path ./demo_clip/clips/000da9de-0ee5-465a-9a2d-e7e91d3016bb/pai_000da9de-0ee5-465a-9a2d-e7e91d3016bb.json \
    --output-dir ./demo_output \
    --merge

For the default pa-front command above, success looks like one merged export bundle whose PLY is at ./demo_output/<run_id>/ply/pai_000da9de-.../pai_000da9de-....ply — ~1.88 M Gaussians, kl-optimal voxelized from 2.87 M merged (3.18 M pre-merge across 2 chunks) to land in [0.9 * --n-gaussians, --n-gaussians] (default target 2 M). Omit --merge to write per-chunk bundles instead (voxelization is bundled with merge and runs only when the flag is set).

To run another model on the same clip, select its profile, such as --model pa-multiview or --model pq-front.

For pa-front or pa-multiview, add --include-dynamic to retain dynamic Gaussians. Motion uses NCore cuboid tracks by default; use --motion-source learned for model-predicted motion instead.

python run_inference.py \
    --model pa-multiview \
    --ncore-path /path/to/sequence.json \
    --output-dir ./out \
    --merge --include-dynamic

If no tracks are available, inference falls back to learned motion. Add --render-video to render the dynamic scene.

Every exported PLY has two sky files with the same stem. With --merge, the stem is <sequence_id>; without it, each stem is <sequence_id>_chunk<N>.

The <stem>.sky.npz sidecar uses format version 1 and can be loaded with numpy.load(path, allow_pickle=False). It contains exactly these keys, where S is the cubemap face size (448 for the current released profiles) and C is the number of distinct context-camera sensors:

Cubemap directions use the ncore_world (x, y, z) axes. For a nonzero direction, let a = max(|x|, |y|, |z|); the dominant signed axis selects a face, and (u, v) are grid_sample coordinates in [-1, 1] (+u moves right across an image row and +v moves down across image rows):

The <stem>.sky.png quick-look lays these faces out as [[left, front, right], [back, bottom, top]].

Affine row i corresponds to affine_sensor_indices[i] and affine_sensor_ids[i]. Their order comes from the context rig's ordered camera calibrations, which follows the configured profile or repeated --camera-id order; the numeric sensor index is an identifier, not an array index. For a composited RGB color c, apply row [A | b] as clamp(A @ c + b, 0, 1), after sky alpha compositing.

To also produce a calibrated still preview and full source-trajectory video:

This command installs the render extra governed by the nvdiffrast terms linked in the setup section above.

uv sync --extra render
python run_inference.py \
    --model pa-front \
    --ncore-path /path/to/sequence.json \
    --output-dir /tmp/out \
    --merge \
    --render-preview \
    --render-video

Standard 3DGS PLY has no field for an environment cubemap or per-camera ISP correction. Consequently, SuperSplat and ply_viewer display the Gaussian foreground from <stem>.ply but do not automatically show the sky from <stem>.sky.npz; a sidecar-aware renderer must composite it.

Both render outputs currently support NCore F-theta cameras and use their calibrated per-pixel world rays, including the original projection and interpolated exposure start/end rolling-shutter poses. --render-preview writes the first context frame associated with each exported PLY. --render-video reopens the source sequence and streams every original frame from the first configured context camera without loading the full ray sequence into memory. It requires --merge, because a complete trajectory should be rendered against one complete merged scene. The video path currently accepts one resolved NCore sequence per invocation and fails with the required --max-chunks value if the configured cap would truncate its reconstruction. These are source-trajectory checks, not arbitrary novel-view renders. Other camera models and F-theta cameras with an external windshield-distortion model are rejected explicitly rather than silently approximated.

The PLY is a 3DGS PLY (Gaussian Splatting), not a point cloud — generic viewers like MeshLab / macOS Preview will fail to open it. Use one of:

• SuperSplat — browser, no install.
• ply_viewer — shipped in the NuRec container.

--ncore-path accepts two input shapes:

The path is treated as one ncorev4 sequence metadata file. This matches NuRec's own input convention.

./run.sh \
    --ncore-path /path/to/clips/<uuid>/pai_<uuid>.json \
    --output-dir /tmp/out

The path is treated as a list of sequence JSON paths, one per line. Each line may be absolute, relative-to-the-LST-file's directory, or ~/-prefixed; lines starting with # and blank lines are skipped; mixed absolute + relative entries in a single LST are supported.

# example_manifest.lst
/abs/path/to/clips/<uuid_a>/pai_<uuid_a>.json
relative/path/to/clips/<uuid_b>/pai_<uuid_b>.json
~/symlinked/clips/<uuid_c>/pai_<uuid_c>.json

./run.sh \
    --ncore-path /path/to/example_manifest.lst \
    --output-dir /tmp/out \
    --merge

run.sh validates the input + output paths and execs python run_inference.py. You can also call the CLI directly:

python run_inference.py \
    --ncore-path /path/to/sequence.json \
    --output-dir /tmp/out

Output bundles are written under out_dir/<run_id>/ply/<sequence_id>/<stem>.*; the optional <stem>.render.png appears only with --render-preview, and <stem>.render.mp4 appears only with --merge --render-video.

The full test environment installs the training extra governed by the nvdiffrast terms linked in the setup section above.

uv sync --frozen --extra training
.venv/bin/python -m pytest tests/ -q
.venv/bin/ruff check .

The training extra is required to collect the full test suite. Base-only inference installations can run the inference-specific tests directly.

The released model is trained on NVIDIA's Autonomous Vehicle (AV) platform data. While the model generalizes reasonably to cameras with similar field of view and mounting geometry, rectifying your source camera to match the training intrinsics yields noticeably better reconstruction quality.

If your camera has different intrinsics, we recommend performing a one-time camera adaptation (intrinsic rectification) before running inference. Invalid regions introduced by the remapping should be inpainted rather than filled with black to avoid training/inference artifacts.

For the complete rectification workflow, target intrinsics extraction, and script templates, see docs/camera_rectification.md.