NVIDIA Omniverse NuRec Developer Hub

Real-world data to interactive, photorealistic simulation.

NVIDIA Omniverse™ NuRec is a set of APIs and libraries that leverage Gaussian-based methods to reconstruct and render interactive simulations from real-world data.

NuRec 26.10

• Modular SDK source. Inspect and customize GPU-accelerated building blocks, reconstruction models, training systems, and rendering services. Start with the source overview, sensor library, and loss library.
• Container-based development and execution. Build a development container from the GHCR development dependencies image, and use separate runtime images for the reconstruction application and tools. See build setup and the AV benchmark workflow.
• LiDAR v2: Reconstruct a scene from measured rays and render observations for spinning and solid-state sensors with different scan patterns, sensor geometries, and timing, including simultaneous multi-LiDAR rendering at new vehicle poses. See the LiDAR implementation and sensor API.
• Radar reprojection. Update recorded radar returns and Doppler estimates for simulated vehicle and actor poses, supporting closed-loop sensor-fusion evaluation within the coverage of recorded viewpoints. See the repository and build and usage guide.
• A reproducible AV benchmark workflow. Run a defined reconstruction-to-evaluation chain with task timings, image and object metrics, and scene export. See the workflow.
• A Physical AI dataset refresh. Updated reconstructions and benchmark results for Physical AI scenes; the dataset card is the source for available versions.

For release-specific changes, container versions, and known issues, see the NuRec 26.10 release notes.

NVIDIA Omniverse™ NuRec is a set of APIs and libraries that leverage Gaussian-based methods to reconstruct and render interactive simulations from real-world data. It includes libraries and containers for reconstruction, a rendering API, and tools for scene initialization and rendering refinement.

NuRec uses the open-source gsplat library for efficient Gaussian splatting and high-fidelity scene rendering. It supports camera features critical to physical AI simulation, including rolling shutter and fisheye lenses.

Choose the guide that matches your application. Check the hardware and software requirements before running NuRec.

Try an existing scene first: use the pre-reconstructed AV dataset and follow Render the Physical AI Dataset. This path starts at rendering; it does not require you to reconstruct the scene first.

Bring your own data: follow Prepare Data, Generate Auxiliary Data, and Reconstruct an AV Scene. For robotics inputs, use the robotics guide above.

Render a USDZ artifact: the render command uses a trained artifact without the original dataset or a gRPC service. Render camera images by default, or use --lidar for native LiDAR point clouds. Run each modality separately with its own output directory.

Choose a component by the task you want to accomplish. Installation, model details, and supported inputs are maintained at the linked destinations.

NCore provides a common representation for sensor recordings, calibration, and pose across NuRec workflows. Convert a recording once, then use it as input to reconstruction and related data-processing tools.

• What it provides: a shared data representation and conversion guides for bringing your own sensor recordings into NuRec.
• What you can build: a repeatable data-ingestion path for fleet recordings or public driving datasets.

Resources: Repository · Data and conversion guides

Use NuRec's APIs, libraries, and containers to reconstruct and render interactive simulations from real-world data, or integrate individual components into an existing pipeline.

• What it provides: modular reconstruction libraries and containers, camera and LiDAR rendering APIs, tools for scene initialization and rendering refinement, OpenUSD scene artifacts, and simulator-facing interfaces.
• What you can build: reusable environments and synthetic sensor data for physical AI applications, including robotics and autonomous driving.

Resources: Source and SDK · Build setup · AV benchmark workflow · AV guides · Robotics guides

InstantNuRec is a feed-forward neural model that predicts 3D Gaussian scenes from images in one pass. It runs on a single GPU and can initialize NuRec reconstruction or provide standalone predictions for rapid scene inspection before refinement and rendering.

• What you can build: rapid previews from captured images and an initial scene representation for subsequent reconstruction.

Resources: Repository and usage · Model

Fixer removes artifacts from neural reconstruction outputs, including flicker, blur, and visual errors in novel-view renders of 3D Gaussian Splat scenes. It cleans rendered outputs before they reach simulation or synthetic data generation.

• What you can build: a render-cleanup pass for trajectories that depart from the original capture path or expose under-observed parts of a reconstruction.

Resources: Repository · Model · Technical blog

Harmonizer is a diffusion model that corrects artifacts, lighting mismatches, and temporal flicker in neural scene renderings. Applied after NuRec reconstruction and rendering, it improves visual consistency for training and simulation. Temporal conditioning stabilizes sequences, and object harmonization aligns inserted assets with the surrounding scene.

• What you can build: a render-refinement pass for novel viewpoints, edited scenes, and inserted assets, with consistent lighting and appearance across frames.

• Built on NVIDIA Cosmos

  The Harmonizer post-processing feature uses Cosmos-Predict2-0.6B-Text2Image and the Harmonizer checkpoints, which are licensed under the NVIDIA Open Model License. The model weights are not part of this repository or of the published images: the server downloads them from Hugging Face on first use, and you must accept the Cosmos-Predict2 access terms on Hugging Face and provide an HF_TOKEN first. See Use Harmonizer for Post-Processing.

Resources: Repository · Model · NuRec integration guide

Asset Harvester is a model pipeline that extracts reusable Gaussian-based 3D actors and objects from driving recordings and reconstructed scenes, including limited or obstructed views. Given a reconstruction and an object ID, it produces a reusable asset for scene population. Reuse harvested assets across traffic configurations and scene edits, with Harmonizer aligning inserted assets to the surrounding environment.

• What you can build: asset libraries drawn from real fleet observations and diverse simulation scenes populated without manually creating every object.

Resources: Repository · Model · Live demo · Benchmark · NuRec integration guide

These components support different entry points. Scene enhancement and asset extraction are optional additions to a workflow; they are not mandatory stages for every reconstruction or rendering task.

• AlpaSim: start with the AlpaSim repository for closed-loop AV simulation.
• CARLA: follow the CARLA NuRec integration guide.
• Isaac Sim: follow Neural Volume Rendering.
• Radar Reprojection: use the CPU library or gRPC service alongside NuRec camera and LiDAR rendering. Follow the build and usage guide for setup and integration.
• Custom simulator: follow the gRPC server guide and API reference.

For rendering implementations and research, see gsplat and 3DGRUT. Use the NuRec integration guides above for the supported product workflow.

NuRec can be used as a complete reconstruction and rendering workflow or as building blocks within another pipeline. The source tree separates accelerated libraries, models, training systems, data loading, and application entry points. For example, the sensor library exposes differentiable projection kernels and Python model wrappers; the loss library provides fused computation and configurable loss modules.

• gsplat is an independent open-source project. gsplat is a CUDA-accelerated Gaussian splatting library with Python bindings, hosted by the Nerfstudio project and developed by contributors from UC Berkeley, NVIDIA, and other institutions. It is not an NVIDIA product.
• NuRec renders through gsplat. NuRec uses a single gsplat-based rendering path for cameras and LiDAR, built on gsplat's 3DGUT rasterizer, and builds against a pinned gsplat revision.
• NVIDIA contributes NuRec capabilities upstream. Features developed for NuRec, such as 3DGUT, LiDAR rasterization, NCore V4 capture support, and the HiGS-based inference rendering path, are contributed to gsplat, where any gsplat user can adopt them.
• Pick the level you need. Use gsplat directly if you already have a Gaussian splatting training pipeline and want faster or more capable rendering. Use NuRec if you want the full pipeline: multi-sensor data handling, dynamic actors, USDZ export, enhancement, and serving.

NuRec assets are distributed through six channels. Use the one that matches what you need.

Choosing between GHCR and NGC. Use the GHCR development dependencies image to build a development container for this repository. Runtime containers are distributed through NVIDIA NGC. Follow the release notes for the image names and tags appropriate to your release.

Prepare a GPU-enabled development container and build the NuRec application and tools. Follow the source setup guide for software requirements, cloning, container setup, and build instructions.

Before submitting a change, follow the contributor tests and checks for formatting, builds, and full or targeted tests.

Choose and customize a configuration for your data and reconstruction task. Follow the configuration guide for presets, overrides, and time-range settings.

For source-checkout training and checkpoint validation, follow the reconstruction guide and validation guide. To resume training, see Resuming a prior run.

Render camera images or native LiDAR point clouds from a USDZ artifact without the original recording or a live gRPC service. Follow the source rendering guide for commands, camera and LiDAR selection, resolution defaults, and separate invocations for each modality.

NuRec agent skills guide coding agents through dataset discovery, data onboarding, reconstruction, rendering, and AV benchmarking. The skills are included in this repository under skills/. See the NuRec Agent Skills index for available skills and usage guidance.

Pick data at the stage where you want to enter the workflow. Each dataset card contains its current contents, access requirements, and license.

The AV datasets require acceptance of their access terms. Follow the dataset card and the relevant NuRec guide for download and preparation.

Explore NVIDIA technical articles for worked examples and tutorials. Use the official documentation for current setup and API details.

• Scale AV Perception Across Vehicle Platforms with NVIDIA Omniverse NuRec — Render reconstructed scenes through a different vehicle's camera rig.
• Optimizing a Neural Reconstruction Pipeline Using NVIDIA Nsight Developer Tools — Profile and optimize the NuRec reconstruction pipeline.
• How to Enhance 3D Gaussian Reconstruction Quality for Simulation — Explore artifact cleanup with Fixer.
• Reconstruct a Scene in NVIDIA Isaac Sim Using Only a Smartphone — Follow a capture-to-simulation robotics example.
• Revolutionizing Neural Reconstruction and Rendering in gsplat with 3DGUT — Learn the rendering and reconstruction foundations.

Use the official NuRec documentation for detailed procedures. These links take you directly to common tasks.

For sensor schemas, conventions, and converters, use the NCore documentation. For release-specific behavior, use the NuRec release notes.

• Usage questions and NuRec container help: NVIDIA Developer Forum — NuRec.
• NuRec source bugs, documentation issues, and feature requests: use NuRec Issues. For source questions, use the Question / General Discussion issue form.

For NuRec code and documentation changes, follow CONTRIBUTING.md. Pull requests are reviewed on GitHub, imported into NVIDIA's source repository for review and testing, and published in a subsequent source update. GitHub pull requests are then closed with a reference to the landed commit rather than merged directly. Discuss new features, public API or configuration changes, and substantial refactors in an issue first. For companion repositories, follow each repository's contribution policy.

To report a security vulnerability, please follow the process described in SECURITY.md.

This project is licensed under the Apache License 2.0. See LICENSE.md for details. Third-party components are listed in THIRD_PARTY_NOTICES.md.

Containers, model weights, companion repositories, and datasets have their own terms; consult the linked container page, repository, or model and dataset card.

Vendored PyTorch3D code

nurec/libs/pytorch3d_knn/ contains k-nearest-neighbor kernels derived from PyTorch3D, copyright Meta Platforms, Inc. and affiliates. The files derived from PyTorch3D are under the BSD-3-Clause license. Its text is in nurec/libs/pytorch3d_knn/LICENSE and is repeated in the header of each of those files. The other source files in that directory carry NVIDIA Apache-2.0 headers.