Get Started with NuRec: For Autonomous Vehicles#

For developers working on autonomous vehicle software, NVIDIA Omniverse NuRec is integrated with a few different simulation platforms. To use NuRec in an AV simulation platform, you can get the pre-reconstructed scenes from the NVIDIA NuRec-trained Physical AI dataset on HuggingFace and then use them on a simulation platform that has built a NuRec integration, like AlpaSim or CARLA.

To build a scene yourself instead, work through the following steps in order. Each one produces the input for the next: sensor data becomes NCore-format data, NCore data gains the auxiliary data that training needs, training produces a reconstructed USDZ scene, and evaluation tells you whether that scene is good enough to use.

Step 1: Set Up#

Check that you have the right hardware and software setup to run NuRec.

Step 2: Get Compatible Data#

The NCore data format is a standardized format for sensor data recordings from various sources, including camera and lidar sensors. NuRec requires input data to be in the NCore format.

If you get the pre-reconstructed scenes from the NVIDIA NuRec-trained Physical AI dataset on HuggingFace, the scenes are already trained and exported, so skip to Step 6.

To reconstruct scenes yourself, choose one of the following starting points, described in full in Prepare Data for Use with NuRec:

Start from pre-converted data. Get sample data that is already in the NCore format from the NVIDIA NCore-converted Physical AI dataset on HuggingFace.
Start from raw sensor data. Get a clip from the NVIDIA Physical AI raw dataset and convert it to the NCore format with the NVIDIA Physical AI Data Converter.

Step 3: Generate Auxiliary Data#

NuRec training needs auxiliary data that the NCore recording does not carry on its own. Generate it from your NCore-format data by following Generate NuRec Auxiliary Data.

Step 4: Generate Reconstructed Scenes#

Reconstruction converts real-world data into a 3D scene, output as a USDZ file. To create these reconstructed scenes, use your NCore-formatted data and follow the steps outlined in Reconstruct an AV Scene with NuRec.

Training writes the scene to artifacts/last.usdz, which is the artifact the remaining steps consume.

Step 5: Evaluate the Reconstructed Scene#

Before you take a scene into a simulator, render it with NuRec and check its quality:

Use the NuRec Render Command to render images and video from the reconstructed scene.
Run NuRec Validation to measure reconstruction quality and interpret the resulting metrics.
Generate Novel Views with NuRec to check how the scene holds up from camera positions that the original recording never captured.

Step 6: Render Reconstructions in a Simulation#

To render reconstructed scenes, you can use existing NuRec-powered simulation platforms like AlpaSim or CARLA:

You can also use the NuRec gRPC API in your own simulation platform.