Use the NuRec gRPC API Server#

To interface with driving simulators, or to generate arbitrary novel views, NuRec implements a gRPC server you can use to serve rendered images for given camera and actor posing.

There are three aspects to rendering with the grpc server:

  • Building the gRPC protobuf package for the client

  • Launching the server

  • Querying it as the client

Building the gRPC Protobuf Package#

Complete Software Setup first. Build the client archive from the same source release as the server:

nurec_dev bazelisk build //nurec/grpc/release:customer_distribution_tarball
export NUREC_BIN="$(nurec_dev bazelisk info bazel-bin)"
mkdir -p "$NUREC_SRC/outputs/grpc-client"
tar -xf "$NUREC_BIN/nurec/grpc/release/customer_distribution_tarball.tar" \
  -C "$NUREC_SRC/outputs/grpc-client"

On the client machine, install Python 3.13. From the extraction root, install the dependencies and generate the modules:

cd "$NUREC_SRC/outputs/grpc-client"
python3.13 -m venv .venv
source .venv/bin/activate
python -m pip install -r nurec/requirements.txt
python nurec/grpc/update_generated.py

Launching the gRPC Server#

Launch the gRPC server by running from source:

    nurec_dev bazelisk run //:run -- \
        serve-grpc \
        --artifact-glob "$NUREC_SRC/outputs/<RUN-ID>/artifacts/last.usdz"

serve-grpc Command Line Parameters#

Every serve-grpc parameter is optional. --artifact-glob pre-registers scenes at startup, as in the example above; omit it to start with no scenes and register each one on demand with the per-request metadata headers described in Register Scenes on Demand.

Server Configuration:

  • --host: GRPC server host (default: “localhost”)

  • --port: GRPC server port (default: 8080)

  • --health-port: Port for gRPC health checks. If unset, health is served on the main gRPC port. Set it to a different port to run a dedicated health-only server (default: None)

  • --max-workers: Maximum number of gRPC handler threads for the single-process server. This is a thread pool, not the multi-process worker pool — for that, see --multiproc-workers (default: 1)

Scene Management:

  • --artifact-glob: Glob expression for scenes to pre-register at startup. Must end in .usdz to find relevant files (default: None)

  • --test-scenes-are-valid/--no-test-scenes-are-valid: Try to load each detected scene before coming online (default: False)

  • --cache-size: Maximum number of models to cache with count-based LRU. If OOM occurs during loading, spare backends are automatically evicted and load is retried (default: 10)

  • --max-backends-per-scene: Maximum number of cached backends allowed per scene. Must be positive and no greater than --cache-size. Unset applies no per-scene limit (default: None)

  • --download-cache-dir: Directory for downloaded scene files (default: “~/.cache/nurec/downloaded_scenes”)

  • --download-cache-size: Maximum number of downloaded scenes to keep in cache (default: 5)

A backend is one loaded, renderable instance of a scene. Several backends can exist for the same scene so that concurrent requests for it do not serialize, which is why the cache is sized in backends rather than scenes — the get_loaded_scenes RPC reports the same quantity as loaded_instance_count. Setting --max-backends-per-scene bounds how much of the cache a single popular scene can occupy. Once a scene reaches its limit, further requests wait for and reuse a checked-in backend instead of constructing another one. Cancellation or the client deadline ends that wait, and competing waiters have no ordering guarantee, so set a client deadline when you configure this. Both flags mean something different under the multi-process pool, where --cache-size counts scenes and --max-backends-per-scene has no effect — see Scene Cache and Readiness.

Rendering Options:

  • --renderer [default|gsplat]: Renderer backend selection. default uses the artifact’s trained renderer (PyTorch forward pass), and gsplat forces the GSplatRenderer. Under --multiproc-workers greater than 0, an unset or explicit default is coerced to gsplat (default: “default”)

  • --allowed-image-formats TEXT: Comma-separated allow-list of the camera response formats the server accepts. Supported values are PNG, JPEG, and RGB_UINT8_PLANAR, with RGB accepted as an alias for RGB_UINT8_PLANAR. Left unset, the server applies no format restriction (default: None)

  • --ray-chunk-size: Maximum number of rays processed in a single forward pass. Set if seeing GPU OOMs (default: 2^62)

  • --tracks-timestamp-ray-stride: Subsample the per-track rolling-shutter timestamp-estimation ray grid roughly N-fold at inference time. Pass 1 to restore full estimation and its cost; there is no separate opt-out switch (default: 16)

  • --enable-editing-actors: Enable the ability to edit the actors in a scene by using poses specified in the render request. If this is disabled, NuRec uses the original actor poses specified in the USDZ. (default: False)

  • --egocar-hood-dir: Directory with egocar hood images (default: None)

Note

As of 26.08 the nrend renderer backend is removed. --renderer no longer accepts nrend, and the deprecated --enable-nrend / --no-enable-nrend flags are gone. Use --renderer gsplat for the fast path.

The older --use-gsplat/--no-use-gsplat pair survives as a deprecated alias for --renderer gsplat. It is hidden from --help and emits a DeprecationWarning, and it only promotes the backend to gsplat while --renderer is left at its default, so an explicit --renderer always wins. --no-use-gsplat warns but selects nothing. Prefer --renderer in new deployments.

--allowed-image-formats is admission control rather than a rendering preference. A request asking for a format outside the allow-list is rejected with INVALID_ARGUMENT before the server loads a scene or renders anything, so a misconfigured client fails fast instead of consuming GPU time. Passing --allowed-image-formats JPEG, for example, turns away both PNG and raw planar RGB requests. The restriction applies to render_camera and batch_render_camera as well as the deprecated render_rgb and batch_render_rgb endpoints, and it behaves the same under single-process and multi-process serving. Whatever you configure is reported back through get_server_config, which lets a client discover the policy instead of discovering it through a failed render.

The value is parsed case-insensitively, tolerates whitespace around the commas, and is stored in canonical form, so --allowed-image-formats "jpeg, rgb" becomes JPEG,RGB_UINT8_PLANAR. That canonical form is what get_server_config reports, which means a client sees RGB_UINT8_PLANAR even where the operator wrote RGB. A malformed list fails at server startup rather than on the first render: an unsupported format name, a duplicate entry, or an empty entry between commas each abort the launch with a message naming the problem.

Harmonizer Options:

  • --enable-harmonizer/--no-enable-harmonizer: Load stateless (non-temporal) Harmonizer post-processing. A render request must set apply_harmonizer = true to use it; false or omitted leaves color output unharmonized (default: False)

  • --harmonizer-url: URL of the Harmonizer checkpoint (default: https://huggingface.co/nvidia/Harmonizer/blob/main/harmonizer_nontemporal.pt)

  • --harmonizer-cache: Full path to local Harmonizer cache dir (default: ~/.cache/nurec/harmonizer)

  • --harmonizer-model-filename: Filename of the Harmonizer checkpoint (default: harmonizer_nontemporal.pt)

  • --harmonizer-resolution: Resolution for Harmonizer processing (default: (576, 1024))

Temporal Harmonizer Options:

  • --enable-harmonizer-temporal/--no-enable-harmonizer-temporal: Load the temporal Harmonizer, which conditions each enhanced frame on earlier enhanced frames. A color request that sets apply_harmonizer = true must include a valid harmonizer_session_id; false or omitted leaves its color output unharmonized, and a supplied session ID resets that session’s history. Mutually exclusive with --enable-harmonizer (default: False)

  • --harmonizer-temporal-url: URL of the temporal Harmonizer checkpoint (default: https://huggingface.co/nvidia/Harmonizer/resolve/main/diffusion_harmonizer.pkl)

  • --harmonizer-temporal-model-filename: Filename of the temporal Harmonizer checkpoint (default: diffusion_harmonizer.pkl)

  • --harmonizer-temporal-resolution: Resolution for temporal Harmonizer processing (default: (576, 1024))

  • --max-harmonizer-sessions: Maximum number of concurrent temporal Harmonizer sessions (default: 32)

See Temporal Harmonizer Sessions for the session lifecycle and its restrictions.

Both checkpoints are fetched on first use and cached under --harmonizer-cache, so most deployments never set the URL or filename flags.

Harmonizer is built on NVIDIA Cosmos and its weights are licensed under the NVIDIA Open Model License. NuRec does not redistribute them. The temporal mode also downloads the gated Cosmos-Predict2-0.6B-Text2Image base model, so accept its terms on Hugging Face and set HF_TOKEN first. See Use Harmonizer for Post-Processing.

The previous --enable-difix, --difix-url, --difix-cache, --difix-model-filename, and --difix-resolution flags are deprecated. They still work and redirect to their --harmonizer-* equivalents, but should not be used in new configurations. The Difix configuration packages, which are a separate surface, were removed rather than deprecated — see Migrate From Difix Configuration.

For what Harmonizer does and its training-side limitations, see Use Harmonizer for Post-Processing.

Multi-Process Serving:

  • --multiproc-workers: Number of worker processes. 0 runs the single-process server (default: 0)

Passing a value greater than 0 launches a multi-process worker pool, which adds its own startup, admission-control, and MPS parameters. For those flags and the deployment model, see Serve NuRec with Multiple Worker Processes.

Performance Profiling:

  • --enable-timing: Enable timing of the different parts of the rendering pipeline (default: False)

  • --timing-verbosity: Verbosity level for timing output: NONE, SUMMARY, BASIC, DETAILS (default: “BASIC”)

  • --timing-logfile: File to write timing results to (e.g., timing.log) (default: None)

  • --timing-synchronize: Synchronize GPU before timing measurements (default: False)

  • --profiling-backend: Profiling backend to use: NONE, TRACY, or NVTX (default: “NONE”)

  • --metrics-output-dir: Directory to save render time metrics. If not specified, metrics collection is disabled (default: None)

Register Scenes on Demand#

Instead of listing every scene at startup with --artifact-glob, you can register a scene when its first request arrives. Attach both of the following metadata headers to a scene-bearing unary RPC.

Header

Description

x-nre-scene-url

An http(s):// URL to download, or a file:// URI or absolute path available to the server.

x-nre-scene-id

The scene ID. It may contain letters, digits, _, -, and ..

Warning

On-demand registration makes the server fetch whatever the caller names. x-nre-scene-url accepts an arbitrary http(s):// URL, which the server requests outbound with no host allow-list, and an arbitrary file:// URI or absolute path, which it resolves with no confinement to a parent directory. A caller that can set this header can therefore drive requests to hosts the server can reach but the caller cannot, and can distinguish existing from non-existing server-local paths through the resulting error messages. Only enable on-demand registration on an endpoint restricted to trusted callers, and put authentication and a URL and path allow-list in front of it. Deployments serving untrusted traffic should pre-register scenes with --artifact-glob instead.

Send the headers together, and make sure the metadata scene ID matches the scene_id in the protobuf request. If one scene ID has more than one registered artifact URL, include both headers so NuRec can select the intended artifact.

NuRec distinguishes registrations by scene ID and artifact URL. Temporary signing credentials do not create a new registration, but fields that select different content do. Keep artifact URLs immutable and publish changed content at a new path or object version. As of 26.08 NuRec also records the artifact’s SHA256 in its cache keys, so if you do republish different bytes at the same URL, the newly downloaded artifact cannot reuse the stale loaded backend.

Downloaded scenes are stored under --download-cache-dir and bounded by --download-cache-size.

Serve with Multiple Worker Processes#

For higher render throughput on one GPU, serve-grpc --multiproc-workers N runs an owner process that holds a single read-only copy of the weights and shares it to N worker processes over CUDA IPC, behind one gRPC endpoint. The gRPC API is unchanged, but the pool is gsplat-only, needs externally managed NVIDIA MPS to scale past one worker, and does not support asset editing or temporal Harmonizer sessions. Per-request actor re-posing through --enable-editing-actors is unaffected and works in both modes; what the pool refuses is the edit_assets and restore_model_parameters RPCs.

See Serve NuRec with Multiple Worker Processes for sizing, MPS setup, admission control, and shutdown.

Server Behavior#

These behaviors apply in both single-process and multi-process mode.

  • API version. get_version() currently reports grpc_api_version 0.0.5, up from 0.0.1 in the 26.06 release. Every increment so far has been additive — new discovery keys, new RPCs, or new request/response fields, all with a byte-identical proto wire format for existing clients — so clients that negotiate on major.minor need no change. A client asserting an exact patch such as (0, 0, 2) will mismatch after upgrading, even on a single-process upgrade; relax those checks to major.minor.

  • Health checking. The server implements the gRPC health checking protocol at the standard health service endpoint. Set --health-port to serve health on a dedicated port.

  • Out-of-memory recovery. When GPU memory is exhausted during model loading, the server evicts spare cached backends and retries the load.

  • Dynamic scene downloading. Scenes can be registered per request through the x-nre-scene-url and x-nre-scene-id metadata headers. See Register Scenes on Demand.

Render from the Command Line with render-grpc#

The bundled render-grpc client drives a running server from the command line. It is the quickest way to render a trajectory and to confirm a server is working. To build your own client instead, see API Reference.

For a camera trajectory that includes color and uses --apply-harmonizer, render-grpc detects whether the target server has temporal Harmonizer enabled. If so, it creates one session for the trajectory, sends frames in trajectory order, and closes the session after rendering. Temporal targets do not support --repeat-count > 1.

Launch it with the following command:

    nurec_dev bazelisk run //:run -- \
        render-grpc \
        --artifact-path "$NUREC_SRC/outputs/<RUN-ID>/artifacts/last.usdz" \
        --output-dir "$NUREC_SRC/outputs/render" \
        --camera-id <CAMERA-ID>

render-grpc Command Line Parameters#

Required Parameters#

  • --artifact-path TEXT: Path to the NuRec artifact last.usdz

  • --output-dir TEXT: Directory for rendered output. See Output layout

Optional Parameters#

Connection:

  • --host TEXT: gRPC server host (default: localhost)

  • --port INTEGER: Port to run the gRPC server on (default: 8080)

  • --wait-server INTEGER: Wait up to N seconds for the gRPC server to come up before rendering, where 0 waits indefinitely. If unset, the client fails fast on the first RPC (default: unset)

Camera output:

  • --height INTEGER: Height of the image (default: 300)

  • --camera-id TEXT: Camera ID (default: camera_front_wide_120fov)

  • --image-format [png|jpeg]: Image format for the output, options are PNG or JPEG (default: jpeg)

  • --output-type [color|distance-colorized|semantics-colorized]: Camera render product to request. Repeatable, so several output types can be produced in one pass (default: color). Must be left at the default color when --lidar is set; any other value is rejected.

  • --colorization-max-distance FLOAT: Upper bound of the distance colormap in meters. Applies when --output-type is distance-colorized. Uses the server default when unset.

  • --overlay-bboxes [none|model|artifact]: Track source for cuboid wireframes drawn on every requested output. model uses the checkpoint’s learned tracks and artifact uses the USDZ sequence_tracks (default: none)

  • --bbox-color INTEGER INTEGER INTEGER: RGB color, 0–255 each, for bounding box edges when --overlay-bboxes is model or artifact. The server default is 0 255 0 (green).

Rendering behavior:

  • --lidar: Call render_lidar instead of render_camera and save point clouds instead of images (default: disabled)

  • --frame-step INTEGER: Step size in frames (default: 1)

  • --repeat-count INTEGER: Render the same request sequence this many times. Not supported for a color trajectory against a temporal target (default: 1)

  • --max-concurrent-renders INTEGER: Cap the number of in-flight render RPCs during the concurrent fan-out. Unset is unbounded (default: unset)

  • --apply-harmonizer: Request Harmonizer processing for every color frame rendered by render-grpc. The target server must have Harmonizer enabled. For a temporal target, render-grpc creates one session for the trajectory, sends frames in order, and closes the session after rendering. Requires --output-type color (which may be combined with other camera products) and cannot be used with --lidar.

  • --temporal-harmonizer-session: Deprecated compatibility option; you can remove this from your commands without any effect on rendering.

  • --sequential: Send gRPC requests one by one, with no parallel batch after warmup (default: disabled)

  • --disable-rolling-shutter: Disable rolling shutter by applying the frame-end timestamps to full frames, which is useful for debugging

  • --demo-actor-transform: Apply a precomputed transformation to actor poses to generate demos

  • --shutdown-server-on-completion: Shutdown the server on completion

  • --rig-name TEXT: Rig name for the inpainted ego hood (e.g. hyperion8.0 or hyperion8.1). Set to None to disable inpainting.

Output Layout#

Camera images are written to <output-dir>/<output-type>/<camera-id>/<frame-index>.<ext>, where the <output-type> directory is color, distance_colorized, or semantics_colorized — one per value passed to --output-type. LiDAR outputs written with --lidar stay flat under --output-dir.

A --repeat-count above 1 writes each pass under a zero-based repeat-NNN subdirectory, starting at repeat-000. At the default, outputs go directly under --output-dir with no repeat subdirectory.

Concurrency#

By default the client fans renders out concurrently. Leaving --max-concurrent-renders unset starts unbounded, but the first RESOURCE_EXHAUSTED shed installs a self-imposed limit of one, so later and retried calls serialize for the rest of the run. Against a server with admission control enabled, set this flag to the server’s --multiproc-max-in-flight. Sheds are retried with capped backoff either way. See Admission Control and Client Fan-Out.

API Reference#

The nurec.grpc.protos package contains every protobuf message and service method the server exposes. Use this section when you build your own client rather than driving the server with render-grpc.

It covers the message types and RPCs you will use most, not the complete schema. For the full generated definitions, see NuRec gRPC API: sensorsim for the service and render messages, and NuRec gRPC API: common for the shared types.

Common Message Types#

Pose and Transformation Types#


    Pose: grpc_types.Pose
        vec: Vec3  # 3D position vector
            x: float  # X coordinate
            y: float  # Y coordinate
            z: float  # Z coordinate
        quat: Quat  # Quaternion representing rotation
            x: float  # X component of quaternion
            y: float  # Y component of quaternion
            z: float  # Z component of quaternion
            w: float  # W component of quaternion

    PosePair: PosePair
        start_pose: Pose  # Initial pose of the sensor
        end_pose: Pose  # Final pose of the sensor (must be different from start_pose)

Camera Types#


    CameraSpec: CameraSpec
        logical_id: str  # Unique identifier for the camera
        trajectory_idx: int  # Index of the trajectory this camera belongs to
        resolution_w: int  # Width of the camera's resolution
        resolution_h: int  # Height of the camera's resolution
        shutter_type: enum  # Type of shutter mechanism
        camera_param: One of:
            ftheta_param: FthetaCameraParam  # F-theta camera parameters
                principal_point_x: float  # X coordinate of principal point
                principal_point_y: float  # Y coordinate of principal point
                reference_poly: enum  # Reference polynomial type
                pixeldist_to_angle_poly: List[float]  # Polynomial coefficients for pixel to angle conversion
                angle_to_pixeldist_poly: List[float]  # Polynomial coefficients for angle to pixel conversion
                max_angle: float  # Maximum angle supported by the camera

Dynamic Object Types#


    DynamicObject: DynamicObject
        track_id: str  # Unique identifier for the track
        pose_pair: PosePair  # Start and end poses of the object
        aspect_state: AspectState  # Optional. Per-frame aspect visibility for this actor

    AspectState: AspectState
        aspects: List[str]  # The complete set of visible aspects for this frame
        gains: List[AspectGain]  # Optional per-aspect intensity for continuous fades

    AspectGain: AspectGain
        aspect: str  # The aspect the gain applies to
        gain: float  # Optional. Scales the aspect's baked alphas (0..1); unset = 1.0
        cold_rgb: List[float]  # RGB color at gain 0; empty = alpha-only fade

Whole-set semantics (dynamic_objects). A non-empty dynamic_objects list is the complete set of actors for that request, not a set of overrides. The server draws exactly the tracks you list, at the poses you supply, and draws no other dynamic actor. Any recorded actor you leave out is absent from that frame. There is no partial-override mode, so moving one actor means re-sending every other actor in the same request. Omission is the supported way to remove an actor. The bundled render-grpc client works this way, and its remove edit is a track left out of an otherwise complete list.

An empty list is a different instruction rather than a smaller version of the same one. It means “make no statement”, and the server draws every recorded actor at its captured pose. Sending your first entry changes what the field means instead of adding one edit.

Warning

A request that lists only the actors you moved renders without the actors you omitted, at transport status OK. Nothing in the response reports the drop. To confirm your actor set is complete, compare the render against one sent with an empty dynamic_objects list.

To re-pose one actor and keep the rest:

  1. Call get_dynamic_objects once per scene and keep the returned trajectories.

  2. For each frame, select the actors whose trajectory span overlaps [frame_start_us, frame_end_us). Listing an actor outside its recorded span freezes it into the frame as a visible, stationary object.

  3. Interpolate each selected actor’s recorded pose at the frame start and end timestamps, substitute your own pose_pair for the actors you drive, and send the full list.

Set overlay_bounding_boxes to OVERLAY_BOUNDING_BOX_SOURCE_MODEL while you build this. The overlay is drawn from the same request list, so a frame with one box means the server received one actor. That source also rejects an unknown track_id with INVALID_ARGUMENT and names the scene’s known track IDs, which a plain render does not.

Per-frame aspect control (aspect_state). A scene baked with nurec compile-asset-edits carries switchable Gaussian clusters (“aspects”) for each edited actor — an asset’s appearance variants and individual panels, for example. aspect_state selects which of those clusters render for one request. Aspect keys are opaque strings published in the artifact’s embedded asset-edits manifest, in the per-control state_aspects and element_aspects tables, such as "appearancemode:alternate", "appearancemode:shared:AssemblyL/part_2", and "panelstate:on". The server attaches no meaning to aspect strings and performs no animation: all animation logic lives in the client.

  • Stateless, full-set semantics. When aspect_state is present, the server shows exactly the listed aspects and hides every other baked cluster of that actor’s edit, for this request only. The server keeps no aspect state between requests, so send the complete visible set every time. An empty aspects list is meaningful: it hides every baked cluster of the actor. When the field is absent, the actor renders its scheduled and baked state for the frame.

  • Gains. Each AspectGain scales one aspect’s baked alphas for continuous opacity fades. gain carries explicit presence so an entry sent only for cold_rgb does not read as 0.0; unset means 1.0. A non-empty cold_rgb sets the color at gain 0, and the aspect blends toward its baked color as the gain rises, while empty means an alpha-only fade.

  • Prerequisites. Honored on every render RPC that carries dynamic_objects — render_camera, render_lidar, and batch renders, per item. The server must be launched with --enable-editing-actors, since requests carrying dynamic_objects are rejected without it, and the scene must be an edit-baked artifact. Supported under both single-process and multi-process serving.

  • Capability discovery. A server predating this field drops it as an unknown protobuf field and returns success without applying it. Before relying on aspect_state, check get_server_config for the key asset_edits.aspect_state with value stateless-v1. The key is advertised only when the server runs with --enable-editing-actors.

  • Validation. A supporting server fails the render with INVALID_ARGUMENT on an unknown track ID, an aspect string the scene has no baked cluster for, or a duplicate aspect_state entry for the same track. The error message lists the scene’s edit-drivable tracks, or the track’s valid aspects. Success from a supporting server therefore means the state was applied.

Service Endpoints#

Server Information#


    get_version()
    # Returns: VersionId (version_id, git_hash, grpc_api_version)

Server Configuration#


    get_server_config()
    # Returns: ServerConfig (server_config: map<string, string>)
    # Current server configuration as string key-value pairs. Use to detect unexpected
    # default changes or configuration parameter mismatches between client and server.

Scene Management#


    get_available_scenes()
    # Returns: AvailableScenesReturn (scene_ids: List[str])
    # Every scene currently registered with the server. Registration does not imply that a
    # scene is loaded or that it has been validated successfully.

    get_loaded_scenes()
    # Returns: LoadedScenesReturn
    # A point-in-time snapshot of the scene instances currently cached by the server.

    shut_down()
    # Shuts down the gRPC server
    # Returns: Empty

Each scenes[] entry in get_loaded_scenes() contains a scene_id, a loaded_instance_count, and a reusable_instance_count. The response-level loaded_instance_capacity reports the global cache size. Sum the per-scene counts to derive fleet totals. A scene can appear with a loaded_instance_count greater than 1 when several cached instances exist for it.

Use get_loaded_scenes() rather than get_available_scenes() when you need to know whether a scene is ready to render.

Camera Management#


    get_available_cameras(scene_id: str)
    # Returns: AvailableCamerasReturn (available_cameras: List[AvailableCamera])

    get_available_trajectories(scene_id: str)
    # Returns: AvailableTrajectoriesReturn (available_trajectories: List[AvailableTrajectory])

Asset Management#


    get_external_asset_objects(scene_id: str)
    # Get list of track IDs for external asset objects in a scene
    # Returns: ExternalAssetObjectsReturn (track_ids: List[str])

    get_dynamic_objects(scene_id: str)
    # Get all controllable dynamic objects in a scene
    # Returns: AvailableDynamicObjectsReturn (dynamic_objects: List[DynamicObjectTrack])

    edit_assets(scene_id: str, replace: List[ReplaceAssetAction], insert: List[DynamicObjectTrack])
    # Replace and insert assets in the scene
    # Returns: EditAssetsResponse (success: bool, message: str)

    restore_model_parameters(scene_id: str)
    # Restore the original training parameters, undoing asset edits. Whole-scene: there is no
    # selective restore.
    # Returns: Empty

Warning

Known issue in 26.10. get_dynamic_objects returns a first timestamp_us near 2**64 for any actor whose first recorded pose predates the scene start. That pose has a negative microsecond offset, and the offset wraps when it is written to the unsigned timestamp_us field, so the entry sorts as the far future and the trajectory is no longer in time order. One Physical AI scene showed this for 6 of its 30 actors, so treat it as common rather than rare.

Detect it by testing timestamp_us > 2**63, or by checking that each trajectory’s timestamps strictly increase. The true value is recoverable — timestamp_us - 2**64 yields the original negative offset — so you can either drop the affected entry or convert it and re-sort. Do one or the other before using a trajectory to place an actor at a chosen time.

Rendering is not affected. render_camera and render_lidar place actors from the DynamicObject.pose_pair you supply for every actor the request lists, not from the trajectory this call returns, so only consumers that read the returned trajectory see the wrapped value. An actor the request omits is not drawn at all. See Dynamic Object Types. A fix is targeted for 26.11.

Always check EditAssetsResponse.success. On failure, nothing was installed.

Both an insert’s asset_id and a replace’s replacement_id resolve the same way. If the ID names a track in the scene’s asset bank, reported by get_external_asset_objects, that asset is used. Otherwise the ID is treated as a path to a server-local .ply file.

Asset insertion and replacement require a topology-edit-compatible target layer. In particular, a plain artifact whose target is an interpolated-table deformable layer still rejects these edits because they would invalidate its row-aligned trajectories. See Edit Actors in a USDZ dataset for the complete compatibility warning.

Warning

Resolving an asset ID to a server-local .ply path is a trusted-caller feature. The file must be reachable by the server process, and there is no upload step, asset-root confinement, or traversal and symlink policy. Do not expose this endpoint to untrusted callers.

Asset editing requires the single-process server. See Asset Editing Under the Pool.

Baked asset edits compared with in-place editing. edit_assets is one of two editing mechanisms. The other is an artifact baked offline with nurec compile-asset-edits, which replaces captured actors with SimReady assets whose states bake as switchable Gaussian clusters. Such scenes are driven per frame with DynamicObject.aspect_state, described under Dynamic Object Types. The two are mutually exclusive per scene: on an edit-baked artifact, edit_assets insert and replace requests fail, because they would reorder layer rows and invalidate the baked cluster addressing. Bake the assets you want into the artifact instead, or serve a plain artifact with a topology-edit-compatible target layer for edit_assets. Unlike edit_assets, baked asset edits and their per-frame aspect_state driving also work under multi-process serving.

Ego Vehicle#


    get_available_ego_masks()
    # Get available ego vehicle mask metadata
    # Returns: AvailableEgoMasksReturn (ego_mask_metadata)

Rendering#


    render_camera(request: CameraRenderRequest)
    # Render one camera. Returns: CameraRenderReturn, with one CameraRenderOutput per requested
    # product, in request order. `render_products` selects color, distance-colorized, and/or
    # semantics-colorized output; several products in one request share a single render pass.

    batch_render_camera(request: BatchCameraRenderRequest)
    # Renders multiple cameras of one scene in a single request. Phase 1 performs the common setup
    # (backend init, actor updates) once; phase 2 renders each camera sequentially.
    # Returns: BatchCameraRenderReturn, with one item per camera.

    render_lidar(request: LidarRenderRequest)
    # Render a LiDAR point cloud from the specified sensor pose
    # Returns: LidarRenderReturn (point_xyzs_buffer, point_intensities_buffer, num_points)

    render_rgb(request: RGBRenderRequest)          # deprecated
    batch_render_rgb(request: BatchRGBRenderRequest)  # deprecated

render_products accepts any subset of the following values, and the response carries one CameraRenderOutput per entry in the same order. Each output holds image_bytes plus per-product metadata such as semantics_class_mapping and distance_colormap_vmax_m.

CameraRenderProduct value

Output

CAMERA_RENDER_PRODUCT_COLOR

Standard RGB render.

CAMERA_RENDER_PRODUCT_DISTANCE_COLORIZED

Metric ray distance from the sensor, colorized with a turbo colormap.

CAMERA_RENDER_PRODUCT_SEMANTICS_COLORIZED

Semantic class colors.

Set overlay_bounding_boxes to draw cuboid wireframes on every requested product, sourced either from the checkpoint’s learned tracks or from the artifact’s sequence_tracks.

Note

render_rgb and batch_render_rgb are deprecated as of 26.08 and still work in this release. Migrate to render_camera with render_products = [CAMERA_RENDER_PRODUCT_COLOR] for identical behavior, then add the distance and semantics products as needed.

Harmonizer. Start the server with either --enable-harmonizer (non-temporal) or --enable-harmonizer-temporal (temporal), then set apply_harmonizer = true on an RGBRenderRequest or CameraRenderRequest to post-process its color output. False or omitted leaves the render unharmonized. If apply_harmonizer is true and the server was not started with a Harmonizer mode enabled, the request fails with FAILED_PRECONDITION. A temporal opt-in requires a valid harmonizer_session_id. When no harmonization is requested and no session ID is provided, the server returns an unharmonized rendering without creating or changing temporal session state. If a session ID is provided, the server resets that session’s history so that a later harmonized request cannot use stale history. Batch requests may opt in per embedded request for the non-temporal model; temporal Harmonizer is not supported in batches.

CUDA IPC (Same-Node) Delivery#


    open_ipc_session(request: OpenIpcSessionRequest)
    # For a client co-located with the server on the same node and same physical GPU. The server
    # pre-allocates and IPC-exports a small pool of GPU frame buffers, sized from
    # requested_width/requested_height/requested_buffer_count.
    # Returns: OpenIpcSessionResponse (session_id, one cudaIpcMemHandle_t per buffer, buffer_size,
    # device_uuid).

    close_ipc_session(request: CloseIpcSessionRequest)
    # Best-effort teardown. Returns: CloseIpcSessionResponse (success, error_message).

After open_ipc_session, set use_cuda_ipc=true and ipc_session_id on RGBRenderRequest or CameraRenderRequest to deliver into one of that session’s buffers instead of image_bytes; the response’s buffer_id identifies which pooled buffer now holds the frame. image_format must be RGB_UINT8_PLANAR when use_cuda_ipc=true — the CUDA IPC path always delivers that format regardless of what’s requested, so any other value aborts INVALID_ARGUMENT. Set released_buffer_id on the next request once the client is done reading a previously-returned buffer (after fencing that read with its own cudaEventSynchronize), so the server can reuse it.

For CameraRenderRequest/render_camera, use_cuda_ipc=true only supports a single CAMERA_RENDER_PRODUCT_COLOR output (an empty render_products list, which already defaults to COLOR, or an explicit render_products=[CAMERA_RENDER_PRODUCT_COLOR]) with overlay_bounding_boxes unset — the response carries exactly one CameraRenderOutput, with format/buffer_id set instead of image_bytes. Any other render_products combination, or a bounding-box overlay, aborts INVALID_ARGUMENT: the IPC path writes straight from the float32 raster into the client’s mapped buffer via a fused pack kernel, so it has no host-visible image to draw an overlay onto and no way to also produce a second, differently-typed product (e.g. DISTANCE_COLORIZED).

batch_render_rgb items may set use_cuda_ipc=true too, freely mixed with JPEG-encoded items in the same batch. Every use_cuda_ipc item in one batch must share the same ipc_session_id — they draw from one session’s buffer pool — and all of their buffers are reserved atomically up front: either the whole batch’s IPC delivery fits in the session’s currently-free buffers (after applying any released_buffer_id releases from the same batch), or the call aborts RESOURCE_EXHAUSTED with none of it delivered. batch_render_camera does not support use_cuda_ipc (aborts INVALID_ARGUMENT); use render_camera for CUDA IPC delivery of a single camera at a time.

This path is not supported under --multiproc-workers (see below); the relevant RPCs abort UNIMPLEMENTED rather than silently ignoring use_cuda_ipc.

Clients must call cudaIpcCloseMemHandle for every mapped buffer before close_ipc_session; otherwise freeing the exported allocation is undefined behavior.

Harmonizer Session Management#


    create_harmonizer_session()
    # Open a session that holds the frame history for one coherent render sequence
    # Returns: HarmonizerSessionId (session_id)

    end_harmonizer_session(request: HarmonizerSessionId)
    # Release a session and its frame history
    # Returns: Empty

These RPCs are available only when the server runs with --enable-harmonizer-temporal. See Temporal Harmonizer Sessions.

Coordinate Conversion Utilities#

The nurec.grpc.pose module provides utility functions for converting between coordinate representations:

This module requires a full source checkout. It is not included in the standalone protocol archive.

from nurec.grpc.pose import (
    grpc_pose_to_tquat,      # gRPC Pose -> 7D tquat [x,y,z,qx,qy,qz,qw]
    tquat_to_grpc_pose,      # 7D tquat -> gRPC Pose
    grpc_pose_to_torch_se3,  # gRPC Pose -> SE3 4x4 torch.Tensor
    se3_to_grpc_pose,        # SE3 4x4 matrix -> gRPC Pose
)

nurec.grpc.serve re-exports all four for backwards compatibility, so older code keeps working. Import from nurec.grpc.pose in new client code: it carries only the pose math, where nurec.grpc.serve pulls in the whole serving stack, including gaussians, rendering, the viewer, and PyTorch Lightning.

Temporal Harmonizer Sessions#

The temporal Harmonizer refines each frame using the frames enhanced before it, which removes flicker and frame-to-frame drift across a rendered sequence. Because it carries state, it renders through an explicit session rather than one-shot calls.

Launch the server with --enable-harmonizer-temporal, then follow this lifecycle.

  1. Call create_harmonizer_session() to get a session_id.

  2. Set apply_harmonizer = true and that value on the harmonizer_session_id field of every CameraRenderRequest (or RGBRenderRequest) in the sequence. A harmonized render that omits it fails with INVALID_ARGUMENT, and an unrecognized value fails with NOT_FOUND.

  3. Call end_harmonizer_session() when the sequence is complete.

The server holds at most --max-harmonizer-sessions open at once and returns RESOURCE_EXHAUSTED beyond that, so always end a session even if the sequence fails partway through.

session = client.create_harmonizer_session(common_pb2.Empty())
try:
    for pose in trajectory:                     # in trajectory order, one at a time
        request = sensorsim_pb2.CameraRenderRequest(
            scene_id=scene_id,
            sensor_pose=pose,
            harmonizer_session_id=session.session_id,
            apply_harmonizer=True,
            render_products=[sensorsim_pb2.CameraRenderProduct.CAMERA_RENDER_PRODUCT_COLOR],
        )
        response = client.render_camera(request)
finally:
    client.end_harmonizer_session(session)

For a color trajectory against a temporal target, the bundled render-grpc client does all of this automatically.

Observe the following restrictions.

  • One session per sequence. A session holds the frame history for a single coherent trajectory. Reusing one session across separate or interleaved sequences corrupts the temporal conditioning.

  • Send frames in order, one at a time. Await each render before sending the next. Concurrent or out-of-order requests build the history incorrectly and are not rejected.

  • Batch rendering is refused. batch_render_camera and batch_render_rgb both abort under the temporal Harmonizer. Use the singular render calls with a session.

  • Inference only. The temporal Harmonizer is not supported for training.

  • Mutually exclusive with --enable-harmonizer. Enable one or the other, not both.

  • Not supported with multi-process serving. See Before You Adopt the Pool.

  • Device placement. serve-grpc does not initialize a distributed process group, so the multi-GPU guard that blocks Harmonizer during distributed training does not apply here, and you do not need to restrict visible GPUs to serve. The temporal model’s TorchScript trace does bake in a device, so if a render fails with a device mismatch, pin the server to one GPU with CUDA_VISIBLE_DEVICES. See Use Harmonizer for Post-Processing.

Example Usage#

Here’s a step-by-step guide to render scenes from USDZ artifacts using the NuRec gRPC API:

Create a Python file with this structure, replacing pass with the snippets from steps 1–6 in order:

import asyncio

async def main():
    # Paste the snippets here, keeping them indented inside main.
    pass

asyncio.run(main())

  1. Connect to the Server:


    import grpc.aio
    from nurec.grpc.protos.sensorsim_pb2_grpc import SensorsimServiceStub

    channel = grpc.aio.insecure_channel("localhost:8080")
    client_service = SensorsimServiceStub(channel)

  1. Discover Available Scenes:


    from nurec.grpc.protos.common_pb2 import Empty
    response = await client_service.get_available_scenes(Empty())
    scene_ids = response.scene_ids
    scene_id = scene_ids[0]  # Choose a registered scene.

  1. (Optional) Read the Server Configuration:


    from nurec.grpc.protos.common_pb2 import Empty
    config_response = await client_service.get_server_config(Empty())
    # config_response.server_config is a dict mapping config key -> string value
    # Use to verify expected defaults or detect parameter mismatches
    for key, value in config_response.server_config.items():
        print(f"{key}: {value}")

  1. Get Scene Information:


    from nurec.grpc.protos.sensorsim_pb2 import AvailableTrajectoriesRequest, AvailableCamerasRequest

    # Get trajectories
    trajectories = await client_service.get_available_trajectories(
        AvailableTrajectoriesRequest(scene_id=scene_id)
    )

    # Get cameras
    cameras = await client_service.get_available_cameras(
        AvailableCamerasRequest(scene_id=scene_id)
    )
    front_wide_camera = next(
        (camera for camera in cameras.available_cameras
         if camera.logical_id == "camera_front_wide_120fov"), None
    )
    if front_wide_camera is None:
        raise ValueError("camera_front_wide_120fov is unavailable")
    trajectory = trajectories.available_trajectories[0].trajectory
    middle_pose = trajectory.poses[len(trajectory.poses) // 2]
    middle_timestamp = middle_pose.timestamp_us

  1. Render an Image:


    from nurec.grpc.protos.sensorsim_pb2 import (
        CameraRenderProduct,
        CameraRenderRequest,
        ImageFormat,
        PosePair,
    )
    from nurec.grpc.protos import common_pb2 as common

    def multiply_quaternions(a, b):
        """Hamilton product, with components ordered x, y, z, w."""
        x, y, z, w = a
        bx, by, bz, bw = b
        return (
            w * bx + x * bw + y * bz - z * by,
            w * by - x * bz + y * bw + z * bx,
            w * bz + x * by - y * bx + z * bw,
            w * bw - x * bx - y * by - z * bz,
        )


    def compose_camera_pose(rig_pose, camera_pose):
        """Compose world-from-rig with rig_to_camera, which stores camera-to-rig."""
        q = rig_pose.quat
        rotation = (q.x, q.y, q.z, q.w)
        norm_squared = sum(value * value for value in rotation)
        if norm_squared == 0:
            raise ValueError("Rig trajectory contains a zero quaternion")
        inverse = tuple(value / norm_squared for value in (-q.x, -q.y, -q.z, q.w))
        offset = camera_pose.vec
        translated = multiply_quaternions(multiply_quaternions(rotation, (offset.x, offset.y, offset.z, 0)), inverse)
        c = camera_pose.quat
        orientation = multiply_quaternions(rotation, (c.x, c.y, c.z, c.w))
        return common.Pose(
            vec=common.Vec3(
                x=rig_pose.vec.x + translated[0],
                y=rig_pose.vec.y + translated[1],
                z=rig_pose.vec.z + translated[2],
            ),
            quat=common.Quat(x=orientation[0], y=orientation[1], z=orientation[2], w=orientation[3]),
        )


    pose = compose_camera_pose(middle_pose.pose, front_wide_camera.rig_to_camera)

    request = CameraRenderRequest(
        scene_id=scene_id,
        resolution_h=300,
        resolution_w=533,  # floor(300 * 1920 / 1080)
        camera_intrinsics=front_wide_camera.intrinsics,
        frame_start_us=middle_timestamp,
        frame_end_us=middle_timestamp + 1,
        sensor_pose=PosePair(
            start_pose=pose,
            end_pose=pose
        ),
        image_format=ImageFormat.JPEG,
        image_quality=95,
        # Request one or more products in a single call. Pass any subset of COLOR,
        # DISTANCE_COLORIZED, and SEMANTICS_COLORIZED; the server returns one
        # CameraRenderOutput per entry, in the same order.
        render_products=[CameraRenderProduct.CAMERA_RENDER_PRODUCT_COLOR],
    )

    response = await client_service.render_camera(request)

    # response.camera_render_outputs[i] holds image_bytes plus per-product metadata
    # (semantics_class_mapping, distance_colormap_vmax_m).
    with open("output.jpg", "wb") as f:
        f.write(response.camera_render_outputs[0].image_bytes)

Note

The server fits the render to the source camera’s aspect ratio, so the returned image rarely matches resolution_w exactly. For the 1920x1080 front-wide camera, a 400x300 request still comes back 533x300, because the width is derived from the requested height. Request the width the aspect ratio implies, as the examples do, and the response matches what you asked for.

Warning

Known issue in 26.10. render_camera accepts a nonfinite sensor_pose component and returns a badly smeared image at transport status OK, so the status code alone does not tell you the render is usable. Setting a single value such as sensor_pose.start_pose.vec.x to NaN is enough to trigger it. Check that every pose component is finite before you send the request.

Server state is not affected: repeating the original valid request returns the expected image, with no restart needed. A fix that rejects nonfinite pose values with INVALID_ARGUMENT, naming the offending field, is targeted for 26.11.

  1. Render Multiple Cameras (Batch):


    from nurec.grpc.protos.sensorsim_pb2 import (
        BatchCameraRenderRequest,
        BatchCameraRenderRequestItem,
        CameraRenderProduct,
        CameraRenderRequest,
        ImageFormat,
        PosePair,
    )

    # Build one BatchCameraRenderRequestItem per camera: camera_name + full CameraRenderRequest
    items = []
    for cam in cameras.available_cameras:
        pose = compose_camera_pose(middle_pose.pose, cam.rig_to_camera)
        camera_request = CameraRenderRequest(
            scene_id=scene_id,
            resolution_h=300,
            resolution_w=533,  # floor(300 * 1920 / 1080)
            camera_intrinsics=cam.intrinsics,
            frame_start_us=middle_timestamp,
            frame_end_us=middle_timestamp + 1,
            sensor_pose=PosePair(
                start_pose=pose,
                end_pose=pose,
            ),
            image_format=ImageFormat.JPEG,
            image_quality=95,
            render_products=[CameraRenderProduct.CAMERA_RENDER_PRODUCT_COLOR],
        )
        items.append(BatchCameraRenderRequestItem(camera_name=cam.logical_id, request=camera_request))

    batch_request = BatchCameraRenderRequest(items=items)
    response = await client_service.batch_render_camera(batch_request)

    for item in response.items:
        if not item.success:
            # Per-camera error (item.error_message is set)
            continue
        with open(f"output_{item.camera_name}.jpg", "wb") as f:
            f.write(item.result.camera_render_outputs[0].image_bytes)

Note

Batch rendering is not available under the temporal Harmonizer. See Temporal Harmonizer sessions.

On this page