← Back to packages
Package sensorsim
Services
SensorsimService defines the RPC interface for interacting with the sensor simulator.
It provides methods for rendering RGB images and lidar data, querying available scenes,
cameras, trajectories, and ego masks, as well as shutting down the simulator.
Request:
sensorsim.BatchRGBRenderRequest
Response:
sensorsim.BatchRGBRenderReturn
Deprecated: use batch_render_camera instead. Render a batch of RGB requests.
Request:
common.Empty
Response:
sensorsim.HarmonizerSessionId
Open a temporal-harmonizer session holding the frame history for ONE coherent render sequence (a single trajectory). Use one session per sequence — reusing it across separate or interleaved sequences corrupts the temporal conditioning. Send the session's frames in trajectory order, one at a time (await each render before sending the next); concurrent or out-of-order requests build the history wrongly and are not rejected. Returns FAILED_PRECONDITION unless the server runs --enable-harmonizer-temporal.
Request:
sensorsim.HarmonizerSessionId
Response:
common.Empty
End a temporal Harmonizer session and release its stored state.
Request:
sensorsim.ExternalAssetObjectsRequest
Response:
sensorsim.ExternalAssetObjectsReturn
Return the external asset objects available in a scene.
Request:
sensorsim.EditAssetsRequest
Response:
sensorsim.EditAssetsResponse
Apply the requested asset replacements and insertions to a scene.
Request:
sensorsim.RestoreModelParametersRequest
Response:
common.Empty
Restore the original model parameters after asset editing.
Request:
sensorsim.AvailableDynamicObjectsRequest
Response:
sensorsim.AvailableDynamicObjectsReturn
Return dynamic objects and their trajectories for a scene.
Return the running server configuration.
Request:
ipc.OpenIpcSessionRequest
Response:
ipc.OpenIpcSessionResponse
CUDA IPC session lifecycle. After open_ipc_session, render_rgb may be called with use_cuda_ipc=true to deliver the payload into one of the server-owned, pre-allocated buffers exported as cudaIpcMemHandle_t bytes in the open response.
Request:
ipc.CloseIpcSessionRequest
Response:
ipc.CloseIpcSessionResponse
Close a same-node CUDA IPC session and release its server-owned buffers.
Deprecated: use render_camera instead. Renders an RGB image based on the provided request parameters.
Renders lidar data based on the provided request parameters.
Returns the version of the sensor simulator.
Returns the list of available scenes.
Returns cached / currently resident scene instance counts from BackendCache. Unlike get_available_scenes, scenes may appear with loaded_instance_count > 1 when multiple cached instances exist for the same scene.
Returns the list of available cameras for a given scene.
Shuts down the sensor simulator.
Returns the list of available trajectories for a given scene.
Returns the list of available ego masks.
Messages
Metadata for a single available camera.
|
Field |
Type |
Description |
| 1 |
intrinsics
|
sensorsim.CameraSpec
|
Camera intrinsic and extrinsic parameters. |
| 2 |
rig_to_camera
|
common.Pose
|
The pose from the rig to the camera. |
| 3 |
logical_id
|
string
|
Logical ID for the camera. |
| 4 |
trajectory_idx
|
uint32
|
Index of the trajectory associated with this camera. |
AvailableCamerasRequest specifies the scene for which available cameras are requested.
|
Field |
Type |
Description |
| 1 |
scene_id
|
string
|
The scene ID for which to list available cameras. |
AvailableCamerasReturn contains the list of available cameras for a scene.
AvailableEgoMasksReturn contains metadata for all available ego masks.
AvailableTrajectoriesRequest specifies the scene for which available trajectories are requested.
|
Field |
Type |
Description |
| 1 |
scene_id
|
string
|
The scene ID for which to list available trajectories. |
AvailableTrajectoriesReturn contains the list of available trajectories for a scene.
Metadata for a single available trajectory.
|
Field |
Type |
Description |
| 1 |
trajectory_idx
|
uint32
|
Index of the trajectory. |
| 2 |
trajectory
|
common.Trajectory
|
The trajectory data. |
Batch camera render response
|
Field |
Type |
Description |
| 1 |
camera_name |
string |
Matches request camera_name |
| 2 |
result |
sensorsim.CameraRenderReturn |
|
| 3 |
success |
bool |
|
| 4 |
error_message |
string |
|
RGB color for bounding box wireframe edges. Unset = server default (0, 255, 0).
Each channel is 0-255; values outside this range are rejected with INVALID_ARGUMENT.
|
Field |
Type |
Description |
| 1 |
r |
uint32 |
|
| 2 |
g |
uint32 |
|
| 3 |
b |
uint32 |
|
BivariateWindshieldModelParameters describes the parameters for a bivariate polynomial windshield distortion model.
|
Field |
Type |
Description |
| 1 |
reference_poly
|
sensorsim.BivariateWindshieldModelParameters.ReferencePolynomial
|
The reference polynomial type. |
| 2 |
horizontal_poly
|
repeated
double
|
Coefficients for the horizontal polynomial. |
| 3 |
vertical_poly
|
repeated
double
|
Coefficients for the vertical polynomial. |
| 4 |
horizontal_poly_inverse
|
repeated
double
|
Coefficients for the inverse horizontal polynomial. |
| 5 |
vertical_poly_inverse
|
repeated
double
|
Coefficients for the inverse vertical polynomial. |
|
Field |
Type |
Description |
| 1 |
image_bytes |
bytes |
|
| 2 |
image_height |
uint32 |
|
| 3 |
image_width |
uint32 |
|
| 4 |
render_product |
sensorsim.CameraRenderProduct |
Which product this buffer carries. Matches the corresponding entry in the
request's render_products list (or COLOR for an empty request list). |
| 5 |
semantics_class_mapping |
repeated sensorsim.SemanticsClassEntry |
when output is semantics |
| 6 |
distance_colormap_vmax_m |
optional float |
when output is distance |
|
Field |
Type |
Description |
| 1 |
scene_id |
string |
|
| 2 |
resolution_h |
uint32 |
|
| 3 |
resolution_w |
uint32 |
|
| 4 |
camera_intrinsics |
sensorsim.CameraSpec |
|
| 5 |
frame_start_us |
fixed64 |
|
| 6 |
frame_end_us |
fixed64 |
|
| 7 |
sensor_pose |
sensorsim.PosePair |
|
| 8 |
dynamic_objects |
repeated sensorsim.DynamicObject |
|
| 9 |
image_format |
sensorsim.ImageFormat |
|
| 10 |
image_quality |
float |
|
| 11 |
insert_ego_mask |
bool |
|
| 12 |
ego_mask_id |
sensorsim.EgoMaskId |
if `insert_ego_mask` is true, `ego_mask_id` has to correspond to one of the ego masks in the AvailableEgoMasksReturn |
| 13 |
render_products |
repeated sensorsim.CameraRenderProduct |
Image products to render. Empty list defaults to a single COLOR output. Entries must be
unique; the server returns one CameraRenderOutput per requested product in the same order. |
| 14 |
overlay_bounding_boxes |
sensorsim.OverlayBoundingBoxSource |
Draw 3D cuboid wireframes on every requested output (color, distance, semantics);
selects the source of tracks. Set to UNSPECIFIED to disable. |
| 15 |
distance_colormap_vmax_m |
optional float |
Upper bound of the distance colormap (meters). Unset = server default (80 m). |
| 16 |
bbox_color |
optional sensorsim.BboxColor |
RGB color for bounding box wireframe edges. Unset = server default (0, 255, 0). |
| 17 |
harmonizer_session_id |
optional string |
On a server with temporal Harmonizer enabled, this field is required when a colour request
sets apply_harmonizer true; the render uses the session's HarmonizerState. When
apply_harmonizer is false or unset, supplying this field resets that session's history so a
later opt-in starts a fresh coherent sequence rather than conditioning on stale frames. If
the server is non-temporal, this field is silently ignored.
The id must identify a single coherent render sequence (see create_harmonizer_session).
Only meaningful for the CAMERA_RENDER_PRODUCT_COLOR product. |
| 21 |
apply_harmonizer |
optional bool |
Request Harmonizer postprocessing for this render. Unset or false leaves the color output
unharmonized. On a temporal server, apply_harmonizer=true requires harmonizer_session_id. When
apply_harmonizer is false or unset, a supplied ID resets that session; without an ID, no
temporal session state changes. If apply_harmonizer is true and the server was not started
with a Harmonizer mode enabled, the request fails with FAILED_PRECONDITION.
Only meaningful for CAMERA_RENDER_PRODUCT_COLOR. |
CameraSpec describes the full specification of a camera, including intrinsics, extrinsics, and distortion models.
|
Field |
Type |
Description |
| 1 |
temporary_camera_spec
|
string
|
Deprecated: Temporary camera spec string. |
| 5 |
logical_id
|
string
|
Logical ID for the camera (user-defined). |
| 6 |
trajectory_idx
|
uint32
|
Index of the trajectory associated with this camera. |
| 7 |
resolution_h
|
uint32
|
Image height in pixels. |
| 8 |
resolution_w
|
uint32
|
Image width in pixels. |
| 9 |
shutter_type
|
sensorsim.ShutterType
|
The shutter type for the camera. |
|
|
oneof external_distortion
|
|
10
|
bivariate_windshield_model_param
|
sensorsim.BivariateWindshieldModelParameters
|
Bivariate windshield model parameters. |
|
|
oneof camera_param
|
|
2
|
ftheta_param
|
sensorsim.FthetaCameraParam
|
F-theta camera parameters. |
|
3
|
opencv_pinhole_param
|
sensorsim.OpenCVPinholeCameraParam
|
OpenCV pinhole camera parameters. |
|
4
|
opencv_fisheye_param
|
sensorsim.OpenCVFisheyeCameraParam
|
OpenCV fisheye camera parameters. |
DynamicObject represents a dynamic object in the scene with a unique track ID and pose pair.
|
Field |
Type |
Description |
| 1 |
track_id
|
string
|
Unique identifier for the dynamic object. |
| 2 |
pose_pair
|
sensorsim.PosePair
|
The pose pair for the dynamic object. |
EgoMaskId uniquely identifies an ego mask by camera and rig configuration.
|
Field |
Type |
Description |
| 1 |
camera_logical_id
|
string
|
The logical ID of the camera (e.g., camera_front_wide_120fov). |
| 2 |
rig_config_id
|
string
|
The rig configuration ID (e.g., hyperion 8.0, 8.1). |
Metadata for a single ego mask.
|
Field |
Type |
Description |
| 1 |
ego_mask_id
|
sensorsim.EgoMaskId
|
The unique identifier for the ego mask.
Future: extra metadata like nominal camera_to_rig pose. |
FthetaCameraParam describes the parameters for an F-theta camera model.
|
Field |
Type |
Description |
| 1 |
principal_point_x
|
double
|
The x-coordinate of the principal point in pixels. |
| 2 |
principal_point_y
|
double
|
The y-coordinate of the principal point in pixels. |
| 3 |
reference_poly
|
sensorsim.FthetaCameraParam.PolynomialType
|
The reference polynomial type for the model. |
| 4 |
pixeldist_to_angle_poly
|
repeated
double
|
Coefficients for the pixel distance to angle polynomial. |
| 5 |
angle_to_pixeldist_poly
|
repeated
double
|
Coefficients for the angle to pixel distance polynomial. |
| 6 |
max_angle
|
double
|
The maximum angle (in radians or degrees, as defined by the model). |
LidarRenderRequest specifies the parameters for rendering lidar data.
|
Field |
Type |
Description |
| 1 |
scene_id
|
string
|
The scene ID to render. |
| 2 |
lidar_config
|
sensorsim.LidarSpec
|
Lidar configuration parameters. |
| 3 |
frame_start_us
|
fixed64
|
Start time of the frame in microseconds. |
| 4 |
frame_end_us
|
fixed64
|
End time of the frame in microseconds. |
| 5 |
sensor_pose
|
sensorsim.PosePair
|
The pose of the sensor for this frame. |
| 6 |
dynamic_objects
|
repeated
sensorsim.DynamicObject
|
List of dynamic objects present in the scene. |
LidarRenderReturn contains the rendered lidar point cloud and intensities.
|
Field |
Type |
Description |
| 1 |
point_xyzs
|
repeated
float
|
The xyz coordinates of the lidar points in the end-of-spin lidar space.
[x1, y1, z1, x2, y2, z2, ...] |
| 2 |
point_intensities
|
repeated
float
|
The intensity of the points (range [0, 1]). |
LidarSpec describes the configuration of a lidar device.
Per-scene cached backend instance counts.
|
Field |
Type |
Description |
| 1 |
scene_id |
string |
Scene ID for this group of cached instances. |
| 2 |
loaded_instance_count |
uint64 |
Cached instances for this scene: active + reusable. |
| 3 |
reusable_instance_count |
uint64 |
Spare instances for this scene available for checkout without loading. |
Snapshot of cached scene instances and global cache capacity.
|
Field |
Type |
Description |
| 1 |
scenes |
repeated sensorsim.LoadedSceneEntry |
Per-scene counts, sorted by scene_id. |
| 2 |
loaded_instance_capacity |
uint64 |
Global cached-instance capacity from BackendCache.maxsize. |
OpenCVFisheyeCameraParam describes the parameters for an OpenCV fisheye camera model.
|
Field |
Type |
Description |
| 1 |
principal_point_x
|
double
|
The x-coordinate of the principal point in pixels. |
| 2 |
principal_point_y
|
double
|
The y-coordinate of the principal point in pixels. |
| 3 |
focal_length_x
|
double
|
The x-component of the focal length in pixels. |
| 4 |
focal_length_y
|
double
|
The y-component of the focal length in pixels. |
| 5 |
radial_coeffs
|
repeated
double
|
Radial distortion coefficients. |
| 6 |
max_angle
|
double
|
The maximum angle (in radians or degrees, as defined by the model). |
OpenCVPinholeCameraParam describes the parameters for an OpenCV pinhole camera model.
PosePair represents the poses at the start and end of a frame.
|
Field |
Type |
Description |
| 1 |
start_pose
|
common.Pose
|
The pose at the start of the frame. |
| 2 |
end_pose
|
common.Pose
|
The pose at the end of the frame. |
Deprecated: use CameraRenderRequest instead. RGBRenderRequest specifies the parameters for rendering an RGB image.
|
Field |
Type |
Description |
| 1 |
scene_id
|
string
|
The scene ID to render. |
| 2 |
resolution_h
|
uint32
|
Image height in pixels. |
| 3 |
resolution_w
|
uint32
|
Image width in pixels. |
| 4 |
camera_intrinsics
|
sensorsim.CameraSpec
|
Camera intrinsic and extrinsic parameters. |
| 5 |
frame_start_us
|
fixed64
|
Start time of the frame in microseconds. |
| 6 |
frame_end_us
|
fixed64
|
End time of the frame in microseconds. |
| 7 |
sensor_pose
|
sensorsim.PosePair
|
The pose of the sensor for this frame. |
| 8 |
dynamic_objects
|
repeated
sensorsim.DynamicObject
|
List of dynamic objects present in the scene. |
| 9 |
image_format
|
sensorsim.ImageFormat
|
The image format to use for the output. |
| 10 |
image_quality
|
float
|
The quality of the output image (0.0-1.0 for lossy formats). |
| 11 |
insert_ego_mask
|
bool
|
If false, no ego mask will be applied. |
| 12 |
ego_mask_id
|
sensorsim.EgoMaskId
|
If insert_ego_mask is true, this specifies which ego mask to use. |
| 20 |
harmonizer_session_id
|
optional
string
|
On a server with temporal Harmonizer enabled, this field is required when apply_harmonizer
is true; the render uses the session's HarmonizerState. When apply_harmonizer is false or
unset, supplying this field resets that session's history so a later opt-in starts a fresh
coherent sequence rather than conditioning on stale frames. If the server is non-temporal,
this field is silently ignored.
The id must identify a single coherent render sequence (see create_harmonizer_session). |
| 21 |
apply_harmonizer
|
optional
bool
|
Request Harmonizer postprocessing for this render. Unset or false leaves the color output
unharmonized. On a temporal server, apply_harmonizer=true requires harmonizer_session_id. When
apply_harmonizer is false or unset, a supplied ID resets that session; without an ID, no
temporal session state changes. If apply_harmonizer is true and the server was not started
with a Harmonizer mode enabled, the request fails with FAILED_PRECONDITION. |
Deprecated: use CameraRenderReturn instead. RGBRenderReturn contains the rendered RGB image bytes.
|
Field |
Type |
Description |
| 1 |
image_bytes
|
bytes
|
The rendered image bytes (encoded as specified in the request). |
Per-class metadata when render_product is CAMERA_RENDER_PRODUCT_SEMANTICS_COLORIZED (index, name, display color 0-255)
|
Field |
Type |
Description |
| 1 |
class_index |
int32 |
|
| 2 |
class_name |
string |
|
| 3 |
color_r |
uint32 |
|
| 4 |
color_g |
uint32 |
|
| 5 |
color_b |
uint32 |
|
Enums
Which image product to render and return. COLOR is the zero/default value, so a
default-constructed CameraRenderProduct is a legal (and natural) request.
| Name |
Number |
Description |
CAMERA_RENDER_PRODUCT_COLOR |
0 |
|
CAMERA_RENDER_PRODUCT_DISTANCE_COLORIZED |
1 |
metric ray distance (||point - camera|| in meters), turbo colormap |
CAMERA_RENDER_PRODUCT_SEMANTICS_COLORIZED |
2 |
semantic class colors |
ImageFormat enumerates the supported image formats for RGB rendering.
| Name |
Number |
Description |
UNDEFINED |
0 |
|
PNG |
1 |
|
JPEG |
2 |
|
JPEG2000 |
3 |
AlpaSim currently only supports PNG and JPEG. |
RGB_UINT8_PLANAR |
4 |
|
AVC |
5 |
|
AV1 |
6 |
|
LidarDeviceType enumerates the supported lidar device types.
| Name |
Number |
Description |
PANDAR128 |
0 |
|
AT128 |
1 |
|
Source of tracks for overlaying bounding boxes on output camera images
UNSPECIFIED = no bbox overlay (default)
MODEL = Learned/calibrated tracks in the checkpoint.
ARTIFACT = sequence_tracks.json in the USDZ.
| Name |
Number |
Description |
OVERLAY_BOUNDING_BOX_SOURCE_UNSPECIFIED |
0 |
|
OVERLAY_BOUNDING_BOX_SOURCE_MODEL |
1 |
|
OVERLAY_BOUNDING_BOX_SOURCE_ARTIFACT |
2 |
|
The type of polynomial used for the F-theta model.
| Name |
Number |
Description |
UNKNOWN |
0 |
Unknown polynomial type |
PIXELDIST_TO_ANGLE |
1 |
Polynomial mapping pixeldistances-to-angles ("backward" polynomial) |
ANGLE_TO_PIXELDIST |
2 |
Polynomial mapping angles-to-pixeldistances ("forward" polynomial) |
The reference polynomial direction for the model.
| Name |
Number |
Description |
FORWARD |
0 |
|
BACKWARD |
1 |
|
ShutterType enumerates the possible shutter types for a camera sensor.
| Name |
Number |
Description |
UNKNOWN |
0 |
Unknown shutter type |
ROLLING_TOP_TO_BOTTOM |
1 |
Rolling shutter from top to bottom of the imager |
ROLLING_LEFT_TO_RIGHT |
2 |
Rolling shutter from left to right of the imager |
ROLLING_BOTTOM_TO_TOP |
3 |
Rolling shutter from bottom to top of the imager |
ROLLING_RIGHT_TO_LEFT |
4 |
Rolling shutter from right to left of the imager |
GLOBAL |
5 |
Instantaneous global shutter (no rolling shutter) |
