Skip to content

Sensor Dump Virtual Annotation Dataset

1. Overview

The 51WORLD virtual annotation dataset primarily contains camera sensor data and LiDAR sensor data generated by SimOne.

  • Camera sensor data mainly includes images and corresponding semantic segmentation, instance segmentation, depth annotations, and object detection annotations.
  • LiDAR sensor data mainly includes point cloud data and corresponding 3D bounding box annotations, semantic segmentation annotations, and instance segmentation annotations.

The 51WORLD virtual annotation dataset is rich and comprehensive, meeting users' research needs for image object detection, point cloud object detection, fusion object detection, optical flow, instance segmentation, semantic segmentation, and depth prediction algorithms.

To help users quickly understand and use the 51WORLD virtual annotation dataset, the following five sections describe how to work with it.

Section 2 describes the coordinate system definitions and the relationships among the coordinate systems in the dataset.

Section 3 describes the dataset's parameter configurations and annotation files.

Section 4 describes the dataset's directory structure.

Section 5 describes dataset synchronization.

Section 6 is an appendix covering the dataset's physical material definitions, user tools, and deep learning examples.

2. Dataset Coordinate System Definitions

All coordinate systems are right-handed Cartesian coordinate systems. The rotation order is intrinsic ZYX (Yaw, Pitch, Roll). The Ego Vehicle coordinate system, LiDAR coordinate system, and world coordinate system have X pointing forward, Y pointing left, and Z pointing up. The camera coordinate system has X pointing right, Y pointing down, and Z pointing forward. The coordinate system diagram is shown below.

Figure 2-1: Coordinate System Diagram

The origins of the LiDAR and camera coordinate systems are both located at the top of the Ego Vehicle and coincide in position. The origin of the Ego Vehicle coordinate system is located on the ground directly below the rear axle center. The diagram is shown below:

Figure 2-2: Coordinate System Position Diagram

3. Dataset Parameters and Annotation File Descriptions

3.1 Dataset Collection Parameters

A dataset parameter file is saved as DumpSettings.json each time a virtual dataset is collected. It is stored in the sensor Dump settings directory under the Dump folder and includes weather and environment parameters, camera intrinsic/extrinsic parameters and characteristics, LiDAR intrinsic/extrinsic parameters, collection time, and other information. The dataset collection parameters are described in the table below.

Table 3-1: DumpSettings.json Parameter Descriptions

Parameter Field Unit Description
simulator \ Simulator name. This dataset is generated by SimOne.
version \ SimOne version number.
weather timeOfDay uses 24-hour format; e.g., 900 = 9:00, 1450 = 14:30. cloudDensity, rainDensity, snowDensity, fogDensity, humidity, dirtiness, adhesion are all continuous values in the range 0–1; higher values indicate greater intensity. Weather and time parameters, including: timeOfDay (scene time), cloudDensity, rainDensity, snowDensity, fogDensity, humidity (road surface moisture), dirtiness (road surface dirt), adhesion (road surface friction).
camera pos unit: m; rot unit: rad. Camera intrinsic and extrinsic parameters. pos and rot represent the camera's position and rotation relative to the Ego Vehicle coordinate system; height and width are the image dimensions; fov is the camera's horizontal field of view; cx, cy, fx, fy are the camera calibration intrinsic parameters; exposure, tint, saturation, and other parameters simulate camera characteristics such as exposure, color temperature, and saturation.
lidar pos unit: m; rot unit: rad; range unit: m; horizontalResolution, leftAngle, rightAngle, verticalAngles unit: degrees. LiDAR intrinsic and extrinsic parameters. pos and rot represent the LiDAR's position and rotation relative to the Ego Vehicle coordinate system; channels is the number of LiDAR lines; horizontalResolution is the horizontal resolution; leftAngle and rightAngle define the sensing range; verticalAngles lists the emission angles for each LiDAR beam; range is the detection distance.
createTime Machine time. Records the time at which this dataset was created.

3.2 Camera Object Annotation File Description

Camera object annotation files are stored as CameraInfo.json in the Dump folder at the configured camera path. These files save camera position information and 2D/3D annotation information for detected objects. The camera object annotation file parameters are described below.

Table 3-2: Per-Frame Record Parameter Information

Reference Field Unit Description
pos m Camera position in the world coordinate system (x, y, z)
rot rad Camera rotation in the world coordinate system
vel m/s Camera velocity in the world coordinate system
localACC m/s² IMU acceleration in the Ego Vehicle coordinate system
localAngVel rad/s IMU angular velocity in the Ego Vehicle coordinate system
bboxes \ Collection of 2D bounding boxes for all objects below the occlusion ratio threshold
bboxesCulled \ Collection of 2D bounding boxes for all objects above the occlusion ratio threshold
bboxes3D \ Collection of 3D bounding boxes for all objects

Table 3-3: Per-Object Parameters for bboxes and bboxesCulled

Reference Field Unit Description
id \ Object ID
type \ Object type
bbox pixel Object coordinates in the image coordinate system (upper-left x and y, lower-right x and y)
obbox pixel Oriented coordinates of the object in the image coordinate system (center, width, height, angle)
pixelRate ratio, range 0–1 Unoccluded pixel area / total BBOX rectangle area of the model
rectRate ratio, range 0–1 Minimum bounding rectangle area of unoccluded pixels / total BBOX rectangle area of the model

Table 3-4: Per-Object Parameters for bboxes3D

Reference Field Unit Description
id \ Object ID
type \ Object type
pos m Object position in the world coordinate system
rot rad Object rotation in the world coordinate system
size m Object dimensions (length, width, height) in the world coordinate system
vel m/s Object velocity in the world coordinate system
localAcc m/s² IMU acceleration of the object in the Ego Vehicle coordinate system
localAngVel rad/s IMU angular velocity of the object in the Ego Vehicle coordinate system
relativePos m Object position relative to the camera coordinate system
relativeRot rad Object rotation relative to the camera coordinate system

3.3 LiDAR Object Annotation File Description

LiDAR object annotation files are stored as LidarInfo.json in the Dump folder at the configured LiDAR path. These files save LiDAR pose information and 3D annotation information. The LiDAR object annotation file parameters are described below.

Table 3-5: Per-Frame Record Parameter Information

Reference Field Unit Description
pos m LiDAR position in the world coordinate system (x, y, z)
rot rad LiDAR rotation in the world coordinate system
vel m/s LiDAR velocity in the world coordinate system
localAcc m/s² IMU acceleration in the Ego Vehicle coordinate system
localAngVel rad/s IMU angular velocity in the Ego Vehicle coordinate system
bboxes3D Collection of 3D bounding boxes for all objects

Table 3-6: Per-Object Parameters for bboxes3D

Reference Field Unit Description
id \ Object ID
type \ Semantic type of the object
pos m Object position in the world coordinate system
rot rad Object rotation in the world coordinate system
size m Object dimensions (length, width, height) in the world coordinate system
vel m/s Object velocity in the world coordinate system
localAcc m/s² IMU acceleration of the object in the Ego Vehicle coordinate system
localAngVel rad/s IMU angular velocity of the object in the Ego Vehicle coordinate system
relativePos m Object position relative to the LiDAR coordinate system
relativeRot rad Object rotation relative to the LiDAR coordinate system

3.4 Millimeter-Wave Radar Object Annotation File Description

Millimeter-wave radar Dump files are stored as RadarInfo.json in the Dump folder at the path configured in the web interface. These files save the radar's installation position and angle information, as well as detected object information. The specific Dump file fields are described below.

Table 3-7: Per-Frame Record Parameter Information

Reference Field Unit Description
pos m Millimeter-wave radar position in the world coordinate system (x, y, z)
rot rad Millimeter-wave radar rotation in the world coordinate system

Table 3-8: Per-Object Parameters for bboxes3D

Reference Field Unit Description
id \ Object ID
subId \ Object sub-ID. If the same object has multiple points, their subId values differ.
type \ Object type
pos m Object position (x, y, z)
vel m/s Object absolute velocity (x, y, z)
range m Relative distance to the object
angle 0–2π Object angle (angle between the object and the Ego Vehicle heading)
snr dB Object signal-to-noise ratio
rangerate m/s Relative velocity of the object with respect to the Ego Vehicle along the Ego Vehicle heading
rsc \ Radar cross-section of the object
probability 0–1 Object confidence score

4. Dataset Directory Structure Description

The 51WORLD virtual annotation dataset contains a wide variety of data, meeting users' research needs for image object detection, point cloud object detection, fusion object detection, optical flow, instance segmentation, semantic segmentation, and depth prediction algorithms. The overall dataset directory structure is as follows. Users can also download specific dataset components based on their requirements.

    SimOne
        |--- train
            |--- scene1
                |---image_label 
                |---pcd_label 
                |---pcd_bin  
                |---image
                |---image_segmentation
                |---depth
                |---image_instance
                |---flow_flagbit_forward
                |---flow_groundtruth_forward
                |---flow_flagbit
                |---flow_groundtruth
                |---flow_panoptic
                |---video
                |---DumpSettings.json
            |--- scene2
            ...
        |--- test
            |---scene1
                |---pcd_bin
                |---image
                |---video
                |---DumpSettings.json
            |--- scene2
            ...

The descriptions for each directory and file are as follows.

Table 4-1: Folder and File Descriptions

Filename / Folder Name Description
scene Used to record the data release time or scene information
image_label Stores image object annotation files
pcd_label Stores point cloud object annotation labels
pcd_bin Stores binary point cloud data, including x, y, z, and intensity
image Stores simulated image data
Image_segmentation Stores image semantic segmentation data; pixel values 1–31 each represent one category
depth Stores depth maps
image_instance Stores image instance segmentation maps; each color represents one class
flow_panoptic Stores panoptic segmentation images; categories are distinguished by color
video Stores video and labels; labels correspond to video frames by timestamp
DumpSettings.json Stores dataset configuration parameters at the time of download, such as camera intrinsic/extrinsic parameters

5. Dataset Synchronization

The sensor dataset uses offline synchronization. When sensors output data at the same or multiple frame rates, all sensor outputs are strictly time-synchronized. In the 51WORLD virtual annotation dataset, every type of data is fully synchronized — precise to each vehicle's position and heading and each pedestrian's pose and action. Using SimOne's sensor dataset synchronization mechanism, camera and LiDAR data can also be fully synchronized, as demonstrated in the LiDAR simulation output shown earlier. Based on fully synchronized camera and LiDAR datasets, it is easier to test and train perception fusion algorithms.

6. Appendix

6.1 User Tools

To help users work with the dataset, we provide two tools: a data loading tool and a KITTI conversion tool. See the user_tools folder for details.

6.2 Physical Materials

This dataset annotates 31 physical materials covering dynamic and static obstacles, buildings, and environmental elements. The physical materials and their corresponding IDs and RGB colors are listed in the table below.

Table 6-1: 51WORLD Dataset Physical Materials

type Physical Material Assets RGB
1 Foliage Trees, shrubs, relatively tall flowers 107,142,35
2 Building Various types of buildings 70,70,70
3 Road Driving lanes 128,64,128
4 Pedestrian Pedestrians and small items on pedestrians (e.g., phones, backpacks, suitcases) 220,20,60
5 Pole Traffic sign or traffic light poles, streetlights with poles, other ground-level poles 153,153,153
6 Car Passenger cars 0,0,142
7 Static Unclassified static objects: roadside bus stops, phone booths, etc. 0,0,0
8 Bicycle Dynamic bicycles on the road 119,11,32
9 Fence Fences, building enclosures 190,153,153
10 Sky Sky 70,130,180
11 SideWalk Sidewalks 244,35,232
12 RoadMark Lane markings 240,240,240
13 TrafficSign Traffic signs, directional signs 220,220,0
14 Wall Boundary walls 102,102,156
15 TrafficLight Traffic lights 250,170,30
16 Terrain Grass, sand, dirt, roadside flower bed ground, low flowers in flower beds 152,251,152
17 Rider People on bicycles, people on motorcycles 255,0,0
18 Truck Trucks, concrete mixers, box trucks 0,0,70
19 Bus Coaches, buses 0,60,100
20 SpecialVehicle Special vehicles: police cars, ambulances, fire trucks, trains, light rail 0,80,100
21 Motorcycle Motorcycles, electric scooters 0,0,230
22 Dynamic Unclassified dynamic objects (e.g., small animals); also used for movable objects such as temporary roadside furniture, pedestrians' suitcases, strollers 111,74,0
23 GuardRail Traffic barriers 180,165,180
24 Ground Other flat ground, water surfaces 81,0,81
25 Bridge Bridges, overpasses, pedestrian bridges 150,100,100
26 SpeedLimitSign Speed limit signs 220,220,0
27 StaticBicycle Static bicycles on the roadside 169,11,32
28 Parking Parking lots, roadside parking areas 250,170,160
29 RoadObstacle Static obstacles on the road: traffic cones, water barriers, pylons, dividers, etc. 230,150,140
30 Tunnel Tunnels 150,120,90
31 TrashCan Trash cans 151,124,0

Baidu Netdisk: https://pan.baidu.com/s/1FDeb3mcY8J79A6o6IydKOQ

Extraction code: 5151

6.4 Algorithm Examples

This dataset includes three simple algorithm examples: an image object detection algorithm, a LiDAR object detection algorithm, and a fusion object detection algorithm. The image object detection algorithm is an unofficial implementation of YOLOv2; the LiDAR object detection algorithm is an unofficial implementation of PIXOR; and the fusion object detection algorithm is the MVX-NET algorithm from the mmdetection3d framework. Users can select an algorithm example to develop and optimize based on their requirements. For details, see: