Generate RoadRunner Scenario from Recorded Sensor Data

This example uses:

This example shows how to generate a RoadRunner scenario containing actor trajectories from recorded Global Positioning System (GPS) data and preprocessed actor track list.

RoadRunner Scenario is an interactive editor that enables you to design scenarios for simulating and testing automated driving systems. Generating scenarios from recorded vehicle data enables you to mimic real-world driving scenarios and improve the test coverage of automated driving systems.

This figure shows these steps.

Smooth GPS data and format actor track list
Reconstruct ego-vehicle trajectories
Extract non-ego actor properties from the track list
Export actor trajectories to CSV files
Import map in RoadRunner with same GeoReference
Import CSV actor trajectories in RoadRunner
Simulate RoadRunner Scenario

Load Sensor Data

Download a ZIP file containing a subset of sensor data from the PandaSet data set, and then unzip the file. This file contains GPS data, an actor track list, and camera information. In this example, you use the camera data for visual validation of the generated scenario.

dataFolder = tempdir;
dataFilename = "PandasetSensorData_23a.zip";
url = "https://ssd.mathworks.com/supportfiles/driving/data/" + dataFilename;
filePath = fullfile(dataFolder,dataFilename);
if ~isfile(filePath)
    websave(filePath,url)
end
unzip(filePath,dataFolder)
dataset = fullfile(dataFolder,"PandasetSensorData");
data = load(fullfile(dataset,"sensorData.mat"));

Load the GPS data into the workspace.

gpsData = data.GPSData;

gpsData is a table with these columns:

timeStamp — Time, in seconds, at which the GPS data was collected.
latitude — Latitude coordinate value of the ego vehicle. Units are in degrees.
longitude — Longitude coordinate value of the ego vehicle. Units are in degrees.
altitude — Altitude coordinate value of the ego vehicle. Units are in meters.

Display the first five entries of gpsData.

gpsData(1:5,:)

ans=5×4 table
    timeStamp     latitude    longitude    altitude
    __________    ________    _________    ________

    1.5576e+09     37.374      -122.06      42.858 
    1.5576e+09     37.374      -122.06      42.858 
    1.5576e+09     37.374      -122.06      42.854 
    1.5576e+09     37.374      -122.06      42.849 
    1.5576e+09     37.374      -122.06      42.848

Load the actor track list data. Alternatively, you can generate actor track list by processing raw camera or lidar sensor data. For more information on how to generate actor track list from camera data, see the Extract Vehicle Track List from Recorded Camera Data for Scenario Generation example. For generating track list from lidar data, see the Extract Vehicle Track List from Recorded Lidar Data for Scenario Generation example.

tracklist = data.ActorTracklist

tracklist = 
  actorTracklist with properties:

         TimeStamp: [400×1 double]
          TrackIDs: {400×1 cell}
          ClassIDs: {400×1 cell}
          Position: {400×1 cell}
         Dimension: {400×1 cell}
       Orientation: {400×1 cell}
          Velocity: []
             Speed: []
         StartTime: 1.5576e+09
           EndTime: 1.5576e+09
        NumSamples: 400
    UniqueTrackIDs: [20×1 string]

tracklist is actorTracklist object that contains the track list data.

Read and display the first five entries of track list data.

sampleData = readData(tracklist, RowIndices = (1:5)')

sampleData=5×2 table
    TimeStamp      ActorInfo  
    __________    ____________

    1.5576e+09    {6×1 struct}
    1.5576e+09    {6×1 struct}
    1.5576e+09    {6×1 struct}
    1.5576e+09    {6×1 struct}
    1.5576e+09    {6×1 struct}

Display ActorInfo for the first timestamp.

sampleData.ActorInfo{1,1}

ans=6×1 struct array with fields:
    TrackID
    ClassID
    Position
    Dimension
    Yaw
    Pitch
    Roll
    Speed
    Velocity

Load the camera data recorded from a forward-facing monocular camera mounted on the ego vehicle.

cameraData = data.CameraData;

The camera data is a table with two columns:

timeStamp — Time, in seconds, at which the image data was captured.
fileName — Filenames of the images in the data set.

The images are located in the Camera folder in the dataset directory. Create a table that contains the file paths of these images for each timestamp by using the helperUpdateTable function.

imageFolder = "Camera";
cameraData  = helperUpdateTable(cameraData,dataset,imageFolder);

Display the first five entries of cameraData.

cameraData(1:5,:)

ans=5×2 table
    timeStamp                                       filePath                                   
    __________    _____________________________________________________________________________

    1.5576e+09    {["C:\Users\richeekr\AppData\Local\Temp\PandasetSensorData\Camera\0001.jpg"]}
    1.5576e+09    {["C:\Users\richeekr\AppData\Local\Temp\PandasetSensorData\Camera\0002.jpg"]}
    1.5576e+09    {["C:\Users\richeekr\AppData\Local\Temp\PandasetSensorData\Camera\0003.jpg"]}
    1.5576e+09    {["C:\Users\richeekr\AppData\Local\Temp\PandasetSensorData\Camera\0004.jpg"]}
    1.5576e+09    {["C:\Users\richeekr\AppData\Local\Temp\PandasetSensorData\Camera\0005.jpg"]}

Remove redundant data from workspace to save memory.

clear data

Crop and Preprocess Sensor Data

Crop the GPS, actor track list, and camera data relative to the GPS timestamp range by using the helperCropData function.

startTime = gpsData.timeStamp(1);
endTime = gpsData.timeStamp(end);

% Pack all the tables in a cell array. 
recordedData = {gpsData,tracklist,cameraData};

% Crop the data.
recordedData = helperCropData(recordedData,startTime,endTime);

The timestamp values of the recorded data set are in the POSIX® format, which Scenario Builder for Automated Driving Toolbox™ supports. Use the helperNormTimeInSecs function to normalize the timestamps using these arguments:

scale — Scale by which to convert the timestamp. Because the recorded timestamps are already in seconds, specify this argument as 1.
offset — Offset of the simulation start time. Specify the start time as the first timestamp in gpsData.

scale = 1;
offset = startTime;
recordedData =  helperNormTimeInSecs(recordedData,offset,scale);

Extract the GPS data, actor track list, and camera data with updated timestamp values from recordedData.

gpsData = recordedData{1,1}; 
tracklist = recordedData{1,2};
cameraData = recordedData{1,3};

Remove recordedData from the workspace.

clear recordedData

Create Ego Trajectory

Convert geographic GPS coordinates to local east-north-up (ENU) coordinates by using the latlon2local function. Specify an origin by using the latitude and longitude values of the first entry in the GPS data. The transformed coordinates define the trajectory waypoints of the ego vehicle. Units are in meters.

geoReference = [gpsData.latitude(1) gpsData.longitude(1) 0];    
[xEast,yNorth,zUp] = latlon2local(gpsData.latitude,gpsData.longitude,gpsData.altitude,geoReference);
waypoints = [xEast yNorth zUp];

Raw GPS data often contains noise. Smooth the GPS waypoints by using the smoothdata function.

window = round(size(waypoints,1)*0.2);
waypoints = smoothdata(waypoints,"rloess",window);

If GPS data suffers from inaccuracies in position and orientation, then you must improve your ego vehicle localization to generate an accurate ego trajectory. For more information, see the Ego Vehicle Localization Using GPS and IMU Fusion for Scenario Generation example.

Create the ego trajectory using the waypoints and their corresponding time of arrivals by using the waypointTrajectory (Sensor Fusion and Tracking Toolbox) System object™. For this example assume that no tracked vehicle leaves the ground at any of the waypoints, therefore set all the altitude values to 0. You must set the ReferenceFrame property of this System object to "ENU" because Scenario Builder for Automated Driving Toolbox supports only the ENU format for local coordinate data.

waypoints = double([waypoints(:,1:2) zeros(size(zUp))]);
egoTrajectory = waypointTrajectory(waypoints,gpsData.timeStamp,ReferenceFrame="ENU");

Extract Non-Ego Actor Trajectories

Visualize the actor track list and camera images by using the birdsEyePlot and helperPlotActors functions.

% Initialize the figure with bird's eye plot.
currentFigure = figure(Visible="on",Position=[0 0 1400 600]);
hPlot = axes(uipanel(currentFigure,Position=[0 0 0.5 1],Title="Non-Ego Actors"));
bep = birdsEyePlot(XLim=[0 70],YLim=[-35 35],Parent=hPlot);
camPlot = axes(uipanel(currentFigure,Position=[0.5 0 0.5 1],Title="Camera View"));
helperPlotActors(bep,camPlot,tracklist,cameraData)

Extract actor properties such as entry time, exit time, and dimension from the track list data by using the actorprops function. The function uses extracted ego trajectory information to return the non-ego actor properties in the world frame.

nonEgoActorInfo = actorprops(tracklist,egoTrajectory);

Display the first five entries of nonEgoActorInfo.

nonEgoActorInfo(1:5,:)

ans=5×10 table
    Age                   TrackID                    ClassID    EntryTime    ExitTime           Dimension                    Mesh               Waypoints           Speed              Yaw      
    ___    ______________________________________    _______    _________    ________    _______________________    ______________________    ______________    ______________    ______________

     10    "a6ebfc4a-1adb-4920-af12-f7ac56602785"       1           0        0.89983     2.037    5.273    1.825    1×1 extendedObjectMesh    { 10×3 double}    { 10×1 double}    { 10×1 double}
    400    "609aed9d-6681-4e61-83a4-d1e263723107"       1           0           39.9     1.911    4.672    1.527    1×1 extendedObjectMesh    {400×3 double}    {400×1 double}    {400×1 double}
     10    "1823b4a0-d5fa-41c0-9335-98ca77c933b6"       1           0        0.89983     2.043    4.537     1.87    1×1 extendedObjectMesh    { 10×3 double}    { 10×1 double}    { 10×1 double}
    139    "4cd79810-9938-47ba-a3a9-e81221024549"       1           0         13.799     2.199    4.827    1.968    1×1 extendedObjectMesh    {139×3 double}    {139×1 double}    {139×1 double}
    400    "64bf8e46-c5ba-47af-b6fe-40aef7c1aa71"       1           0           39.9     1.981    4.974     1.58    1×1 extendedObjectMesh    {400×3 double}    {400×1 double}    {400×1 double}

Create CSV Trajectories for RoadRunner Scenario

RoadRunner Scenario can import actor trajectories using CSV files. Create a folder in the example working folder, and write CSV files for the ego and non-ego actors to that folder using the helperWriteRoadRunnerCSV function.

csvFolder = "CSVTrajectories";
if ~isfolder(csvFolder) 
    mkdir(csvFolder)
end
helperWriteRoadRunnerCSV(egoTrajectory,nonEgoActorInfo,csvFolder)

Create RoadRunner Scenario

To open RoadRunner using MATLAB®, specify the path to your RoadRunner project. This code shows the path for a sample project folder location in Windows®.

rrProjectPath = "C:\RR\MyProject";

Specify the path to your local RoadRunner installation folder. This code shows the path for the default installation location in Windows.

rrAppPath = "C:\Program Files\RoadRunner R2022b\bin\win64";

Open RoadRunner using the specified path to your project.

rrApp = roadrunner(rrProjectPath,InstallationFolder=rrAppPath);

This example requires a RoadRunner Scene to visualize the vehicle trajectories on road. You can build the scene using HD map and save the RoadRunner scene file. For more information on creating scenes, refer Import Scene Data.

The example provides a RoadRunner scene, which is created using the example Generate Road Scene Using Lanes from Labeled Recorded Data from the PandaSet data set. Copy the RoadRunner scene to the RoadRunner project. The geographic coordinates of scene origin is set to the first latitude and longitude values of the GPS data.

filename = "pandasetScene.rrscene";
copyfile(filename,fullfile(rrProjectPath,"Scenes"));
openScene(rrApp,filename);

Create a new scenario by using the newScenario function of the roadrunner object.

newScenario(rrApp)

Import the trajectories of the ego and non-ego actors from the corresponding CSV files by using the importScenario function.

format = "CSV Trajectory";
csvDir = dir(csvFolder+filesep+"*.csv");
for k=1:numel(csvDir)
    filename = fullfile(pwd,csvFolder,csvDir(k).name);
    importScenario(rrApp,filename,format)
end

Create a scenario simulation object for the current RoadRunner Scenario using the createSimulation function. The simulation object enables you to programatically interact with the RoadRunner scenario simulation using MATLAB.

rrSim = createSimulation(rrApp);

Define the simulation parameters of RoadRunner Scenario. Specify the maximum simulation time as 17 as the RoadRunner Scene is created for a 17 seconds sequence. To plot the simulation results, enable data logging.

endTime = 17;
set(rrSim,MaxSimulationTime=endTime);
set(rrSim,Logging="on");

Run the simulation. Monitor the status of the simulation, and wait for the simulation to complete. Because RoadRunner Scenario cannot remove actors after their exit times, scenario simulation may can fail due to collision. To avoid stopping the scenario on collision, remove fail conditions using these steps:

1. In the Logic editor, click the condition node at the end of the scenario.

2. In the Attributes pane, click Remove Fail Condition.

set(rrSim,SimulationCommand="Start")
 while strcmp(get(rrSim,"SimulationStatus"),"Running")
     simstatus = get(rrSim,'SimulationStatus');
     pause(1)
 end

To view the scenario from the ego vehicle view or chase view, in the Simulation pane, in the Camera, section set Camera View to either Follow or Front. Then, set the Actor attribute to vehicle21, which is the ego vehicle for the scenario in this example.

This figure shows a scenario created in RoadRunner Scenario using RoadRunner scene and actor track list.