groundTruthMultiSensor

R2026b

Ground truth label data for multiple sensors

Since R2026b

Description

The groundTruthMultiSensor object contains information about the ground truth data source, label definitions, and marked label annotations for multiple sensors. The source of the signals can be a video, image sequence, lidar point cloud, or any other custom format containing multiple signals. You can export or import a groundTruthMultiSensor object from the Multi-Sensor Labeler app.

You can use groundTruthMultiSensor objects to generate training data for deep learning applications. The approach depends on the signal type and task.

Task	Signal Type	Approach
3-D Semantic Segmentation	Point Cloud	Use semanticPointLabelTrainingData directly with the groundTruthMultiSensor object.
3-D Object Detection	Point Cloud	Use pointCloudObjectDetectorTrainingData directly with the groundTruthMultiSensor object.
2-D Object Detection, Semantic Segmentation, Instance Segmentation	Image	Use gatherLabelData and writeFrames to extract label data and image frames, then manually create datastores for training.

For details on generating training data, see Export Multi-Sensor Ground Truth and Create Training Data.

Creation

To export a groundTruthMultiSensor object from the Multi-Sensor Labeler app, on the app toolstrip, select Export Labels > To Workspace. The app exports the object to the MATLAB^® workspace. To create a groundTruthMultiSensor object programmatically, use the groundTruthMultiSensor function (described here).

Syntax

gTruth = groundTruthMultiSensor(dataSources,labelDefs,roiData,sceneData)

Description

gTruth = groundTruthMultiSensor(dataSources,labelDefs,roiData,sceneData) returns an object containing ground truth labels that can be imported into the Multi-Sensor Labeler app.

• dataSources specifies the sources of the ground truth data and sets the DataSource property.

• labelDefs specifies the label, sublabel, and attribute definitions of the ground truth data and sets the LabelDefinitions property.

• roiData specifies the identifying information, position, and timestamps for the marked region of interest (ROI) labels and sets the ROILabelData property.

• sceneData specifies the identifying information and timestamps for marked scene labels and sets the SceneLabelData property.

Properties

DataSource — Sources of ground truth data

vector of MultiSignalSource objects

Sources of ground truth data, specified as a vector of MultiSignalSource objects. These objects contain information that describe the data sources from which ground truth data was labeled. This table describes the type of MultiSignalSource objects that you can specify in this vector.

Class	Data Source
vision.labeler.loading.VideoSource	Video file
vision.labeler.loading.ImageSequenceSource	Image sequence folder
vision.labeler.loading.CustomImageSource	Custom image format
vision.labeler.loading.PointCloudSequenceSource	Point cloud sequence folder
vision.labeler.loading.VelodyneLidarSource	Velodyne packet capture (PCAP) file
vision.labeler.loading.HesaiLidarSource	Hesai packet capture (PCAP) file
vision.labeler.loading.OusterLidarSource	Ouster packet capture (PCAP) file
vision.labeler.loading.E57Source	E57 file
vision.labeler.loading.RosbagSource	ROS or ROS 2 bag file (.bag, .db3, .mcap)
vision.labeler.loading.LasFileSequenceSource	LAS or LAZ file sequence
vision.labeler.loading.CustomPointCloudSource	Custom point cloud format

To specify additional data sources, create a new type of MultiSignalSource object by using the vision.labeler.loading.MultiSignalSource class.

LabelDefinitions — Label definitions

table

Label definitions, specified as a table. To create this table, use one of these options.

• In the Multi-Sensor Labeler app, create label definitions, and then export them as part of a groundTruthMultiSensor object.

• Use a labelDefinitionCreatorMultiSensor object to generate a label definitions table. If you save this table to a MAT-file, you can then load the label definitions into a Multi-Sensor Labeler app session by selecting Open > Label Definitions from the app toolstrip.

• Create the label definitions table at the MATLAB command line.

This table describes the required and optional columns of the table specified in the LabelDefinitions property.

Column	Description	Required or Optional
Name	Strings or character vectors specifying the name of each label definition.	Required
SignalType	SignalType enumerations that specify the signal type supported for each label definition. Valid values are Image for image signals such as videos or image sequences, PointCloud for lidar signals, or Time for scene label definitions. If a label definition supports multiple signal types, then the label definition has a separate row for each signal type. For example, consider a label definition named car. In the Multi-Sensor Labeler app, you draw this label as a rectangle in image signals and a cuboid in lidar point cloud signals. In the LabelDefinitions table, car appears twice and has these Name, SignalType, and LabelType values. Name SignalType LabelType "car" Image Rectangle "car" PointCloud Cuboid	Required
LabelType	labelType (Computer Vision Toolbox) enumerations that specify the type of each label definition. For ROI label definitions with a SignalType of Image, valid LabelType enumerations are: labelType.Rectangle labelType.RotatedRectangle labelType.Line labelType.PixelLabel labelType.Polygon labelType.ProjectedCuboid labelType.Point For ROI label definitions with a SignalType of PointCloud, valid LabelType enumerations are: labelType.Cuboid labelType.SemanticPointLabel and . For scene label definitions, the valid LabelType enumerations are: labelType.SceneLogical labelType.SceneNumeric labelType.SceneString You can also add ground truth data that is not an ROI or scene label to a groundTruthMultiSensor object. In the label definitions table, specify a label definition whose labelType is Custom.	Required
Group	Strings or character vectors specifying the group to which each label definition belongs.	Optional If you create label definitions at the MATLAB command line, you do not need to include a Group column. If you export label definitions from the Multi-Sensor Labeler app or create them using a labelDefinitionCreatorMultiSensor object, the label definitions table includes this column, even if you did not specify groups. The app assigns each label definition a Group value of "None".
Description	Strings or character vectors that describe each label definition.	Optional If you create label definitions at the MATLAB command line, you do not need to include a Description column. If you export label definitions from the Multi-Sensor Labeler app or create them using a labelDefinitionCreatorMultiSensor object, the label definitions table includes this column, even if you did not specify descriptions. The Description for these label definitions is an empty character vector.
LabelColor	1-by-3 row vectors of RGB triplets that specify the colors of the label definitions. Values are in the range [0, 1]. The color yellow (RGB triplet [1 1 0]) is reserved for the color of selected labels in the Multi-Sensor Labeler app.	Optional When you define labels in the Multi-Sensor Labeler app, you must specify a color. Therefore, an exported label definitions table always includes this column. When you create label definitions using the labelDefinitionCreatorMultiSensor object without specifying colors, the returned label definition table includes this column, but all column values are empty.
PixelLabelID	Scalars, column vectors, M-by-3 matrices of integer-valued label IDs. PixelLabelID specifies the pixel label values used to represent a label definition. Pixel label ID values must be between 0 and 255.	Optional When you define pixel labels in the Multi-Sensor Labeler app or the labelDefinitionCreatorMultiSensor object, the generated label definitions table includes this column. When creating a label definitions table at the MATLAB command line, if you set LabelType to labelType.PixelLabel for any label, then this column is required.
SemanticPointLabelId	Scalars of integer-valued label IDs. SemanticPointLabelId specifies the semantic point label values used to represent a label definition for point cloud semantic segmentation. Semantic point label ID values must be between 0 and 255.	Optional When you define semantic point labels in the Multi-Sensor Labeler app or the labelDefinitionCreatorMultiSensor object, the generated label definitions table includes this column. When creating a label definitions table at the MATLAB command line, if you set LabelType to labelType.SemanticPointLabel for any label, then this column is required.
Hierarchy	Structures containing sublabel and attribute data for each label definition.	Optional When you define sublabels or attributes in the Multi-Sensor Labeler app or the labelDefinitionCreatorMultiSensor object, the generated label definitions table includes this column.

ROILabelData — ROI label data

ROILabelData object

ROI label data across all signals, specified as a ROILabelData object. The ROILabelData object is from the vision.labeler.labeldata namespace.

For Rectangle, RotatedRectangle, Cuboid, ProjectedCuboid, Polygon, and Line label types, ground truth data that is not a floating-point array has a data type of single.

For SemanticPointLabel label types, the label data is stored as indices that map each point in the point cloud to a semantic class defined in the SemanticPointLabelId column of the LabelDefinitions table. When you gather label data using the gatherLabelData function, point cloud signals that contain semantic point labels include a SemanticPointLabelData column in the output table. This column contains file paths to MAT files that store the per-point label arrays, similar to how the PixelLabelData column stores pixel-level label data for image signals.

SceneLabelData — Scene label data

SceneLabelData object

Scene label data across all signals, specified as a SceneLabelData object. The SceneLabelData object is from the vision.labeler.labeldata namespace.

Object Functions

selectLabelsByLabelName	Select multi-sensor ground truth by label name
selectLabelsByLabelType	Select multi-sensor ground truth by label type
selectLabelsByGroupName	Select multi-sensor ground truth by label group name
selectLabelsBySignalName	Select multi-sensor ground truth by signal name
selectLabelsBySignalType	Select multi-sensor ground truth labels by signal type
gatherLabelData	Gather time-synchronized label data from ground truth
writeFrames	Write signal frames for multi-sensor ground truth data to disk
changeFilePaths	Change file paths in multi-sensor ground truth data

Examples

Create Ground Truth from Multiple Signals

Create ground truth data for a video signal and a lidar point cloud sequence signal that captures the same driving scene. Specify the signal sources, label definitions, and ROI and scene label data.

Create the video data source from an MP4 file.

sourceName = fullfile(toolboxdir("pointcloud"),"pcdata","01_city_c2s_fcw_10s.mp4");
sourceParams = [];
vidSource = vision.labeler.loading.VideoSource;
vidSource.loadSource(sourceName,sourceParams);

Create the point cloud sequence source from a folder of point cloud data (PCD) files.

pcSeqFolder = fullfile(toolboxdir("pointcloud"),"pcdata","lidarSequence");
lidarSourceData = load(fullfile(pcSeqFolder,"timestamps.mat"));

sourceName = pcSeqFolder;
sourceParams = struct;
sourceParams.Timestamps = lidarSourceData.timestamps;

pcseqSource = vision.labeler.loading.PointCloudSequenceSource;
pcseqSource.loadSource(sourceName,sourceParams);

Combine the signal sources into an array.

dataSource = [vidSource pcseqSource]

dataSource =

  1×2 heterogeneous MultiSignalSource (VideoSource, PointCloudSequenceSource) array with properties:

    SourceName
    SourceParams
    SignalName
    SignalType
    Timestamp
    NumSignals

Create a table of label definitions for the ground truth data by using a labelDefinitionCreatorMultiSensor object.

• The Car label definition appears twice. Even though Car is defined as a rectangle, you can draw rectangles only for image signals, such as videos. The labelDefinitionCreatorMultiSensor object creates an additional row for lidar point cloud signals. In these signal types, you can draw Car labels as cuboids only.

• The label definitions have no descriptions and no assigned colors, so the Description and LabelColor columns are empty.

• The label definitions have no assigned groups, so for all label definitions, the corresponding cell in the Group column is set to "None".

• Road is a pixel label definition, so the table includes a PixelLabelID column.

• No label definitions have sublabels or attributes, so the table does not include a Hierarchy column for storing such information.

ldc = labelDefinitionCreatorMultiSensor;
addLabel(ldc,"Car","Rectangle");
addLabel(ldc,"Truck","ProjectedCuboid");
addLabel(ldc,"Lane","Line");
addLabel(ldc,"Road","PixelLabel");
addLabel(ldc,"Sunny","Scene");
labelDefs = create(ldc)

labelDefs =

  7×7 table

      Name       SignalType       LabelType        Group      Description    LabelColor    PixelLabelID
    _________    __________    _______________    ________    ___________    __________    ____________

    {'Car'  }    Image         Rectangle          {'None'}       {' '}       {0×0 char}    {0×0 double}
    {'Car'  }    PointCloud    Cuboid             {'None'}       {' '}       {0×0 char}    {0×0 double}
    {'Truck'}    Image         ProjectedCuboid    {'None'}       {' '}       {0×0 char}    {0×0 double}
    {'Lane' }    Image         Line               {'None'}       {' '}       {0×0 char}    {0×0 double}
    {'Lane' }    PointCloud    Line               {'None'}       {' '}       {0×0 char}    {0×0 double}
    {'Road' }    Image         PixelLabel         {'None'}       {' '}       {0×0 char}    {[       1]}
    {'Sunny'}    Time          Scene              {'None'}       {' '}       {0×0 char}    {0×0 double}

Create ROI label data for the first frame of the video.

numVideoFrames = numel(vidSource.Timestamp{1});
carData = cell(numVideoFrames,1);
laneData = cell(numVideoFrames,1);
truckData = cell(numVideoFrames,1);
carData{1} = [304 212 37 33];
laneData{1} = [70 458; 311 261];
truckData{1} = [309,215,33,24,330,211,33,24];
videoData = timetable(vidSource.Timestamp{1},carData,laneData, ...
                      VariableNames={"Car","Lane"});

Create ROI label data for the first point cloud in the sequence.

numPCFrames = numel(pcseqSource.Timestamp{1});
carData = cell(numPCFrames, 1);
carData{1} = [27.35 18.32 -0.11 4.25 4.75 3.45 0 0 0];
lidarData = timetable(pcseqSource.Timestamp{1},carData,VariableNames={"Car"});

Combine the ROI label data for both sources.

signalNames = [dataSource.SignalName];
roiData = vision.labeler.labeldata.ROILabelData(signalNames,{videoData,lidarData})

roiData =

  ROILabelData with properties:

                lidarSequence: [34×1 timetable]
    video_01_city_c2s_fcw_10s: [204×2 timetable]

Create scene label data for the first 10 seconds of the driving scene.

sunnyData = seconds([0 10]);
labelNames = ["Sunny"];
sceneData = vision.labeler.labeldata.SceneLabelData(labelNames,{sunnyData})

sceneData =

  SceneLabelData with properties:

    Sunny: [0 sec    10 sec]

Create a ground truth object from the signal sources, label definitions, and ROI and scene label data. You can import this object into the Multi-Sensor Labeler app for manual labeling or to run a labeling automation algorithm on it. You can also extract training data from this object for deep learning models by using the gatherLabelData function.

gTruth = groundTruthMultiSensor(dataSource,labelDefs,roiData,sceneData)

gTruth =

  groundTruthMultiSensor with properties:

          DataSource: [1×2 vision.labeler.loading.MultiSignalSource]
    LabelDefinitions: [7×7 table]
        ROILabelData: [1×1 vision.labeler.labeldata.ROILabelData]
      SceneLabelData: [1×1 vision.labeler.labeldata.SceneLabelData]

Tips

• groundTruthMultiSensor objects with video-based data sources rely on the video reading capabilities of your operating system. A groundTruthMultiSensor object created using video data sources remains consistent only for the same platform that was used to create it. To create a platform-independent groundTruthMultiSensor object, convert the videos into sequences of images and include the associated timestamps with the image sequences.

• To create a groundTruthMultiSensor object containing ROI label data but no scene label data, specify the SceneLabelData property as an empty array. To create this array, at the MATLAB command prompt, enter this code.

  sceneData = vision.labeler.labeldata.SceneLabelData.empty

Version History

Introduced in R2026b