Combine Reviewed Images and Export

Step 5 of 5 in Get Started with Team-Based Labeling

The Image Labeler app enables the project owner of a distributed labeling team to perform these tasks.

Create a labeling project.
Distribute images into label tasks and assign them to label task owners.
Distribute labeled images into review tasks and assign them to review task owners.
Combine and export the labeled and reviewed images.

For more information about how to set up a team labeling project as a project owner, see Create Project for a Labeling Team and Publish Tasks. To label images as a label task owner and share labeled images for review, see Label Images and Send for Review. To create a review task as a project owner, see Create Review Tasks for Labeled Images. For more information about how to review labeled images and share feedback as a review task owner, see Review Labeled Images and Send Feedback.

This example shows the steps the project owner must follow after the review task owner shares feedback. To export the ground truth labels, the project owner must follow these steps.

Open an existing labeling project.
Check the status of the labeled and reviewed images.
Export the combined labels as a groundTruth object.

Open Existing Labeling Project

Open the Image Labeler app using one of these options:

MATLAB^® Toolstrip: On the Apps tab, under Image Processing and Computer Vision, click the app icon.
In the MATLAB command prompt, enter
```
imageLabeler
```

To open an existing team project, such as the one you created in the Create Project for a Labeling Team and Publish Tasks step, follow these steps.

In the Image Labeler app interface, select Open Project.
In the file explorer, navigate to the folder that contains your previously created project. Select the .prj file, and click Open.

Once you select and open the team labeling project, the app enables you to open the project in Project Owner or Task Owner mode. To access project created by you as a project owner, select Project Owner.

Image Labeling and Review Status

The label task owner completes their label task and shares the labels for review. The review task owner then reviews these image labels and shares feedback with the project owner and the label task owner. The project owner then combines the reviewed image ground truth labels and exports them as a groundTruth object.

Task Status Summary After Review Feedback

The Tasks pane shows the details and progress of the assigned labeling and review tasks. Previously in this example, the project owner created and published three label tasks, Task 1, Task 2, and Task 3, and three review tasks, Task 4, Task 5, and Task 6. Out of 150 images in the project, each label task was assigned 50 images. The label task owners labeled and shared 144 labeled images for review and the project owner then created three review tasks, assigning 50 labeled images each to Task 4 and Task 5, and 44 labeled images to Task 6. The review task owners reviewed 136 of the labeled images and shared their feedback with the project owner.

Task pane showing published label tasks

Status of Reviewed Images

As a project owner, you can see the status of the labeled or reviewed images in the Image Browser by setting the Image Filters option.

To see the status of all the images in the project, set Image Filters to All images in current project.
To see the status of images in a particular review task, select a review task in the Tasks pane. For this example, select Task 5. Then, in the Image Browser, set Image Filters to Images in selected task. The Image Browser now displays only the image thumbnails for images assigned to Task 5.

The thumbnails of reviewed images in the Image Browser have status badges in their corners that indicate their current review status, as shown in this table.

Image Thumbnail with Status Badge	Image Status	Description
	`Image locked by a review task owner`	Image is assigned to a review task, and the review is in progress.
	`Image ready for export`	Image labels have been reviewed, accepted, and are ready for export.
	`Image locked by a label task owner`	Image requires labeling by the label task owner. Images requiring labeling comes from these sources: Label tasks created by the project owner Images with labels rejected by the review task owner and sent for relabeling

Combine and Export Image Labels

You can export the accepted labels to a MAT file or to a variable in the MATLAB workspace. In both cases, Image Labeler exports the accepted labels as a groundTruth object. You can combine and export labels from:

All images in the project
Accepted images that are ready for export
Only a task

Task panel showing published label tasks

You can export image labels as one of these options:

MATLAB^® workspace variable
MAT file, which you can share directly with individuals on your team or place in a shared network location

Export Labels from All Images

Gather the most recent ground truth data of each image is all contributors and export it. The ground truth data can include:

Labeled images that have been reviewed and accepted
Labeled images that have been reviewed, rejected, and require correction
Labeled images that have been not been reviewed
Labeled images with review in progress
Unlabeled images
Partially labeled images with labeling in progress

To combine the labels from all of these images and export them to a workspace variable, on the app toolstrip, select Export > All Images > To Workspace.

Export Labels from Accepted Images

Accepted images include only those images with reviewed and accepted labels. To export labels from only accepted images in the project to a workspace variable, on the app toolstrip, select Export > Accepted Images > To Workspace. The ground truth data of images with labels that are not accepted, will be empty.

Export Labels from Task

You can also export images from a specific labeling or review task. To export all the labels from review Task 5 to the MATLAB workspace, in the Tasks pane, select Task 5. On the app toolstrip, select Export > Selected Task > To Workspace. The ground truth data of images that do not belong to the selected task is empty.

Export Label Definitions

To export the label definitions as a MAT file, from the app toolstrip, select Export > Label Definitions. You can then import these label definitions into another app session by selecting Import > Label Definitions.

Ground Truth Object

The Image Labeler app exports the labels to a groundTruth object. The ground truth object contains information about the:

Data source or data sources
Label definitions
Drawn ground truth labels

You can share this object with:

Algorithm developers, who can use it to train algorithms, such as an object detection or semantic segmentation network. For more details, see Training Data for Object Detection and Semantic Segmentation.
Validation engineers, who can use it to validate algorithms.

Display the properties of the exported groundTruth object. The information in your exported object might differ from the information shown here.

gTruth

gTruth = 

  groundTruth with properties:

          DataSource: [1×1 groundTruthDataSource]
    LabelDefinitions: [3×5 table]
           LabelData: [150×3 table]

Data Source

The DataSource property of gTruth is a groundTruthDataSource object that contains the paths to the images. Display the properties of this object.

gTruth.DataSource

ans = 

groundTruthDataSource for an image collection with properties

                      Source: {
                              ' ...\matlab\toolbox\vision\visiondata\NewTsukuba\image0001.jpg';
                              ' ...\matlab\toolbox\vision\visiondata\NewTsukuba\image0002.jpg';
                              ' ...\matlab\toolbox\vision\visiondata\NewTsukuba\image0003.jpg'
                               ... and 147 more
                              }

Label Definitions

The LabelDefinitions of gTruth is a table that contains information about the label definitions. To save the label definitions in their own MAT file, from the app toolstrip, select Save > Label Definitions. You can then import these label definitions into another app session by selecting Import > Label Definitions.

Display the label definitions table. Each row contains information about an ROI label definition or a scene label definition.

gTruth.LabelDefinitions

ans =

  3×5 table

         Name            Type              LabelColor            Group      Description
    _______________    _________    ________________________    ________    ___________

    {'Lamp'       }    Rectangle    {[0.5862 0.8276 0.3103]}    {'None'}    {0×0 char} 
    {'VideoCamera'}    Rectangle    {[     0.5172 0.5172 1]}    {'None'}    {0×0 char} 
    {'BustPresent'}    Scene        {[     0 0.7241 0.6552]}    {'None'}    {0×0 char}

Label Data

The LabelData of gTruth is a table that contains the details of the marked labels. Display the label data table.

Each row in the table corresponds to an image in the data set, and the columns specify details for each ROI or scene label. In this example, each element of the first column is a cell array with the coordinates of a rectangular Lamp ROI label. Each element of the second column is a cell array with the coordinates of a rectangular VideoCamera ROI label. Each element of the third column is a logical scalar that indicates the presence or absence of the Bust object in the scene.

gTruth.LabelData

ans =

  150×3 table

           Lamp               VideoCamera        BustPresent
    ___________________    __________________    ___________

    {[312 145 157 180]}    {[ 101 76 184 98]}       true    
    {[315 148 151 174]}    {[ 99 70 188 110]}       true    
    {[322 141 149 179]}    {[116 60 181 112]}       true    
    {[327 140 152 170]}    {[118 59 189 103]}       false   
    {[333 128 161 183]}    {[120 51 190 106]}       true    

             :                     :                  :     

    {0×0 double       }    {0×0 double      }       false   
    {0×0 double       }    {0×0 double      }       false   
    {0×0 double       }    {0×0 double      }       false   
    {0×0 double       }    {0×0 double      }       false   
    {[ 383 59 146 148]}    {0×0 double      }       true