Segment Objects Using Segment Anything Model (SAM) in Image Segmenter

Since R2024b

This example shows how to segment objects in an image using the Segment Anything Model (SAM) in the Image Segmenter app. The SAM is an automatic segmentation technique that you can use to instantaneously segment objects from the background, or automatically segment an entire image and interactively create masks for selected objects. In this example, you segment an image using SAM in these ways.

• Segment an object by dragging a region-of-interest (ROI) around an object to segment it instantaneously.

• Segment an object by marking a few points on the object, and, optionally, the background.

• Segment an object by splitting the entire image into regions, and selecting regions to add to the object mask.

Optionally, you can refine a segmentation by applying the SAM to a mask with a single connected region, or refine a mask with multiple disconnected regions using other techniques, such as morphological methods. To learn more about the SAM, see Get Started with Segment Anything Model for Image Segmentation.

This example requires the Image Processing Toolbox™ Model for Segment Anything Model support package. You can install the Image Processing Toolbox Model for Segment Anything Model from the Add-On Explorer. For more information about installing add-ons, see Get and Manage Add-Ons.

For optimal processing speed, use a supported GPU device. Processing image data on a GPU requires a Parallel Computing Toolbox™ license. For more information, see GPU Computing Requirements (Parallel Computing Toolbox).

Load Image into Image Segmenter App

Read an image into the workspace.

I = imread("DogTrio.jpg");

For best SAM performance, use an image with a data range of [0, 255], such as one with a uint8 data type. If your input image has a larger data range, rescale the range of pixel values using the rescale function.

From the MATLAB® Toolstrip, open the Image Segmenter app. On the Apps tab, in the Image Processing and Computer Vision section, click Image Segmenter. On the app toolstrip, click Load Image, and then select Load Image from Workspace. In the Import from Workspace dialog box, select the image you read into the workspace. The app displays the image you selected.

You can also open the app using the imageSegmenter command, specifying the name of the image variable.

imageSegmenter(I)

Select Segment Anything Model

On the Image Segmenter app toolstrip, in the Segmentation Tools section, select Segment Anything Model.

Segment Object Using ROI

The Image Segmenter app opens a new tab for SAM segmentation. Click Draw ROI and draw a rectangular ROI around the object that you want to segment. To draw an ROI, position the cursor over the image and click, then drag until the rectangle covers your desired image region. When you are done drawing the ROI, the Image Segmenter app automatically segments the object inside. The blue shading indicates the segmented area. To adjust the segmented area, you can change the size or position of the ROI by dragging its borders.

When you are satisfied with your mask preview, click Apply. The yellow shading indicates the segmented mask area.

To return to the main Image Segmenter app and create a new mask, click Close SAM.

Segment Object Using Marked Points

You can also instantaneously segment objects by interactively marking points on the image.

To create a new mask, on the Segmentation tab of the Image Segmenter app, in the app toolstrip, select New Segmentation, then New Segmentation. You can also clone the existing mask by selecting New Segmentation, then Clone Segmentation.

Select Mark Foreground, then click the object in the image to create foreground points. When you interactively add points, the Image Segmenter app automatically adds regions to the mask based on your selections and displays a preview of the mask you create shaded in blue. To erase a point you have marked, select Erase and click on the point. To erase all the points you have marked and restart, you can click Clear Marks. In most cases, you can produce a satisfactory mask for the object by marking a few points on the object while using Mark Foreground only. However, you can refine your segmentation mask by specifying parts of the image for SAM to ignore.

To remove regions from your mask, select Mark Background, and click the image to set red points on areas extraneous to the object.

You can mark foreground and background points to segment an image in scenarios such as when an object or region is not localized to one part of the image, or an object has very well-defined boundaries relative to the rest of the image.

When you are satisfied with your mask preview, click Apply to return to the Segment Anything Model menu. The yellow shading indicates the segmented mask area.

To return to the main Image Segmenter app and create a new mask, click Close SAM.

Segment Object Using Automatic Region Boundaries

On the Image Segmenter app toolstrip, in the Segmentation Tools section, select Segment Anything Model. To create a new mask, on the Segmentation tab of the Image Segmenter app, in the app toolstrip, select New Segmentation, then New Segmentation.

On the Image Segmenter app toolstrip, in the Segmentation Tools section, select Segment Anything Model. To automatically split the entire image into regions and select regions to add to a mask, click Show Region Boundaries.

The Image Segmenter app automatically splits the image into regions, whose boundaries appear on the image as thin borders. Select regions to add them to a mask. Before selecting a region, pause on the region to preview it in solid blue. When you click on a region, the app shades it in blue. To remove a region you have added from the mask, select Erase and click on the region.

When you are satisfied with your selection, select Apply to perform segmentation. The yellow shaded region indicates the segmented mask area.

Specify Automatic Segmentation Parameters

For most cases, such as in this example, you can use the default SAM parameters to automatically segment the full image into regions. To change the default parameters of the model, in the Automatic Segmentation section of the Segment Anything Model toolstrip, click Segmentation Settings. The app opens the SAM Automatic Segmentation Parameters dialog box.

You can modify these SAM automatic segmentation settings, based on your application.

• To segment small, densely packed objects relative to the image size, increase the Point Grid Size value.

• To improve the quality of smaller masks, increase the Crop Level value. To improve processing speed when the Crop Level value is greater than 1, you can specify a higher Point Grid Downscale Factor value.

• To reduce the number of false positives, at the possible expense of missing some true positives in the segmentation, increase the Score Threshold value.

• To reduce the number of overlapping segmentations, at the possible expense of eliminating segmentations with only minor overlap, decrease the Select Strongest value.

You can modify these full image segmentation settings, based on your available computing resources.

• To improve processing speed when you are using a CPU, or have limited computing resources, decrease the Point Grid Size value.

• To improve processing speed when you are using a GPU, at the expense of higher memory usage, increase the Point Batch Size value.

• To improve processing speed, set the Max Object Area and Min Object Area value to the largest and smallest known area, respectively, for the objects being detected in the image.

To apply new segmentation parameters, click Update Segmentation. Otherwise, to restore the default parameters, click Restore Defaults.

Refine Mask

To refine a mask, such as the mask produced from an automatically segmented image in the Segment Object Using Region Boundaries section, on the Image Segmenter app toolstrip, expand the Segmentation Tools gallery and, from the Refine Mask section, select Segment Anything Model. You can refine a mask using SAM only when it consists of a single connected region of pixels. If your mask contains multiple disconnected regions of pixels, use another refinement tool, such as Morphology, Active Contours, or Fill Holes, to refine your mask.

When you select Segment Anything Model, the Image Segmenter app automatically creates an ROI around the mask image and refines the segmentation. You can adjust the ROI size or position by dragging its borders. To further refine the mask, select Mark Foreground and click the image to mark points on the object which have been missed in automatic segmentation. Click Mark Background to mark points on regions you want to remove from the mask.

When you are satisfied with your refined mask selection, click Create Mask. To view the binary mask, click Show Binary.

When you are done segmenting the image, to save final binary mask image and the segmented version of the original image, click Export and select Export Images. You can also generate the code used to perform the segmentation (this requires Statistics and Machine Learning Toolbox™). Use the code to apply the same segmentation algorithm to similar images. To get the code, click Export and select Generate Function.