Enhanced Color Replacement with Customized Code

Introduction

A color detection algorithm identifies pixels in an image that match a specified color or color range. The detected pixels in the original image are replaced by the pixels from a different image. This process is known as color replacement or Chroma Key.

This model internally uses the same sub-system as the Color Detection and Color Replacement examples.

In this example, you will choose the reference color for detection from selected position on your iOS device screen. You will configure the model and develop this user interface by updating code to get the touched location and corresponding pixel info from a video frame.

This example shows:

Prerequisites

Required Hardware

Task 1 - Build model and import generated project

In this task, you will build a pre-configured model. You will import generated project in Xcode for further development.

1. Open the Color Replacement with Touch model.

2. Load the template image into the matrix RGB_background by executing the following command in the MATLAB command window:

RGB_background = imread('image_640_480.jpg');

3. In the model, double click on the FromApp block. Note the field Method name is pickColorFromDeviceScreen. In the next tasks you will implement this method.

4. Connect your configured Apple iOS device to your host computer. If you have not previously configured your device, run ioshwsetup from the MATLAB command prompt.

5. In your Simulink model, click Simulation > Model Configuration Parameters to open Configuration Parameters dialog.

6. Select the Hardware Implementation pane and set the Hardware board to Apple iOS Device.

7. In your Simulink model, click the Deploy to Hardware button on the toolbar.

8. Locate the generated project folder ioscolorreplacementcustom_ert_rtw in your current working directory.

9. Open the Xcode project for the application under ioscolorreplacementcustom_ert_rtw/ioscolorreplacementcustom

Task 2 - Run a base application

In this task you will modify the generated app code to run a simple case of color replacement detecting a fixed color.

1. In the Xcode IDE project window, open ioscolorreplacementcustom_ert_rtw > ioscolorreplacementcustom > InfoViewController.mm.

2. In the InfoViewController class, add the following custom implementation of pickColorFromDeviceScreen method. This method returns RGB components of fixed color as an output of FromApp block in the Simulink model. Note that a method declaration can be added anywhere inside the class as long as it does not overlap with other declarations.

- (void) pickColorFromDeviceScreen:(Byte *)rgb_send
{
/* Hardcoded values for sending Red color as the reference */
rgb_send[0] = 211;
rgb_send[1] = 27;
rgb_send[2] = 10;
}

3. In Xcode IDE, select Xcode > Product > Run to run the app on your device.

4. The color replacement camera app appears, detecting red colors in the images of scene.

Task 3 - Update code to use touched position

In this task, you will update code in the application to get screen touch location and choose the reference color at that location . You will add extra code to calculate the pixel location on the video frame and get color components for detection. You will run the updated application to see result using touch interface.

1. In the ViewController class in the ViewController.mm, add the following touchesBegan method. Note this callback has code to read the touch position and find the RGB components of corresponding pixel in the input video frame.

Byte rgbaValue[4];
- (void)touchesBegan:(NSSet *)
touches withEvent:(UIEvent *)event
{
UITouch *touch = [touches anyObject];
/*  Get the specific point that was touched */
CGPoint touchPosition = [touch locationInView:self.view];

int new_X =  (touchPosition.x * Input.cols
             /imageView.bounds.size.width);
int new_Y =  (touchPosition.y * Input.rows
             /imageView.bounds.size.height);

/* The pixel value read from is in BGRA format */
rgbaValue[0] = Input.at<cv::Vec4b> (new_Y,new_X)[2];
rgbaValue[1] = Input.at<cv::Vec4b> (new_Y,new_X)[1];
rgbaValue[2] = Input.at<cv::Vec4b> (new_Y,new_X)[0];
rgbaValue[3] = Input.at<cv::Vec4b> (new_Y,new_X)[3];

/*NSLog(@"Input(%d %d): %d %d %d %d",new_X, new_Y,
rgbaValue[0],rgbaValue[1],rgbaValue[2],rgbaValue[3]);*/

}

2. In the InfoViewController class in the InfoViewController.mm, replace the pickColorFromDeviceScreen method with the following implementation to return RGB components of selected reference color as an output of FromApp block in the Simulink model.

- (void) pickColorFromDeviceScreen:(Byte *)rgb_send
{
/* copy the RGB color values to return variable for sending */
rgb_send[0] = rgbaValue[0];
rgb_send[1] = rgbaValue[1];
rgb_send[2] = rgbaValue[2];
}

3. In the ViewController.h, add the following declaration.

extern Byte rgbaValue[4];

4. In Xcode IDE, select Xcode > Product > Run to run the app on your device.

5. Once the updated application appears on device, touch at different regions on the screen image to observe the color replacement.

Other Things to Try

Summary

This example introduced a workflow to develop a custom feature in iOS application using Simulink Support Package for Apple iOS Devices. In this example you have learned: