Main Content

Read Video Files on NVIDIA Hardware

With MATLAB® Coder™ Support Package for NVIDIA® Jetson™ and NVIDIA DRIVE® Platforms, you can generate CUDA® code for the MATLAB VideoReader object to read files containing video data on the NVIDIA target hardware. The generated code uses the GStreamer library API to read the video files.

Sobel Edge Detection on Video File

In this example, you use GPU Coder™ and the MATLAB Coder Support Package for NVIDIA Jetson and NVIDIA DRIVE Platforms to generate and deploy a CUDA executable for a Sobel edge detection application on the Jetson TX2 board. This CUDA application reads the contents of a video file, performs the edge detection operation, and displays the output video on the NVIDIA hardware.

Requirements

  1. GPU Coder.

  2. MATLAB Coder Support Package for NVIDIA Jetson and NVIDIA DRIVE Platforms.

  3. Image Processing Toolbox™ toolbox for the rhinos.avi sample video file used in this example.

  4. NVIDIA CUDA toolkit.

  5. GStreamer and SDL libraries on the target.

  6. Environment variables for the compilers and libraries on the host and the target. For more information, see Third-Party Hardware (GPU Coder), Environment Variables (GPU Coder), and Install and Setup Prerequisites for NVIDIA Boards.

  7. NVIDIA Jetson TX2 embedded platform.

Create a Live Hardware Connection Object

The support package software uses an SSH connection over TCP/IP to execute commands while building and running the generated CUDA code on the Jetson platforms. Connect the target platform to the same network as the host computer. Alternatively, use an Ethernet crossover cable to connect the board directly to the host computer. Refer to the NVIDIA documentation on how to set up and configure your board.

To communicate with the NVIDIA hardware, you must create a live hardware connection object by using the jetson function. To create a live hardware connection object, provide the host name or IP address, user name, and password of the target board. For example to create live object for Jetson hardware:

hwobj = jetson('jetson-tx2-name','ubuntu','ubuntu');

The software performs a check of the hardware, compiler tools and libraries, IO server installation, and gathers information on the peripherals connected to the target. This information is displayed in the command window as shown.

Checking for CUDA availability on the Target...
Checking for 'nvcc' in the target system path...
Checking for cuDNN library availability on the Target...
Checking for TensorRT library availability on the Target...
Checking for prerequisite libraries is complete.
Gathering hardware details...
Checking for third-party library availability on the Target...
Gathering hardware details is complete.
 Board name         : NVIDIA Jetson TX2
 CUDA Version       : 10.0
 cuDNN Version      : 7.6
 TensorRT Version   : 6.0
 GStreamer Version  : 1.14.5
 V4L2 Version       : 1.14.2-1
 SDL Version        : 1.2
 OpenCV Version     : 4.1.1
 Available Webcams  : Microsoft® LifeCam Cinema(TM)
 Available GPUs     : NVIDIA Tegra X2

Alternatively, to create live object for DRIVE hardware:

hwobj = drive('drive-px2-name','ubuntu','ubuntu');

Note

When there is a connection failure, a diagnostics error message is reported on the MATLAB command line. The most likely cause of a connection failure is incorrect IP address or host name of the target.

The videoReaderDeploy Entry-Point Function

Create a MATLAB file videoReaderDeploy.m that acts as the entry-point function for code generation. The videoReaderDeploy.m function creates a VideoReader object called vObj to read the rhinos.avi video file located on the target hardware. The function then uses the hasFrame and readFrame methods of the VideoReader object to determine and read valid video frames from the input file. The function performs Sobel edge detection through a 2-dimensional spatial gradient operation and displays the edge detected image on the target hardware. The function finds the horizontal gradient(h) and vertical gradient (v) of the input image with respective Sobel kernels.

function videoReaderDeploy()

% Create Jetson hardware object
hwobj = jetson();
vidName = '/home/ubuntu/Videos/rhinos.avi';

% Create video reader object
vObj = VideoReader(hwobj,vidName,'Width',320,'Height',240);

% Create display object on the target
dispObj = hwobj.imageDisplay;

% Grab frame from the video pipeline
while vObj.hasFrame
    img = vObj.readFrame();
    
    % Sobel edge detection
    kernel = [1 2 1;0 0 0;-1 -2 -1];
    h = conv2(img(:,:,2),kernel,'same');
    v = conv2(img(:,:,2),kernel','same');
    e = sqrt(h.*h + v.*v);
    edgeImg = uint8((e > 100) * 240);
    
    % Display edge detected image
    image(dispObj,edgeImg);
end
end

For code generation, the VideoReader function requires the full path to the video file on the target hardware. The MATLAB Coder Support Package for NVIDIA Jetson and NVIDIA DRIVE Platforms uses the GStreamer library API to read the video files on the target platform. The software supports file (container) formats and codecs that are compatible with GStreamer. For more information, see https://gstreamer.freedesktop.org/documentation/plugin-development/advanced/media-types.html?gi-language=c. For other code generation limitations of the VideoReader function, see Limitations.

Generate CUDA Executable Using GPU Coder

Create a GPU code configuration object for generating an executable. Use the coder.hardware function to create a configuration object for the Jetson platform and assign it to the Hardware property of the code configuration object cfg. Use the BuildDir property to specify the folder for performing remote build process on the target. If the specified build folder does not exist on the target, then the software creates a folder with the given name. If no value is assigned to cfg.Hardware.BuildDir, the remote build process happens in the last specified build folder. When there is no stored build folder value, the build process takes place in the home folder. Set the GenerateExampleMain property to generate an example CUDA C++ main function and compile it. This example does not require modifications to the generated main files. Use the putFile method of the Jetson object to move the input video file to the target platform.

cfg = coder.gpuConfig('exe');
cfg.Hardware = coder.hardware('NVIDIA Jetson');
cfg.Hardware.BuildDir = '~/remoteBuildDir';
cfg.GenerateExampleMain = 'GenerateCodeAndCompile';
hwobj.putFile('rhinos', hwobj.workspaceDir);

To generate CUDA code, use the codegen command and pass the GPU code configuration object along with the videoReaderDeploy entry-point function. After the code generation takes place on the host, the generated files are copied over and built on the target.

codegen('-config ',cfg,'videoReaderDeploy','-report');

Run the Executable

To run the executable on the target hardware, use the runApplication method of the hardware object. In the MATLAB Command Window, enter:

pid = runApplication(hwobj,'videoReaderDeploy');

A window opens on the target hardware display showing the edge detected output of the input video.

Specifying the Video File at Runtime

Instead of specifying the video file during code generation time, you can modify the entry-point function and code configuration object to accept a variable file name when running the executable. 

function videoReaderDeploy(vfilename)

% Create Jetson hardware object
hwobj = jetson();

% Create video reader object
vObj = VideoReader(hwobj,vfilename,'Width',640,'Height',480);

% Create display object on the target
dispObj = hwobj.imageDisplay;

% Grab frame from the video pipeline
while vObj.hasFrame
    img = vObj.readFrame();
    
    % Sobel edge detection
    kernel = [1 2 1;0 0 0;-1 -2 -1];
    h = conv2(img(:,:,2),kernel,'same');
    v = conv2(img(:,:,2),kernel','same');
    e = sqrt(h.*h + v.*v);
    edgeImg = uint8((e > 100) * 240);
    
    % Display edge detected image
    image(dispObj,edgeImg);
end
end

Create a custom main file to handle the variable file name input when running the executable. A snippet of the code is shown.

static void main_videoReaderDeploy(const char* const vfilename)
{
 videoReaderDeploy(vfilename);
}

//
// Arguments     : int32_T argc
//                 const char * const argv[]
// Return Type   : int32_T
//
int32_T main(int32_T, const char * const argv[])
{
	//Initialize the application
	videoReaderDeploy_initialize();
	
	//Invoke entry-point function
	main_videoReaderDeploy(argv[1]);
	
	//Terminate the application
	videoReaderDeploy_terminate();
	return 0;
}

Modify the code configuration object to include this custom main file.

cfg = coder.gpuConfig('exe');
cfg.Hardware = coder.hardware('NVIDIA Jetson');
cfg.Hardware.BuildDir = '~/remoteBuildDir';
cfg.CustomSource = 'main.cu';

To generate CUDA code, use the codegen command and pass the GPU code configuration object along with the videoReaderDeploy entry-point function.

vfilename = coder.typeof('a',[1,1024]);
codegen('-config ',cfg,'-args',{vfilename},'videoReaderDeploy','-report');

Limitations

See Also

Functions

Objects

Related Examples

More About