Streaming Data Framework for MATLAB Production Server Basics
Use Streaming Data Framework for MATLAB® Production Server™ to read from and write to event streaming platforms, such as Kafka®. Using this framework, you can:
Develop a streaming analytic function in MATLAB that filters, transforms, records, or processes event stream data.
Connect to a streaming source and test how the analytic function reads from and writes to event streams by using Streaming Data Framework for MATLAB Production Server functions.
Simulate the production environment for testing your streaming analytic algorithms (requires MATLAB Compiler SDK™).
Package the analytic function (requires MATLAB Compiler SDK) and deploy it to MATLAB Production Server.
Install Streaming Data Framework for MATLAB Production Server
Install the Streaming Data Framework for MATLAB Production Server support package from the MATLAB Add-On Explorer. For information about installing add-ons, see Get and Manage Add-Ons (MATLAB).
After your installation is complete, find examples in
is the root folder of support packages on your system. To get the path to this folder, use
Streaming Data Framework for MATLAB Production Server has the same system requirements as MATLAB. For more information, see System Requirements for MATLAB.
Write Streaming Analytic MATLAB Function
The event stream analytic function typically consumes a stream of input events and can produce a stream of output events. It can filter, transform, record, or process the stream of events by using any MATLAB functionality that is deployable to MATLAB Production Server.
Event stream analytic functions process windows or batches of events. An event consists of three parts:
Key — Identifies the event source
Timestamp — Indicates the time at which the event occurred
Body — Contains event data, specified as an unordered set of (name, value) pairs
Analytic functions read events into a timetable. Each row of the timetable represents a streaming event, typically in chronological order. If the analytic function produces results, they must also be timetables.
When processing a stream, you can call an analytic function several times, because the window size is typically much smaller than the number of messages in the stream. The stateless execution model of MATLAB Production Server isolates the processing of each window, so the processing of one window does not affect the processing of the next. Stateful functions that require interaction between the processing of consecutive windows specify a MATLAB structure that is preserved between windows and passed to the next invocation of the analytic function.
An analytic function can have one of three signatures:
|results = ||Stateless analytic function that emits a stream of results|
|[ results, state ] = ||Stateful analytic function that preserves state between batches and emits a stream of results|
|Stateless analytic function that does not emit a stream of results|
Stateless Analytic Function
plotSierpinski function is an example of a stateless
plotSierpinski plots the X and Y columns of the
input timetable. The source code for this function and a script to run it is located in
function howMany = plotSierpinski(xyData) hold on arrayfun(@(x,y)plot(x,y,'ro-', 'MarkerSize', 2), [xyData.X], [xyData.Y]); hold off drawnow count = height(xyData); howMany = timetable(xyData.Properties.RowTimes(end), count); end
Stateful Analytic Function
recamanSum function is an example of a stateful
analytic function. In stateful functions, the data state is shared between events and past
events can influence the way current events are processed.
computes the cumulative sum of a numeric sequence. In returns two values:
cSum— A table that contains the cumulative sum of the elements in the stream
state— A structure that contains the final value of the sequence
The source code for the
recamanSum function, it initialization
initRecamanSum, and the scripts used to run the analytic
function are located in the
function [cSum, state] = recamanSum(data, state) timestamp = data.Properties.RowTimes; key = data.key; sum = cumsum(data.R) + state.cumsum; state.cumsum = sum(end); cSum = timetable(timestamp, key, sum); end
Process Kafka Events Using MATLAB
To process events from a stream, you create an object to connect to the stream, read events from the stream, iterate the streaming analytic function to process the several windows of events, and, if the analytic function produces results, create a different stream object to write the results to stream.
The following code sample gives an
overview of processing one window of events using the framework. Assume that you have a
Kafka host running at the network address
that has a topic
recamanSum_data. Also, assume that the
recamanSum_data topic contains the first 1000 elements of the Recamán
KafkaStreamobject for reading from and writing to the
inKS = kafkaStream("kafka.host.com",9092,"recamanSum_data");
Read events from the
recamanSum_datatopic into a timetable
tt = readtimetable(inKS);
recamanSumfunction and calculate the cumulative sum of a part of Recamán's sequence in
recamanSumis a stateful function, first call the
initRecamSumfunction, which initializes state.
state = initRecamanSum(); [results, state] = recamanSum(tt,state);
For a detailed example of how to process several windows of events, see Process Kafka Events Using MATLAB.
Simulate Production Using Development Version of MATLAB Production Server
Before deploying to MATLAB Production Server, you can test the streaming analytic function using the development version of MATLAB Production Server, which acts as a local test server. For a detailed example, see Test Streaming Analytic Function Using Local Test Server.
Deploy Streaming Analytic to MATLAB Production Server
You can also package the analytic function and deploy it to MATLAB Production Server. For a detailed example, see Deploy Streaming Analytic Function to MATLAB Production Server.