ExportOptions

Export options for event stream

Since R2022b

This object requires Streaming Data Framework for MATLAB^® Production Server™.

Description

An ExportOptions object specifies how MATLAB exports data from timetables to external event streams, such as Kafka^® streams. The object contains properties that control the data export process, including the transformation of event data to the specified type.

Creation

Create an ExportOptions object by using the detectExportOptions function. This function detects and populates the export rules based on the configuration of the event stream specified by stream and the variables in the timetable row specified by row.

opts = detectExportOptions(stream,row)

After creating opts, use setvartype to change the types of the variables as they are exported from MATLAB to the stream. To export stream data, use the writetimetable function.

Properties

Scope — Scope of schema

"None" | "Event" | "Window" | "Stream"

Scope of the schema, specified as one of these options:

• "None" — Schema has no scope.

• "Event" — Schema applies to a single event.

• "Window" — Schema applies to all events in the window.

• "Stream" — Schema applies to all windows in the stream.

Use Scope to optimize how often to refresh the schema.

Data Types: string | char

Schema — Schema describing variables being exported

JSON-formatted text | function handle | Schema object

Schema describing the MATLAB variables being exported to the event stream, specified as one of these values:

• JSON-formatted text (string or character vector)

• Function handle that evaluates to JSON-formatted text

• Schema object that produces JSON-formatted text

This schema defines the rules for converting variables to the appropriate data type.

Example: '[{"name":"N","type":"int64","size":[1,1],"missingValue":0,"categorical":false},{"name":"X","type":"double","size":[1,1],"missingValue":"","categorical":false}]' describes a schema with two variables: N (int64) and X (double).

Data Types: char | string | function_handle

OutputLocation — Location of schema

"Event" | "Window"

Location of the schema, specified as one of these options:

• "Event" — Schema is embedded in a single event.

• "Window" — Schema is embedded in an event window.

Data Types: char | string

Format — Structure of text representation of schema

"Event" | "InfluxDB"

Structure of the text representation of the schema, specified as one of these options:

• "Event" — Native, default even stream format

• "InfluxDB" — Format specifically used by InfluxDB

Data Types: char | string

Content — Content of schema stored in output location

"None" | "Literal" | "GeneratorFunction" | "RegistryID"

Content of the schema as it is stored in OutputLocation, specified as one of these options:

• "None" — No schema specified

• "Literal" — The literal schema (a struct or JSON text)

• "GeneratorFunction" — A function handle that generates the schema

• "RegistryID" — A string that identifies the schema

Data Types: char | string

Object Functions

setvartype

Set data types used to import and export variables to stream

Examples

Create Export Options from Event Stream Data

Assume that you have a Kafka server running at the network address kafka.host.com:9092 that has the topics Triangles and numericTriangles.

Create a KafkaStream object connected to the Triangles topic.

inKS = kafkaStream("kafka.host.com",9092,"Triangles");

Read events from the Triangles topic into a timetable. Preview the data by viewing the first row. The a, b, and c triangle side lengths are stored as strings.

tt = readtimetable(inKS);
row = tt(1,:)

row =

  1×3 timetable

     timestamp      a       b       c  
    ___________    ____    ____    ____

    03-Sep-2022    "15"    "31"    "36"

Use detectExportOptions to generate an ExportOptions object from the Kafka stream object. The function obtains the types used to export the variables from the first row of the timetable.

opts = detectExportOptions(inKS,row);

Use getvartype to confirm that the side length variables are currently exported to the stream as strings.

type = getvartype(opts,["a" "b" "c"]);

type = 

  1×3 string array

    "string"    "string"    "string"

Update the export options so that the side lengths are exported as double values. Confirm the updated options by using getvartype.

opts = setvartype(opts,["a","b","c"],"double");

[name,type] = getvartype(opts);
fprintf("%s: %s\n", [name; type])

a: double
b: double
c: double

Connect to the stream to export data to numericTriangles.

outKS = kafkaStream("kafka.host.com",9092,"numericTriangles", ...
    ExportOptions=opts)

outKS = 

  KafkaStream with properties:

                  Topic: "numericTriangles"
                  Group: "85c42e39-695d-467a-86f0-f0095792e7de"
                  Order: EventTime
                   Host: "kafka.host.com"
                   Port: 9092
      ConnectionTimeout: 30
         RequestTimeout: 61
          ImportOptions: "None"
          ExportOptions: "Source: string"
          PublishSchema: "true"
             WindowSize: 50
            KeyVariable: "key"
            KeyEncoding: "utf16"
                KeyType: "text"
           KeyByteOrder: "BigEndian"
           BodyEncoding: "utf8"
             BodyFormat: "JSON"
              ReadLimit: "Size"
    TimestampResolution: "Milliseconds"

Export the timetable to the new stream. The triangle side lengths in this stream are of type double.

writetimetable(outKS,tt);

Version History

Introduced in R2022b