matlab.compiler.mlspark.SparkContext Class

R2026b

Namespace: matlab.compiler.mlspark
Superclasses:

(To be removed) Interface class to initialize a connection to a Spark enabled cluster

expand all in page

matlab.compiler.mlspark.SparkContext will be removed in a future release. There is no replacement functionality. For more information, see Version History.

Description

A SparkContext object serves as an entry point to Spark™ by initializing a connection to a Spark cluster. It accepts a SparkConf object as an input argument and uses the parameters specified in that object to set up the internal services necessary to establish a connection to the Spark execution environment.

Construction

sc = matlab.compiler.mlspark.SparkContext(conf) creates a SparkContext object initializes a connection to a Spark cluster.

Input Arguments

expand all

Pass the SparkConf object as input to the SparkContext.

Example: sc = matlab.compiler.mlspark.SparkContext(conf);

See matlab.compiler.mlspark.SparkConf for information on how to create a SparkConf object.

Properties

The properties of this class are hidden.

Methods

addJar(To be removed) Add JAR file dependency for all tasks that need to be executed in a SparkContext
broadcast(To be removed) Broadcast a read-only variable to the cluster
datastoreToRDD(To be removed) Convert MATLAB datastore to a Spark RDD
delete(To be removed) Shutdown connection to Spark enabled cluster
getSparkConf(To be removed) Get SparkConf configuration parameters
parallelize(To be removed) Create an RDD from a collection of local MATLAB values
setCheckpointDir(To be removed) Set the directory under which RDDs are to be checkpointed
setLogLevel(To be removed) Set log level
textFile(To be removed) Create an RDD from a text file

Examples

collapse all

The SparkContext class initializes a connection to a Spark enabled cluster using Spark properties.

% Setup Spark Properties as a containers.Map object
sparkProp = containers.Map({'spark.executor.cores'}, {'1'}); 

% Create SparkConf object
conf = matlab.compiler.mlspark.SparkConf('AppName','myApp', ...
                        'Master','local[1]','SparkProperties',sparkProp);
% Create a SparkContext
sc = matlab.compiler.mlspark.SparkContext(conf);

More About

expand all

References

See the latest Spark documentation for more information.

Version History

Introduced in R2016b

collapse all

See Also

Classes

Topics