Main Content

distributed

Create and access elements of distributed arrays from client

    Description

    A distributed array on the client represents an array that is partitioned out among the workers in a parallel pool. You operate on the entire array as a single entity; however, workers operate only on their part of the array and automatically transfer data between themselves when necessary. A distributed array resembles a normal MATLAB® array in the way you index and manipulate its elements, but none of its elements exist on the client. Codistributed arrays that you create inside spmd statements are accessible as distributed arrays from the client.

    Creation

    Use the distributed function or use the "distributed" option of array creation functions such as ones or zeros. For a list of array creation functions that create distributed arrays directly on workers, see Alternative Functionality.

    Description

    example

    D = distributed(ds) creates a distributed array from a datastore ds. D is a distributed array stored in parts on the workers of the open parallel pool.

    To retrieve the distributed array elements from the pool back to an array in the MATLAB workspace, use the gather function.

    example

    D = distributed(X) creates a distributed array from an array X.

    Use this syntax to create a distributed array from local data only if the MATLAB client can store all of X in memory. To create large distributed arrays, use the previous syntax to create a distributed array from a datastore, or the "distributed"option of array creation functions such as ones, zeros, or any other creation functions listed in Alternative Functionality.

    If the input argument is already a distributed array, the result is the same as the input.

    example

    D = distributed(C,dim) creates a distributed array from the Composite object C, with the entries of C concatenated and distributed along the dimension dim. If you omit dim, then the first dimension is the distribution dimension.

    All entries of the Composite object must have the same class. Dimensions other than the distribution dimension must be the same.

    example

    D = distributed(tX) converts the tall array tX into a distributed array distributed along the first dimension. tX must be defined in a parallel environment that can run distributed arrays. (since R2023b)

    Input Arguments

    expand all

    Datastore, specified as one of the following objects.

    ObjectType
    TabularTextDatastore objectText files
    ImageDatastore objectImage files
    SpreadsheetDatastore objectSpreadsheet files
    KeyValueDatastore objectMAT files as well as sequence files you produce using mapreduce
    FileDatastore objectCustom format files
    TallDatastore objectMAT-files and sequence files produced by the write function of the tall data type
    ParquetDatastore objectParquet files
    DatabaseDatastore (Database Toolbox) objectDatabase

    Array to distribute, specified as an array.

    Composite object to distribute, specified as a Composite object.

    Distribution dimension, specified as a scalar integer. The distribution dimension specifies the dimension over which you distribute the Composite object.

    Since R2023b

    Tall array to convert to a distributed array, specified as a tall array. The tall array must be defined in a parallel environment that supports distributed arrays.

    Output Arguments

    expand all

    Distributed array stored in parts on the workers of the open parallel pool, returned as a distributed array.

    Object Functions

    gatherTransfer distributed array, Composite object, or gpuArray object to local workspace
    writeWrite distributed data to an output location

    Several MATLAB toolboxes include functions with distributed array support. For a list of functions in all MathWorks® products that support distributed arrays, see All Functions List (Distributed Arrays).

    Several object functions enable you to examine the characteristics of a distributed array. Most behave like the MATLAB functions of the same name.

    isdistributedTrue for distributed array
    isrealDetermine whether array uses complex storage
    isUnderlyingTypeDetermine whether input has specified underlying data type
    lengthLength of largest array dimension
    ndimsNumber of array dimensions
    sizeArray size
    underlyingTypeType of underlying data determining array behavior

    Examples

    collapse all

    This example shows how to create and load distributed arrays using datastore.

    First, create a datastore using an example data set. This data set is too small to show equal partitioning of the data over the workers. To simulate a large data set, artificially increase the size of the datastore using repmat.

    files = repmat("airlinesmall.csv",10,1);
    ds = tabularTextDatastore(files);
    

    Select the example variables.

    ds.SelectedVariableNames = ["DepTime", "DepDelay"];
    ds.TreatAsMissing = "NA";
    

    Create a distributed table by reading the datastore in parallel. Partition the datastore with one partition per worker. Each worker then reads all data from the corresponding partition. The files must be in a shared location accessible from the workers.

    dt = distributed(ds);
    Starting parallel pool (parpool) using the 'Processes' profile ... connected to 4 workers.

    Finally, display summary information about the distributed table.

    summary(dt) 
    Variables:
    
        DepTime: 1,235,230×1 double
            Values:
    
                min          1
                max       2505
                NaNs    23,510
    
        DepDelay: 1,235,230×1 double
            Values:
    
                min      -1036
                max       1438
                NaNs    23,510
    

    This example shows how to create and retrieve distributed arrays.

    Create a small array and distribute it.

    Nsmall = 50;
    D1 = distributed(magic(Nsmall));
    Starting parallel pool (parpool) using the 'Processes' profile ...
    Connected to parallel pool with 4 workers.
    

    Create a large distributed array directly on the workers, using a build method.

    Nlarge = 1000;
    D2 = rand(Nlarge,"distributed");

    Retrieve elements of a distributed array back to the local workspace. You can use whos to determine where data in the workspace is located by examining the Class variable.

    D3 = gather(D2);
    whos
      Name           Size                Bytes  Class          Attributes
    
      D1            50x50                20000  distributed              
      D2          1000x1000            8000000  distributed              
      D3          1000x1000            8000000  double                   
      Nlarge         1x1                     8  double                   
      Nsmall         1x1                     8  double                   
    

    Start a parallel pool of workers and create a Composite array by using spmd.

    p = parpool('Processes',4);
    Starting parallel pool (parpool) using the 'Processes' profile ...
    Connected to parallel pool with 4 workers.
    
    spmd
        C = rand(3,spmdIndex-1);
    end
    C
     
    C =
     
       Worker 1: class = double, size = [3  0]
       Worker 2: class = double, size = [3  1]
       Worker 3: class = double, size = [3  2]
       Worker 4: class = double, size = [3  3]
     
    

    To create a distributed array out of the Composite array, use the distributed function. For this example, distribute the entries along the second dimension.

    d = distributed(C,2)
    d =
    
        0.6383    0.9730    0.2934    0.3241    0.9401    0.1897
        0.5195    0.7104    0.1558    0.0078    0.3231    0.3685
        0.1398    0.3614    0.3421    0.9383    0.3569    0.5250
    

    You can check how the data is distributed on the workers.

    spmd
        d
    end
    Worker 1: 
      
      This worker does not store any elements of d.
      
    Worker 2: 
      
      This worker stores d(:,1).
      
              LocalPart: [3x1 double]
          Codistributor: [1x1 codistributor1d]
      
    Worker 3: 
      
      This worker stores d(:,2:3).
      
              LocalPart: [3x2 double]
          Codistributor: [1x1 codistributor1d]
      
    Worker 4: 
      
      This worker stores d(:,4:6).
      
              LocalPart: [3x3 double]
          Codistributor: [1x1 codistributor1d]
      
    

    When you are finished with the computations, delete the parallel pool.

    delete(p);
    Parallel pool using the 'Processes' profile is shutting down.
    

    Since R2023b

    This example shows how to convert a tall array into a distributed array.

    Create a tall table using an example data set. If you have Parallel Computing Toolbox™ installed, when you use the tall function MATLAB automatically starts a parallel pool of workers, unless you turn off the default parallel pool preference. The default cluster uses local workers on your machine.

    size = 2000000;
    tt = tall(table((1:size)',randn(size,1),randn(size,1),randn(size,1), ...
        'VariableNames',["Exp","Rep1","Rep2","Rep3"]))
    Starting parallel pool (parpool) using the 'Processes' profile ...
    Connected to parallel pool with 4 workers.
    
    tt =
    
      2,000,000×4 tall table
    
        Exp      Rep1        Rep2         Rep3  
        ___    ________    _________    ________
    
         1      -1.6003      -1.2403     -1.6124
         2      0.76827      0.33907     0.77811
         3       1.4637      0.74255    -0.91635
         4      -1.2478      0.10478    -0.10097
         5        0.619      -1.2974     0.93445
         6       1.8375       0.2142     -1.1036
         7     -0.44354      -1.1438    -0.89011
         8      -1.3351    -0.036768     0.65196
         :        :            :           :
         :        :            :           :
    

    Convert the tall table into a distributed table. MATLAB partitions the data in the tall table along the first dimension and distributes it to the workers.

    dt = distributed(tt);

    Display summary information about the distributed table.

    summary(dt)
    Variables:
    
        Exp: 2,000,000×1 double
            Values:
    
                Min         1  
                Max     2e+06  
    
        Rep1: 2,000,000×1 double
            Values:
    
                Min    -5.1402 
                Max     4.8763 
    
        Rep2: 2,000,000×1 double
            Values:
    
                Min    -4.7961 
                Max     4.9875 
    
        Rep3: 2,000,000×1 double
            Values:
    
                Min    -4.8369 
                Max     5.1454 
    

    Finally, check how much data is stored on each worker. You can observe that the data is partitioned evenly over the workers.

    spmd
        dt
    end
    Worker 1: 
      
      This worker stores dt(1:500000,:).
      
              LocalPart: [500000x4 table]
          Codistributor: [1x1 codistributor1d]
      
    Worker 2: 
      
      This worker stores dt(500001:1000000,:).
      
              LocalPart: [500000x4 table]
          Codistributor: [1x1 codistributor1d]
      
    Worker 3: 
      
      This worker stores dt(1000001:1500000,:).
      
              LocalPart: [500000x4 table]
          Codistributor: [1x1 codistributor1d]
      
    Worker 4: 
      
      This worker stores dt(1500001:2000000,:).
      
              LocalPart: [500000x4 table]
          Codistributor: [1x1 codistributor1d]
      
    

    Tips

    • A distributed array is created on the workers of the existing parallel pool. If no pool exists, distributed starts a new parallel pool unless the automatic starting of pools is disabled in your parallel preferences. If there is no parallel pool and distributed cannot start one, MATLAB returns the result as a nondistributed array in the client workspace.

    Alternative Functionality

    This table lists the available MATLAB functions that create distributed arrays directly on the workers. For more information, see the Extended Capabilities section of the function reference page.

    Version History

    Introduced in R2008a

    expand all