Issues with Hadoop Configuration

6 views (last 30 days)
Nash Gould
Nash Gould on 25 Sep 2019
Commented: Jason Ross on 2 Oct 2019
I am trying to integrate a small Hadoop cluster with 4 nodes into MatLab and in order to test if I am communicating with it properly I have built a short MapReduce script:
%Creating the Hadoop Cluster
setenv('HADOOP_HOME', 'C:\hadoop');
setenv('SPARK_HOME', 'C:\spark');
cluster = parallel.cluster.Hadoop('HadoopInstallFolder', 'C:\hadoop\',...
'SparkInstallFolder', 'C:\\spark\');
cluster.HadoopProperties('mapred.job.tracker')='nodemaster:50031';
cluster.HadoopProperties('fs.default.name')='hdfs://nodemaster:9000';
%Creating a datastore with the library_data csv as the file
ds = datastore('hdfs:///library_input/thing.csv', 'SelectedVariableNames',...
'Subjects', 'ReadSize', 1000);
outputDS = 'hdfs:///library_output/MatLab/';
mr = mapreducer(cluster);
outds = mapreduce(ds, @mapper, @reducer, mr, 'OutputFolder', outputDS);
function mapper(data, ~, intermKVStore)
%Converting the data to a string array
data = string(data.Subjects);
%Storing each subject as a key
parfor member = data
for thing = split(member, ', ')
add(intermKVStore, thing, 1);
end
end
end
function reducer(intermKey, intermValIter, outKVStore)
%Initializing the number of books with the chosen subject as 0
numBooks = 0;
% Adding the number of books up
while hasnext(intermValIter)
numBooks = numBooks + getnext(intermValIter);
end
add(outKVStore, intermKey, numBooks);
end
When I go to run this script though, it gives me the following error:
Error using mapreduce (line 124)
The HADOOP job failed to submit. It is possible that there is some issue with the HADOOP configuration.
Error in HadoopFunc (line 16)
outds = mapreduce(ds, @mapper, @reducer, mr, 'OutputFolder', outputDS);
I am probably configuring the Hadoop cluster wrong for MatLab, but would anybody be able to point me in the direction of a setting I should change in order to be able to run it?

Answers (2)

Jason Ross
Jason Ross on 26 Sep 2019
The instructions for R2019b are here. Note that there have been some changes in setting up and configuring the integration over time (generally -- you need to set a couple configuration options for Hadoop and make sure certain properties aren't "final"), so if you aren't on R2019b I would suggest looking at the documentation for the release you are running. Previous releases are archived here, and you should look in the "MATLAB Distributed Computing Server" topic, under "Configure a Hadoop Cluster" for the relevant steps.
It's also worth verifying that you can run the examples that Hadoop ships with from the CLI. There are instructions here for running a trivial "word count" example. It's also good to verify that the basic hadoop command set works as anticipated, commands like "hadoop dfs -ls" and mkdir/put/get work (and have the correct user IDs, permissions).
If the job is getting to submission there might also be interesting output in the job logs that could shed some more light.
  3 Comments
Nash Gould
Nash Gould on 27 Sep 2019
Even with these values set in yarn-site.xml, I am still getting the same error.

Sign in to comment.


Jason Ross
Jason Ross on 30 Sep 2019
Edited: Jason Ross on 30 Sep 2019
That's too bad ... looking through my notes for my setup here:
  • Did you distribute yarn-site.xml to all the nodes and restart yarn?
  • core-site.xml: I have port 54310 specifed in here. In my example (similar to yours) I use this port number, e.g.
ds = datastore('hdfs://myservername:54310/datasets/sampledataset/*.csv', 'TreatAsMissing', 'NA');
  • hadoop-env.sh: I only have JAVA_HOME set
  • hdfs-site.xml: there's nothing special in here: where hdfs matadata is stored, data disrectories, secondary name node
  • mapred-env.sh: nothing set
  • mapred-site.xml: set mapreduce.map.env and mapreduce.reduce.env to /local/hadoop. mapreduce.application.classpath is set to /local/hadoop/share/hadoop/mapreduce/*,/local/hadoop/share/hadoop/mapreduce/lib/*. Other stuff sets job history and sets yarn as the mr framework.
  • yarn-env.sh: I set a heapsize. That's it.
  • I set the two properties described above, set up yarn to handle Spark, set yarn.scheduler.maximum-allocation-mb to ${yarn.nodemanager.resource.memory-mb}, and set up log retention.
  3 Comments
Jason Ross
Jason Ross on 2 Oct 2019
Looking at your files, I see you are specifying 9000 as the port in core-site.xml. This is not the same port as I'm using. You likely need to change your datastore line to use that port, e.g.
ds = datastore('hdfs://nodemaster:9000/library_input/thing.csv', 'SelectedVariableNames', 'Subjects', 'ReadSize', 1000);
assuming that /library_input/thing.csv is the correct full path to that file -- you should be able to browse the HDFS from the web interface to verify.

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by