How to run the example 'Run mapreduce on a Hadoop Cluster'?

1 vue (au cours des 30 derniers jours)
Jingyu Ru
Jingyu Ru le 22 Juil 2015
Commenté : lov kumar le 4 Juin 2019
Hadoop version 1.2.1 Matlab version 2015a
Linux ubuntu 14.
I install Hadoop for a pseudo-distributed configuration (all of the Hadoop daemons run on a single host).
It is success to run the example 'wordcount'in Hadoop.
And it is success to read the data from the HDFS through the Matlab.
But when I try to run the example in Matlab 'Run mapreduce on a Hadoop Cluster',I failed.
It shows that the Map 0% and Reduce 0%.
There is my Matlab codes
cluster = parallel.cluster.Hadoop;
cluster.HadoopProperties('mapred.job.tracker') = 'localhost:50031';
cluster.HadoopProperties('') = 'hdfs://localhost:8020';
% ds = datastore('hdfs:/user/root/rrr')
outputFolder = '/home/rjy/logs/hadooplog';
mr = mapreducer(cluster);
ds = datastore('airlinesmall.csv','TreatAsMissing','NA',...
meanDelay = mapreduce(ds,@meanArrivalDelayMapper,@meanArrivalDelayReducer,mr,...
Error log is shown as follows,
Error using mapreduce (line 100)
The HADOOP job failed to complete.
Error in run_mapreduce_on_a_hadoop (line 12)
meanDelay = mapreduce(ds,@meanArrivalDelayMapper,@meanArrivalDelayReducer,mr,...
Caused by:
The HADOOP job was not able to start MATLAB for attempt 1 of 'MAP' task 0. The user
home directory '/homes/' for the cluster either did not exist or was not writable by
the HADOOP user. Check the documentation on how to set the user home directory for the
What's the '/homes/' means? I have never seen it before. My hadoop directory is './home/rjy/soft/hadoop-1.2.1'
I wish to know how to solve this problem. I have tried many methods to solve it.
Please give me some suggestions. Thanks.

Réponse acceptée

Esther le 26 Juil 2015
Hi Jingyu,
Try editing the mapred-site.xml to set the value of this property: mapreduce.admin.user.home.dir
which will be set to a value of that of your hadoop directory: './home/rjy/soft/hadoop-1.2.1'
Maybe this will force it to look to your hadoop installation instead of /homes.
Hope this can solve the problem.
Cheers, Esther
  2 commentaires
Jingyu Ru
Jingyu Ru le 28 Juil 2015
Thanks a lot, I have solved the problem by set the mapred-site.xml,
Thank you very much!!! I'm so exciting!!!
Esther le 31 Juil 2015
You're welcome!

Connectez-vous pour commenter.

Plus de réponses (1)

lov kumar
lov kumar le 28 Mai 2019
How to fix this error
Error using mapreduce (line 124)
The HADOOP job failed to submit. It is possible that there is some issue with the HADOOP configuration.
Error in bg (line 14)
meanDelay = mapreduce(ds,@meanArrivalDelayMapper,@meanArrivalDelayReducer,mr,...
  4 commentaires
lov kumar
lov kumar le 4 Juin 2019
I am using this code:
ds = datastore('hdfs://localhost:9000/lov/airlinesmall.csv',...
cluster = parallel.cluster.Hadoop;
cluster.HadoopProperties('mapred.job.tracker') = 'localhost:9000';
cluster.HadoopProperties('') = 'hdfs://localhost:8088';
result = mapreduce(ds,@maxArrivalDelayMapper,@maxArrivalDelayReducer,'OutputFolder','hdfs://localhost:9000/lov');
lov kumar
lov kumar le 4 Juin 2019
I am using single machine.

Connectez-vous pour commenter.


Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by