Friday, August 2, 2013

Hadoop rack configurations

The above script can be used for configuring the rack awareness in Hadop
Refer the blog at  http://huntingcheetah.wordpress.com/2013/02/21/configuring-rack-awareness-in-hadoop/

The code their need a small tweak which has been done as shown in the image.

The requirement was that we had limited storage in one of the nodes of the machine which cannot store more data. hence we have done a hack whereby we grouped the machines to rack so that the data is replicated along the rack  mentioned thus reducing the chances of data getting put into node with less storage .

Apart from that we also added additional directories into one of the node by modifying the hdfs-site.xml of the node by adding the

dfs.data.dir.

No comments: