Trplestore-Bigdata - Cluster Notes

The WORM (Write Once, Read Many) will generate much larger files. It's
primary use is to buffer writes on a cluster before they are migrated
into read-optimized key-range shards (index segment files).

http://sourceforge.net/projects/bigdata/forums/forum/676946/topic/5903581 - storing pages in bigdata and issues

http://sourceforge.net/projects/bigdata/forums/forum/676946/topic/5908980 - multitenancy API

Data can be uploaded in multiple ways to cluster :

Using the MappedRDFDataLoader - which uses multiple threads to efficiently upload data across multiple dataservers simulatneously
Using dataLoader Much rudimentary form of data upload with out Sail interfeace. htis can be extended by us to upload data from the mappers i hope. MappedRDFDataLoader is only possible form their cluster jini federation installation.
MappedRDFdataLoader - efficient and uses Sail interface .Easy to upload the data.Already tried out in by uploading data from mappers for  data from FB

bigdata® - Graph DB

bigdata graph db can be installed as
Local Journal Model
Local DataService  Federation
Local Embedded DataService Federation
Jini Fedaration


Get the bigdata src from the url - http://bigdata.svn.sourceforge.net/viewvc/bigdata/tags/BIGDATA_RELEASE_1_2_2/
I downloaded the tag 1_2_2.

Once downloaded setupo the NAS.

I preferred NFS as my nodes where running linux alone.samba is an option if windows nodes are also thr
install - nfs-kernel-server in your mster machine

and in the client machines install nfs-common applicable to ubuntu

find the folder which you plan to share as a network folder.
add its path to the /etc/exports file
/home/****/NFSHome 192.168.***.0/255.255.255.0(rw)

Now run exportfs -r
to export your folder to the network

At the client machines mount the file system by using commands like


test it first using the mount command

mount -t nfs xxx:/home/xx/NFSHome /home/xx/NFSHome
and then add to your fstab
then run mount -a
this should ensure that your nas folder is setup successfully

Now you can run the install script in your machine whr the src was unzipped
before that modify the value in build.properties.
follow the tutorial - http://sourceforge.net/apps/mediawiki/bigdata/index.php?title=ClusterGuide
i followed it and was quite useful.

Once you run the install you will find the bigdata files installed in nas folder.
move to you nas folder and change and append more configurations like where your zookeeper server should run in /config/bigdataStandalone.config
By default the configuration it takes is bigdataStandalone.config and runs all the service in the master server.

Now run bigdataenv.sh to set the environment parameters.
after that run the bigdata start and see the logs
In my case the initial trys failed because the zookeeper server dint start up..As the ip i set was not resolwing correctly.ensure the /etc/hosts entries are correct.

once it got started the listservice.sh should display somehting like

Waiting 5000ms for service discovery.
Zookeeper is running.
Discovered 1 jini service registrars.
   192.xx.xx.xx
Discovered 7 services
Discovered 0 stale bigdata services.
Discovered 6 live bigdata services.
Discovered 1 other services.
Bigdata services by serviceIface:
  There are 1 instances of com.bigdata.jini.start.IServicesManagerService on 1 hosts
  There are 1 instances of com.bigdata.journal.ITransactionService on 1 hosts
  There are 2 instances of com.bigdata.service.IDataService on 1 hosts
  There are 1 instances of com.bigdata.service.ILoadBalancerService on 1 hosts
  There are 1 instances of com.bigdata.service.IMetadataService on 1 hosts
Bigdata services by hostname:
  There are 6 live bigdata services on graphmaster
    There are 1 com.bigdata.jini.start.IServicesManagerService services
    There are 1 com.bigdata.journal.ITransactionService services
    There are 2 com.bigdata.service.IDataService services
    There are 1 com.bigdata.service.ILoadBalancerService services
    There are 1 com.bigdata.service.IMetadataService services

Thats it..the bigdata started working in a single node..Soon i shall update on multi cluster bigdata configurations!!

------------------------------------------------------------------------------------------------------------------------------

Starting bigdata
cd to the respective NFS folder where the bigdata resides.
select the node where you want to run the zookeeper.
run ./bigdataenv
run ./bigdata start

to check if services are startup run ./listServices.sh

running nanosparqlserver -

nanoSparqlServer.sh port namespace

http://192.168.192.105:9292 - It should give you the web screen!!

--------------------------------------------------------------------------------------

Running Bidata along hadoop is an interesting challenge.
For this you could run the dataservice in each of the nodes that run hadoop.
Then you could also make use of hadoop zookeeper instead of the zookeeperquorum being stauped by bigdata.

For this comment the
org.apache.zookeeper.server.quorum.QuorumPeerMain - class in bigdatacluster.config and the zookeeper configurations.
This should free a good amount of memory required for running zookeeper and offload this to the zookeeper used by hadoop.

Cloudera in Fedora 20

If you are planning using cloudera in fedora 10 you may end up in hiccups like

Transaction check error:
  file /usr/bin/hadoop from install of hadoop-common-2.2.0-3.fc20.noarch conflicts with file from package hadoop-2.0.0+1518-1.cdh4.5.0.p0.24.el6.x86_64

this is because the new fedora 20 brings alongside the hadoop distribution which conflicts with cloudera when you try to install

to remedy you can disable other repo while  installing cloudera :

sudo yum --disablerepo="*" --enablerepo="cloudera*" install pig

you may also need to install

 sudo yum install redhat-lsb
before running above

Load testing thrift services - Custom JMeter Sampler

Thrift Sampler.

In order to load test Thrift services , we need to write a java request based Sampler. For this we need to extend the AbstractJavaSamplerClient .

I have referred the url - http://ilkinbalkanay.blogspot.in/2010/03/load-test-whatever-you-want-with-apache.html - as a beginning.

Here is a sample code snippet

public class ThriftSampler extends AbstractJavaSamplerClient {
   
    private static final Logger log = LoggingManager.getLoggerForClass();   
    private TTransport transport = null;
    private TProtocol protocol = null;
    private Hbase.Client client = null;
    private String tableName = null;
   
   
   
    @Override
    public Arguments getDefaultParameters() {
        Arguments defaultParameters = new Arguments();
        defaultParameters.addArgument("server", "");
        defaultParameters.addArgument("port", "");
        defaultParameters.addArgument("thrift-protocol", "");
        defaultParameters.addArgument("tablename", "");
       
        defaultParameters.addArgument("table-col1","");
        defaultParameters.addArgument("table-col2","");
        defaultParameters.addArgument("table-col3","");
        defaultParameters.addArgument("table-col4","");
        defaultParameters.addArgument("table-col5","");
        defaultParameters.addArgument("table-col6","");
        defaultParameters.addArgument("table-col7","");
        return defaultParameters;
    }
   
    @Override
    public void setupTest(JavaSamplerContext context) {       
        String host = context.getParameter("server");
        String port = context.getParameter("port");
   
        tableName = context.getParameter("tablename");       
        transport = new TSocket(host,Integer.parseInt(port));
        protocol = new TBinaryProtocol(transport, true, true);
        client = new Hbase.Client(protocol);
        try {
            transport.open();
        } catch (TTransportException e) {           
            e.printStackTrace();
        }       
   
    }
   
   
    public SampleResult runTest(JavaSamplerContext context) {
         og.debug("Straing the test");
         SampleResult result = new SampleResult();
         boolean success = true;
         result.sampleStart();
       
         String col1 = context.getParameter("table-col1");
         String col2 = context.getParameter("table-col2");
         String col3 = context.getParameter("table-col3");
         String col4 = context.getParameter("table-col4");
         String col5 = context.getParameter("table-col5");
         String col6 = context.getParameter("table-col6");
         String col7 = context.getParameter("table-col7");   
       
           
         ArrayList mutations = new ArrayList();
         Map attributes = null;
           
        mutations.add(new Mutation(false, Charset.forName("UTF8").encode(FAMILYNAME+COLUMNNAME1), Charset.forName("UTF8").encode(col1),true));
        mutations.add(new Mutation(false, Charset.forName("UTF8").encode(FAMILYNAME+COLUMNNAME1), Charset.forName("UTF8").encode(col2),true));           
        mutations.add(new Mutation(false, Charset.forName("UTF8").encode(FAMILYNAME+COLUMNNAME1), Charset.forName("UTF8").encode(col3),true));
        mutations.add(new Mutation(false, Charset.forName("UTF8").encode(FAMILYNAME+COLUMNNAME1), Charset.forName("UTF8").encode(col4),true));
        mutations.add(new Mutation(false, Charset.forName("UTF8").encode(FAMILYNAME+COLUMNNAME1), Charset.forName("UTF8").encode(col5),true));
        mutations.add(new Mutation(false, Charset.forName("UTF8").encode(FAMILYNAME+COLUMNNAME1), Charset.forName("UTF8").encode(col6),true));
        mutations.add(new Mutation(false, Charset.forName("UTF8").encode(FAMILYNAME+COLUMNNAME1), Charset.forName("UTF8").encode(col7),true));       
           
        try {
            client.mutateRow(tableName, rowKey, mutations, attributes);
        } catch (IOError e) {
            e.printStackTrace();
        } catch (IllegalArgument e) {
            e.printStackTrace();
        } catch (TException e) {
            e.printStackTrace();
        }       
         result.sampleEnd();
         result.setSuccessful(success);
         return result;
    }
   
    @Override
    public void teardownTest(JavaSamplerContext context) {
        super.teardownTest(context);
        transport.close();
    }

The pom.xml for the sampler would include following dependency be like :

org.apache.jmeter
        ApacheJMeter_core
        2.10

 org.apache.jmeter
        ApacheJMeter_java
        2.10

org.apache.thrift
        libthrift
        0.9.0

org.apache.hbase
        hbase
        0.94.6-cdh4.4.0

 Once compiled you need to configure the dependent libraries so that JMeter loads them this is done by configuring the
plugin_dependency_paths=
property in jmeter.properties in the bin folder inside Jmeter.

The newly compiled and packaged jar containing the Thrift sampler is put in the lib/ext folder inside JMeter folder .

in many cases you will need to enable logging to see how your sampler works for this configure the logging inside the jmeter.properties by enabling debug for your class by adding
log_level.=DEBUG
Also add this property
jmeter.loggerpanel.display=true

To enable logging for your code to be displayed in the log console within jmeter.

Once these are deployed you can start with a load by creating a template something like this:
For my case I configured CSVDataset for reading the test data from csv and then using the custom sampler for firing them onto the thrift server.

Then in my CustomThriftSampler I have configured the variables which i have mapped in CSVDataset configuration

Thats it now you can start firing your dats into Hbase through thrift and load test the performance..

Debugging and Testing MR codes within IDE

One of the nightmares associated with writing MR code is with the difficulty associated with debugging and tracing the program. Since it is run as a Job in a cluster many newcomers find it very annoying. A solution to this is writing the jobs and run them in local standalone modes , so tat one cna debug and test as normal codes from within the IDE and then deploy them to clusters for running. And all these need to happen from within the environment.

We were able to do it using spring hadop and eclipse IDE. In short I evelop the jobs within the eclipse IDE debug and test them in single standalone jobtrackers running from within the IDE and then finally deploy them to original clusters.

Here is the spring and hadoop configuation and the test java class :

    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:batch="http://www.springframework.org/schema/batch"
    xmlns:hdp="http://www.springframework.org/schema/hadoop"
    xmlns:context="http://www.springframework.org/schema/context"
    xmlns:util="http://www.springframework.org/schema/util"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
    http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch-2.1.xsd
    http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop.xsd
    http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
    http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-3.0.xsd">   
  
    
        hbase.zookeeper.quorum=xxxxx01 // Only required if you are connecting to hbase
        hbase.zookeeper.property.clientPort=2181 //only required if you are connecting to hbase
        hbase.mapred.outputtable=xxxxxx
    

   
   

  
   
   
    output-path="xxxxx"
    jar-by-class="com.xxx.xxx.xxx.xxx.xxxxx"
    jar="classpath:xxxxx-0.0.1-job.jar"                         
    />
   
      
   
   
 The sample code for testing this job is :

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations={"/ApplicationContext.xml"})
public class XXXXTest {
   
    @Inject
    JobRunner jobRunner;

    @Inject
    Job xxxxJob;

    @Test
    public void test() {
       
        Logger log = Logger.getLogger(XXXXTest.class);
        log.info("Started the test!!");
       
        Configuration conf = xxxxJob.getConfiguration();
       
       //Any configuration that you need to perform upon the job should be done here !!       
       
        try {
            jobRunner.call();
        } catch (Exception e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
       
    }

Common pitfalls!!

Caused by: java.lang.RuntimeException: hbase-default.xml file seems to be for and old version of HBase (0.94.6-cdh4.4.0), this version is Unknown
    at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:68)
    at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:100)
    at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:111)
    at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:120)
    at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:181)

This happens due to some class loading problems in with hbase. Idealy the hbase should be loading from the bundled jars but some times this wired thing happens because of it loading from elsewhere may be from the running projects hbase jars. This can be avoided by adding an hbase-default.xml which mentions not to check for version issues.

Before making this update also make sure there are no duplicate hbase jars in the library that is making this problem.

The contents of hbase-defaul.xml is:

    hbase.defaults.for.version.skip
    true
    Set to true to skip the 'hbase.defaults.for.version' check. Setting this to true can be useful in contexts other than the other side of a maven generation; i.e. running in an ide. You'll want to set this boolean to true to avoid seeing the RuntimException complaint: "hbase-default.xml file seems to be for and old version of HBase (0.92.1), this version is X.X.X-SNAPSHOT"
 
   

DataDrivenInputFormat and DBInputFormat

Our requirement was to read data from a database and insert them into Hbase as a backup flow . For postgresql this was straightforward

        String databaseDriver="org.postgresql.Driver";
        String databaseURL = "jdbc:postgresql://xxxx:5432/xxxxdb";
        String databaseUsername="xxxxxx";
        String databasePassword="xxxxxxxx";
       
        job.setInputFormatClass(DBInputFormat.class);
        String [] fields = {"xwxwxwxw","xwxwxwxw","xwxwxwxw","xwxwxwxw","xwxwxwwxwxw","xwxwxwxw","xwxwxwxw"};
        DBConfiguration.configureDB(conf,databaseDriver ,databaseURL, databaseUsername,databasePassword);  
DBInputFormat.setInput(job,DBRecord.class,"xwxwxwxw",null,"xw",fields);    
        job.setOutputFormatClass(TableOutputFormat.class);

But then we had another dump which was  in SQL server. We got an error saying

java.io.IOException: Incorrect syntax near 'LIMIT'. at org.apache.hadoop.mapreduce.lib.db.DBRecordReader.nextKeyValue(DBRecordReader.java:235)

This was because the default DBInputFormat used LIMIT and OFFSET for creating splits from the database records. In MSSQL and Oracle these were not supported.

There are some DBSpecific record readers. In such cases we can use the DataDrivenInputFormat. Here we are required to give two queries one which retrieve the data for the split and other to retrieve tthe total count of the records. The new configuration looked like this

        String databaseDriver="com.microsoft.sqlserver.jdbc.SQLServerDriver";
        String databaseURL = "jdbc:postgresql://xxxx:1433/xxxxdb";
        String databaseUsername="xxxxxx";
        String databasePassword="xxxxxxxx";
        DBConfiguration.configureDB(conf,databaseDriver ,databaseURL, databaseUsername,databasePassword);
 
       String inputQuery = "SELECT * FROM 'xxxxxxxx' WHERE $CONDITIONS";
       String boundQuery="SELECT MIN(id),MAX(id) FROM 'xxxxxxx'"
      DataDrivenDBInputFormat.setInput(job, DBRecord.class, inputQuery, boundQuery);

Now that run fine except I had some issues like

 main" java.io.IOException: The index 2 is out of range.
    at org.apache.hadoop.mapreduce.lib.db.DataDrivenDBInputFormat.getSplits(DataDrivenDBInputFormat.java:193)
    at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:1063)

Which was because the DBRecord.class for mapping the table to VO was stale and dint included the id as it was written for the DBInputFormat. Now with the DataDrivenInoutFormat I also had to incorporate the id as that is used for calucalting the bound and splits. And so I also had to incorporate the id in the DBRecord.java. That solved the problem !!

Happy coding

Thrift service - learning new

Our requirement was to put data into Hbase from an external system(.Net).
Our first thought was to write our own WebService which would put data into the Hbase through the native client.

Apart from this we had a number of challenges:;

• But we had an issue of separate cluster and dev environmnet .
• Out input data volume is quite high of the tune of 40GB/hr
• Each data log was of the tune of 6-10mb size

So we cannot take risk to write a WebService which can handle this data volume. So went for thrift hoping that it should be able to handle such heavy volume traffic.

So started off with thrift.But issue struck first.The only way our dev environment can access the cluster environmnet was throught the 80 port-basically HTTP protocol. And that too through a httpd proxy.
We though why not the thrift clients should be able to connect through the proxy, we can just proxy the port for the Thrift server component.

However it failed. Now we started  investigating and found that thrift that comes packaged alongside CDH4 does'nt support the  sending over HTTP. It was basically Binary protocol over TCP. So we required something like HAProxy to proxy the TCP protocol over.

Just out of curiosity I tried out the sample Clculator stub skeleton sample that comes with Thrift downloads. I downloaded the src and had to compile it for Centos. This was a challenge as the documentation was so few.

The configure and build commands we had to try and test each and every time.
and finally this worked for us to compile thrift from source for java

configure --without-python --without-cpp --with-java=yes ANT=/home/ibsuser/Applns/apache-ant-1.9.2/bin/ant JAVA_HOME=/usr/java/jdk1.6.0_31

Manually set the JAVA_HOME in the main makefile and the make file for the Java component.

sudo make install

generating java client code -
 thrift -o ./test2 --gen java tutorial.thrift

We ran the calculator service and tried connecting it through the client something like this

TTransport transport;
transport = new TSocket("*********", 9090);
TProtocol protocol = new  TBinaryProtocol(transport);
Calculator.Client client = new Calculator.Client(protocol);
perform(client);

 Our calculator service was started something like this

ServerTransport serverTransport = new TServerSocket(9090);
TServer server = new TSimpleServer(new Args(serverTransport).processor(processor));

We ran a TCPMonitor(Proxy) between them and tried connecting the client and server . It failed.
It was here i came across a blog which told me to start the service as a servlet to support clinet to connect through Http protocol or http proxy. So we followed and started the service as a TServlet.

within a class that extends TServlet
   super(new Calculator.Processor(new CalculatorHandler()),new TBinaryProtocol.Factory());

And now we connected to it through an http thrift client

HttpClient thriftTrans = new THttpClient("***********");
TBinaryProtocol thriftProt = new TBinaryProtocol(thriftTrans);
 thriftTrans.open();
Calculator.Client client = new Calculator.Client(thriftProt);
perform(client);

 Now I can add the Http proxy in between them and it works. So that solved my problem. But I was hesitant going for this as http protocol is not good for large chunks of data. So we reverted to the plan of Using HAProxy and normal vanilla Thrift server and client coniguration.

And I am still reading this for furthur refining our approach

- http://en.wikipedia.org/wiki/Apache_Thrift

Installing CDH4

Download Java from oracle site

Install it.

It will typicaly install to
/usr/java/

Now use alternatives to point java to the oracle jdk based java
alternatives --install /usr/bin/java java /usr/java/jdk1.6.0_45/bin/java 1600

Update the user's ~/.bash_profile to point
export JAVA_HOME=/usr/java/jdk1.6.0_45/bin/java

Do this in all the nodes where hadoop runs

Add cloudera Repo

sudo yum --nogpgcheck localinstall cloudera-cdh-4-0.x86_64.rpm

Install Jobtracker

sudo yum install hadoop-0.20-mapreduce-jobtrackercker

This install the following files to following dirs

logs -  /var/log/hadoop-0.20-mapreduce/

libs - /usr/lib/hadoop/ , /usr/lib/hadoop-0.20-mapreduce/

service daemon -  /etc/rc.d/init.d/hadoop-0.20-mapreduce-jobtracker

configs - /etc/hadoop , /etc/default/hadoop

Install Namenode

sudo yum install hadoop-hdfs-namenode

service daemon /etc/rc.d/init.d/hadoop-hdfs-namenode

script - /usr/lib/hadoop-hdfs/sbin/refresh-namenodes.sh

Install secondaryNamenode
Idealy should be in a machine different from the master.

In slave nodes install :
sudo yum install hadoop-0.20-mapreduce-tasktracker hadoop-hdfs-datanode

daemon service - /etc/rc.d/init.d/hadoop-0.20-mapreduce-tasktracker
library - /usr/lib/hadoop-hdfs/ /usr/lib/hadoop-0.20-mapreduce/  /usr/lib/hadoop-0.20-mapreduce/contrib/

others:
/usr/lib/hadoop-0.20-mapreduce/bin/hadoop-config.sh
/usr/lib/hadoop-0.20-mapreduce/bin/hadoop-daemons.sh
/usr/lib/hadoop-0.20-mapreduce/bin/hadoop
/usr/lib/hadoop-0.20-mapreduce/bin/hadoop-daemon.sh


After this follow the instructions in
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/4.2.0/CDH4-Installation-Guide/cdh4ig_topic_4_4.html#../CDH4-Installation-Guide/cdh4ig_topic_11_2.html?scroll=topic_11_2_2_unique_1

line by line and do the things.
 

1-st configure HDFS in the system and configure the jobracker/tasktracker daemons

some point to note is the the name node directories and data node directories need to be set permission for hdfs:hdfs and the mapred local directory should be set permission for mapred:hadoop .



When configuring the directories  for the namenodes and datanodes there are few points to be taken care of :

• Namenode directories store the metadata and edit logs
• And the Datanode directories store the block-pools

 You can also mention multiple entries for the namenodes metadata directory.
the 2nd one can idealy be a high performance NFS so that if the local disk fails then it can depend on the NFS drive.


You can configure mutliple volumes for the datanode like

 dfs.datanode.data.dir
 /data/1/dfs/dn,/data/2/dfs/dn,/data/3/dfs/dn

So if  /data/1/dfs/dn fails then it trys the next volume.
You can also mention the toleration for the failed nodesby setting the
parameter in the hdfs-site.xml
dfs.datanode.failed.volumes.tolerated .
Which means that hadoop will only return an error if the above mentioned number of volumes fail. Hence in abv example we can mention 3. So that it return an error if and only if it fails to write in all of -

/data/1/dfs/dn,/data/2/dfs/dn,/data/3/dfs/

If multiple volumes are not configured thenthis volume tolerated whoulc be idealy 0 and shouldnt be set.

scp the configuration to various host nodes.
sudo scp -r /etc/hadoop/conf.test_cluster/* testuser@test.com:/etc/hadoop/conf.test_cluster/

  Starting all service in cloudera

First time you suse the hadoop format the hdfs . The command is
 

sudo -u hdfs hadoop namenode -format

We format the user as hdfs user which is important.

for x in `cd /etc/init.d ; ls hadoop-hdfs-*` ; do sudo service $x start ; done

Some FAQ and common issues

Whenever you are getting below error, trying to start a DN on a slave machine:
>  java.io.IOException: Incompatible clusterIDs in /home/hadoop/dfs/data: namenode clusterID
= ****; datanode clusterID = ****
>
> It is because after you set up your cluster, you, for whatever reason, decided to reformat
your NN. Your DNs on slaves still bear reference to the old NN. To resolve this simply delete
and recreate data folder on that machine in local Linux FS, namely /home/hadoop/dfs/data.
Restarting that DN's daemon on that machine will recreate data/ folder's content and resolve
the problem.

In CDH4 every hadoop daemon is configured as service .
And hdfs daemons start with - hadoop-hdfs
while map-reduce daemons -  start with - hadoop-0.20-mapreduce


Often the firewall can block access b/w the nodes . We may have to add rules to enable and disable various ports to overcome this.

	Daemon	Default Port	Configuration Parameter
HDFS	Namenode	50070	dfs.http.address
	Datanodes	50075	dfs.datanode.http.address
	Secondarynamenode	50090	dfs.secondary.http.address
	Backup/Checkpoint node?	50105	dfs.backup.http.address
MR	Jobracker	50030	mapred.job.tracker.http.address
	Tasktrackers	50060	mapred.task.tracker.http.address

Daemon	Default Port	Configuration Parameter	Protocol	Used for
Namenode	8020	fs.default.name?	IPC: ClientProtocol	Filesystem metadata operations.
Datanode	50010	dfs.datanode.address	Custom Hadoop Xceiver: DataNode and DFSClient	DFS data transfer
Datanode	50020	dfs.datanode.ipc.address	IPC: InterDatanodeProtocol, ClientDatanodeProtocol ClientProtocol	Block metadata operations and recovery
Backupnode	50100	dfs.backup.address	Same as namenode	HDFS Metadata Operations
Jobtracker	Ill-defined.?	mapred.job.tracker	IPC: JobSubmissionProtocol, InterTrackerProtocol	Job submission, task tracker heartbeats.
Tasktracker	127.0.0.1:0¤	mapred.task.tracker.report.address	IPC: TaskUmbilicalProtocol	Communicating with child jobs
? This is the port part of hdfs://host:8020/. ? Default is not well-defined. Common values are 8021, 9001, or 8012. See MAPREDUCE-566. Binds to an unused local port. The above infor is taken from cloudera site : http://blog.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/ We may have to write ip rules to enable these ports. Idealy you will have to add these rules to the line just above REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited that is ACCEPT     all  --  hdnode1.test.com  anywhere            ACCEPT     all  --  hdnode2.test.com  anywhere REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited  In my case the OS is Centos and default INPUT FILTER ends with REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited And hence added the ACCEPT just before them. This would be OS specific. The point is just that the namenode should accept communication from ports defined for hadoop services. Now on checking the hdfs status at https://:50070 it shows that the service is available and the declared hadooop datanode are LIVE   In some cases you will witness erors like -10-01 12:43:02,084 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-1574577678-***************-1377251137260 (storage id DS-1585458778-***************-50010-1379678037696) service to ***.***.***.com/***************:8020 beginning handshake with NN 2013-10-01 12:43:02,097 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool BP-1574577678-***************-1377251137260 (storage id DS-1585458778-***************-50010-1379678037696) service to ***.***.***.com/***************:8020 org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException): Datanode denied communication with namenode: DatanodeRegistration(0.0.0.0, storageID=DS-1585458778-***************-50010-1379678037696, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=CID-ac14694a-bacb-4180-a747-464778d2d382;nsid=680798099;c=0)     at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registerDatanode(DatanodeManager.java:648) This happened for us when we where using DNS for hostname and IP resolution rather than adding individual nodes in /etc/hosts file. However we found out that for the NN we needed to add all slave node hostname ip details into /etc/hosts In case of slave we need to add entries for the NN and the respective slave alone.  Also manualy congifure the dfs.hosts file and add the full path to it in hdfs-site.xml Above all try restarting the cluster. It should work now The error basicaly occures when NN or Slave fails in identifying or resolving the name. For us it resolved form terminal but when running the hdfs it failed. Eventhough we were not able to pinpoint the troublemaker. The above steps rectified it..

LVM setup - Some hadoop stories

when creating our first cluster we dint listen to the advices. we made a number of mistakes :

We assigned the hadoop data and log directories to subdirectories within the root(/). It ran fine for some months when finally the hdfs got full.The hdfs consumed entire space in (/) leaving the Os to go for a slow death.
I had other empty partions availabel which i configured as alternaive to the dfs.data.dir argument.

But that couldnt save my root fs. Frequent read write by hdfs into the (/) fragmented my filesystem that a good chunk of the free space became non recoverable. atlast we decided to pull down the cluster.

Learning from the experience we decided that there are two problems when we move with conventional approach to hadoop node.

1) We need to have a mechanism by which we should be able to add extra hard disk to the nodes grow their size, without affecting the cluster existing filesystem. We also want such a system to effectively remove the space allocated when the cluster is not much heavily used. The solution was to go for LVM - Logical Volume Management.

2) Never ever use a root directory for storing the log and data directories of Hadoop.

With this we delved into creating a LVM in each node of our cluster.

I basicaly followed the instruction as given in
http://www.howtoforge.com/linux_lvm

some commands to keep in touch for the exercise are :

fdisk

create and display physical volume

pvcreate
pvdisplay

create and display volume group

vgcreate
vgdisplay

create and display logical volume

lvcreate
lvdisplay

mkfs


Once these logical volumes are created we can dynamically increase and decrease their size by issuing commands like:

lvextend -L80G /dev/fileserver/media
extends the drive form current size to 80GB

and resize the filesystem - 
resize2fs /dev/fileserver/media

for reducing the device..
first unmoun it
umount /media/data1
resize2fs /dev/fileserver/media 70GB
reduce the fs to 70GB

lvreduce -L70G /dev/fileserver/media
reduce size by 10G to 70GB

That way if our hdfs runs out of storage we can add disk in the fly and keep increasing the logicalvolumes .

Hadoop rack configurations

The above script can be used for configuring the rack awareness in Hadop
Refer the blog at  http://huntingcheetah.wordpress.com/2013/02/21/configuring-rack-awareness-in-hadoop/

The code their need a small tweak which has been done as shown in the image.

The requirement was that we had limited storage in one of the nodes of the machine which cannot store more data. hence we have done a hack whereby we grouped the machines to rack so that the data is replicated along the rack  mentioned thus reducing the chances of data getting put into node with less storage .

Apart from that we also added additional directories into one of the node by modifying the hdfs-site.xml of the node by adding the

dfs.data.dir.