Trplestore-Bigdata - Cluster Notes

The WORM (Write Once, Read Many) will generate much larger files. It's
primary use is to buffer writes on a cluster before they are migrated
into read-optimized key-range shards (index segment files).

http://sourceforge.net/projects/bigdata/forums/forum/676946/topic/5903581 - storing pages in bigdata and issues

http://sourceforge.net/projects/bigdata/forums/forum/676946/topic/5908980 - multitenancy API

Data can be uploaded in multiple ways to cluster :

Using the MappedRDFDataLoader - which uses multiple threads to efficiently upload data across multiple dataservers simulatneously
Using dataLoader Much rudimentary form of data upload with out Sail interfeace. htis can be extended by us to upload data from the mappers i hope. MappedRDFDataLoader is only possible form their cluster jini federation installation.
MappedRDFdataLoader - efficient and uses Sail interface .Easy to upload the data.Already tried out in by uploading data from mappers for  data from FB

bigdata® - Graph DB

bigdata graph db can be installed as
Local Journal Model
Local DataService  Federation
Local Embedded DataService Federation
Jini Fedaration


Get the bigdata src from the url - http://bigdata.svn.sourceforge.net/viewvc/bigdata/tags/BIGDATA_RELEASE_1_2_2/
I downloaded the tag 1_2_2.

Once downloaded setupo the NAS.

I preferred NFS as my nodes where running linux alone.samba is an option if windows nodes are also thr
install - nfs-kernel-server in your mster machine

and in the client machines install nfs-common applicable to ubuntu

find the folder which you plan to share as a network folder.
add its path to the /etc/exports file
/home/****/NFSHome 192.168.***.0/255.255.255.0(rw)

Now run exportfs -r
to export your folder to the network

At the client machines mount the file system by using commands like


test it first using the mount command

mount -t nfs xxx:/home/xx/NFSHome /home/xx/NFSHome
and then add to your fstab
then run mount -a
this should ensure that your nas folder is setup successfully

Now you can run the install script in your machine whr the src was unzipped
before that modify the value in build.properties.
follow the tutorial - http://sourceforge.net/apps/mediawiki/bigdata/index.php?title=ClusterGuide
i followed it and was quite useful.

Once you run the install you will find the bigdata files installed in nas folder.
move to you nas folder and change and append more configurations like where your zookeeper server should run in /config/bigdataStandalone.config
By default the configuration it takes is bigdataStandalone.config and runs all the service in the master server.

Now run bigdataenv.sh to set the environment parameters.
after that run the bigdata start and see the logs
In my case the initial trys failed because the zookeeper server dint start up..As the ip i set was not resolwing correctly.ensure the /etc/hosts entries are correct.

once it got started the listservice.sh should display somehting like

Waiting 5000ms for service discovery.
Zookeeper is running.
Discovered 1 jini service registrars.
   192.xx.xx.xx
Discovered 7 services
Discovered 0 stale bigdata services.
Discovered 6 live bigdata services.
Discovered 1 other services.
Bigdata services by serviceIface:
  There are 1 instances of com.bigdata.jini.start.IServicesManagerService on 1 hosts
  There are 1 instances of com.bigdata.journal.ITransactionService on 1 hosts
  There are 2 instances of com.bigdata.service.IDataService on 1 hosts
  There are 1 instances of com.bigdata.service.ILoadBalancerService on 1 hosts
  There are 1 instances of com.bigdata.service.IMetadataService on 1 hosts
Bigdata services by hostname:
  There are 6 live bigdata services on graphmaster
    There are 1 com.bigdata.jini.start.IServicesManagerService services
    There are 1 com.bigdata.journal.ITransactionService services
    There are 2 com.bigdata.service.IDataService services
    There are 1 com.bigdata.service.ILoadBalancerService services
    There are 1 com.bigdata.service.IMetadataService services

Thats it..the bigdata started working in a single node..Soon i shall update on multi cluster bigdata configurations!!

------------------------------------------------------------------------------------------------------------------------------

Starting bigdata
cd to the respective NFS folder where the bigdata resides.
select the node where you want to run the zookeeper.
run ./bigdataenv
run ./bigdata start

to check if services are startup run ./listServices.sh

running nanosparqlserver -

nanoSparqlServer.sh port namespace

http://192.168.192.105:9292 - It should give you the web screen!!

--------------------------------------------------------------------------------------

Running Bidata along hadoop is an interesting challenge.
For this you could run the dataservice in each of the nodes that run hadoop.
Then you could also make use of hadoop zookeeper instead of the zookeeperquorum being stauped by bigdata.

For this comment the
org.apache.zookeeper.server.quorum.QuorumPeerMain - class in bigdatacluster.config and the zookeeper configurations.
This should free a good amount of memory required for running zookeeper and offload this to the zookeeper used by hadoop.

Cloudera in Fedora 20

If you are planning using cloudera in fedora 10 you may end up in hiccups like

Transaction check error:
  file /usr/bin/hadoop from install of hadoop-common-2.2.0-3.fc20.noarch conflicts with file from package hadoop-2.0.0+1518-1.cdh4.5.0.p0.24.el6.x86_64

this is because the new fedora 20 brings alongside the hadoop distribution which conflicts with cloudera when you try to install

to remedy you can disable other repo while  installing cloudera :

sudo yum --disablerepo="*" --enablerepo="cloudera*" install pig

you may also need to install

 sudo yum install redhat-lsb
before running above

Load testing thrift services - Custom JMeter Sampler

Thrift Sampler.

In order to load test Thrift services , we need to write a java request based Sampler. For this we need to extend the AbstractJavaSamplerClient .

I have referred the url - http://ilkinbalkanay.blogspot.in/2010/03/load-test-whatever-you-want-with-apache.html - as a beginning.

Here is a sample code snippet

public class ThriftSampler extends AbstractJavaSamplerClient {
   
    private static final Logger log = LoggingManager.getLoggerForClass();   
    private TTransport transport = null;
    private TProtocol protocol = null;
    private Hbase.Client client = null;
    private String tableName = null;
   
   
   
    @Override
    public Arguments getDefaultParameters() {
        Arguments defaultParameters = new Arguments();
        defaultParameters.addArgument("server", "");
        defaultParameters.addArgument("port", "");
        defaultParameters.addArgument("thrift-protocol", "");
        defaultParameters.addArgument("tablename", "");
       
        defaultParameters.addArgument("table-col1","");
        defaultParameters.addArgument("table-col2","");
        defaultParameters.addArgument("table-col3","");
        defaultParameters.addArgument("table-col4","");
        defaultParameters.addArgument("table-col5","");
        defaultParameters.addArgument("table-col6","");
        defaultParameters.addArgument("table-col7","");
        return defaultParameters;
    }
   
    @Override
    public void setupTest(JavaSamplerContext context) {       
        String host = context.getParameter("server");
        String port = context.getParameter("port");
   
        tableName = context.getParameter("tablename");       
        transport = new TSocket(host,Integer.parseInt(port));
        protocol = new TBinaryProtocol(transport, true, true);
        client = new Hbase.Client(protocol);
        try {
            transport.open();
        } catch (TTransportException e) {           
            e.printStackTrace();
        }       
   
    }
   
   
    public SampleResult runTest(JavaSamplerContext context) {
         og.debug("Straing the test");
         SampleResult result = new SampleResult();
         boolean success = true;
         result.sampleStart();
       
         String col1 = context.getParameter("table-col1");
         String col2 = context.getParameter("table-col2");
         String col3 = context.getParameter("table-col3");
         String col4 = context.getParameter("table-col4");
         String col5 = context.getParameter("table-col5");
         String col6 = context.getParameter("table-col6");
         String col7 = context.getParameter("table-col7");   
       
           
         ArrayList mutations = new ArrayList();
         Map attributes = null;
           
        mutations.add(new Mutation(false, Charset.forName("UTF8").encode(FAMILYNAME+COLUMNNAME1), Charset.forName("UTF8").encode(col1),true));
        mutations.add(new Mutation(false, Charset.forName("UTF8").encode(FAMILYNAME+COLUMNNAME1), Charset.forName("UTF8").encode(col2),true));           
        mutations.add(new Mutation(false, Charset.forName("UTF8").encode(FAMILYNAME+COLUMNNAME1), Charset.forName("UTF8").encode(col3),true));
        mutations.add(new Mutation(false, Charset.forName("UTF8").encode(FAMILYNAME+COLUMNNAME1), Charset.forName("UTF8").encode(col4),true));
        mutations.add(new Mutation(false, Charset.forName("UTF8").encode(FAMILYNAME+COLUMNNAME1), Charset.forName("UTF8").encode(col5),true));
        mutations.add(new Mutation(false, Charset.forName("UTF8").encode(FAMILYNAME+COLUMNNAME1), Charset.forName("UTF8").encode(col6),true));
        mutations.add(new Mutation(false, Charset.forName("UTF8").encode(FAMILYNAME+COLUMNNAME1), Charset.forName("UTF8").encode(col7),true));       
           
        try {
            client.mutateRow(tableName, rowKey, mutations, attributes);
        } catch (IOError e) {
            e.printStackTrace();
        } catch (IllegalArgument e) {
            e.printStackTrace();
        } catch (TException e) {
            e.printStackTrace();
        }       
         result.sampleEnd();
         result.setSuccessful(success);
         return result;
    }
   
    @Override
    public void teardownTest(JavaSamplerContext context) {
        super.teardownTest(context);
        transport.close();
    }

The pom.xml for the sampler would include following dependency be like :

org.apache.jmeter
        ApacheJMeter_core
        2.10

 org.apache.jmeter
        ApacheJMeter_java
        2.10

org.apache.thrift
        libthrift
        0.9.0

org.apache.hbase
        hbase
        0.94.6-cdh4.4.0

 Once compiled you need to configure the dependent libraries so that JMeter loads them this is done by configuring the
plugin_dependency_paths=
property in jmeter.properties in the bin folder inside Jmeter.

The newly compiled and packaged jar containing the Thrift sampler is put in the lib/ext folder inside JMeter folder .

in many cases you will need to enable logging to see how your sampler works for this configure the logging inside the jmeter.properties by enabling debug for your class by adding
log_level.=DEBUG
Also add this property
jmeter.loggerpanel.display=true

To enable logging for your code to be displayed in the log console within jmeter.

Once these are deployed you can start with a load by creating a template something like this:
For my case I configured CSVDataset for reading the test data from csv and then using the custom sampler for firing them onto the thrift server.

Then in my CustomThriftSampler I have configured the variables which i have mapped in CSVDataset configuration

Thats it now you can start firing your dats into Hbase through thrift and load test the performance..

Debugging and Testing MR codes within IDE

One of the nightmares associated with writing MR code is with the difficulty associated with debugging and tracing the program. Since it is run as a Job in a cluster many newcomers find it very annoying. A solution to this is writing the jobs and run them in local standalone modes , so tat one cna debug and test as normal codes from within the IDE and then deploy them to clusters for running. And all these need to happen from within the environment.

We were able to do it using spring hadop and eclipse IDE. In short I evelop the jobs within the eclipse IDE debug and test them in single standalone jobtrackers running from within the IDE and then finally deploy them to original clusters.

Here is the spring and hadoop configuation and the test java class :

    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:batch="http://www.springframework.org/schema/batch"
    xmlns:hdp="http://www.springframework.org/schema/hadoop"
    xmlns:context="http://www.springframework.org/schema/context"
    xmlns:util="http://www.springframework.org/schema/util"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
    http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch-2.1.xsd
    http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop.xsd
    http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
    http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-3.0.xsd">   
  
    
        hbase.zookeeper.quorum=xxxxx01 // Only required if you are connecting to hbase
        hbase.zookeeper.property.clientPort=2181 //only required if you are connecting to hbase
        hbase.mapred.outputtable=xxxxxx
    

   
   

  
   
   
    output-path="xxxxx"
    jar-by-class="com.xxx.xxx.xxx.xxx.xxxxx"
    jar="classpath:xxxxx-0.0.1-job.jar"                         
    />
   
      
   
   
 The sample code for testing this job is :

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations={"/ApplicationContext.xml"})
public class XXXXTest {
   
    @Inject
    JobRunner jobRunner;

    @Inject
    Job xxxxJob;

    @Test
    public void test() {
       
        Logger log = Logger.getLogger(XXXXTest.class);
        log.info("Started the test!!");
       
        Configuration conf = xxxxJob.getConfiguration();
       
       //Any configuration that you need to perform upon the job should be done here !!       
       
        try {
            jobRunner.call();
        } catch (Exception e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
       
    }

Common pitfalls!!

Caused by: java.lang.RuntimeException: hbase-default.xml file seems to be for and old version of HBase (0.94.6-cdh4.4.0), this version is Unknown
    at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:68)
    at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:100)
    at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:111)
    at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:120)
    at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:181)

This happens due to some class loading problems in with hbase. Idealy the hbase should be loading from the bundled jars but some times this wired thing happens because of it loading from elsewhere may be from the running projects hbase jars. This can be avoided by adding an hbase-default.xml which mentions not to check for version issues.

Before making this update also make sure there are no duplicate hbase jars in the library that is making this problem.

The contents of hbase-defaul.xml is:

    hbase.defaults.for.version.skip
    true
    Set to true to skip the 'hbase.defaults.for.version' check. Setting this to true can be useful in contexts other than the other side of a maven generation; i.e. running in an ide. You'll want to set this boolean to true to avoid seeing the RuntimException complaint: "hbase-default.xml file seems to be for and old version of HBase (0.92.1), this version is X.X.X-SNAPSHOT"
 
   

DataDrivenInputFormat and DBInputFormat

Our requirement was to read data from a database and insert them into Hbase as a backup flow . For postgresql this was straightforward

        String databaseDriver="org.postgresql.Driver";
        String databaseURL = "jdbc:postgresql://xxxx:5432/xxxxdb";
        String databaseUsername="xxxxxx";
        String databasePassword="xxxxxxxx";
       
        job.setInputFormatClass(DBInputFormat.class);
        String [] fields = {"xwxwxwxw","xwxwxwxw","xwxwxwxw","xwxwxwxw","xwxwxwwxwxw","xwxwxwxw","xwxwxwxw"};
        DBConfiguration.configureDB(conf,databaseDriver ,databaseURL, databaseUsername,databasePassword);  
DBInputFormat.setInput(job,DBRecord.class,"xwxwxwxw",null,"xw",fields);    
        job.setOutputFormatClass(TableOutputFormat.class);

But then we had another dump which was  in SQL server. We got an error saying

java.io.IOException: Incorrect syntax near 'LIMIT'. at org.apache.hadoop.mapreduce.lib.db.DBRecordReader.nextKeyValue(DBRecordReader.java:235)

This was because the default DBInputFormat used LIMIT and OFFSET for creating splits from the database records. In MSSQL and Oracle these were not supported.

There are some DBSpecific record readers. In such cases we can use the DataDrivenInputFormat. Here we are required to give two queries one which retrieve the data for the split and other to retrieve tthe total count of the records. The new configuration looked like this

        String databaseDriver="com.microsoft.sqlserver.jdbc.SQLServerDriver";
        String databaseURL = "jdbc:postgresql://xxxx:1433/xxxxdb";
        String databaseUsername="xxxxxx";
        String databasePassword="xxxxxxxx";
        DBConfiguration.configureDB(conf,databaseDriver ,databaseURL, databaseUsername,databasePassword);
 
       String inputQuery = "SELECT * FROM 'xxxxxxxx' WHERE $CONDITIONS";
       String boundQuery="SELECT MIN(id),MAX(id) FROM 'xxxxxxx'"
      DataDrivenDBInputFormat.setInput(job, DBRecord.class, inputQuery, boundQuery);

Now that run fine except I had some issues like

 main" java.io.IOException: The index 2 is out of range.
    at org.apache.hadoop.mapreduce.lib.db.DataDrivenDBInputFormat.getSplits(DataDrivenDBInputFormat.java:193)
    at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:1063)

Which was because the DBRecord.class for mapping the table to VO was stale and dint included the id as it was written for the DBInputFormat. Now with the DataDrivenInoutFormat I also had to incorporate the id as that is used for calucalting the bound and splits. And so I also had to incorporate the id in the DBRecord.java. That solved the problem !!

Happy coding

Thrift service - learning new

Our requirement was to put data into Hbase from an external system(.Net).
Our first thought was to write our own WebService which would put data into the Hbase through the native client.

Apart from this we had a number of challenges:;

• But we had an issue of separate cluster and dev environmnet .
• Out input data volume is quite high of the tune of 40GB/hr
• Each data log was of the tune of 6-10mb size

So we cannot take risk to write a WebService which can handle this data volume. So went for thrift hoping that it should be able to handle such heavy volume traffic.

So started off with thrift.But issue struck first.The only way our dev environment can access the cluster environmnet was throught the 80 port-basically HTTP protocol. And that too through a httpd proxy.
We though why not the thrift clients should be able to connect through the proxy, we can just proxy the port for the Thrift server component.

However it failed. Now we started  investigating and found that thrift that comes packaged alongside CDH4 does'nt support the  sending over HTTP. It was basically Binary protocol over TCP. So we required something like HAProxy to proxy the TCP protocol over.

Just out of curiosity I tried out the sample Clculator stub skeleton sample that comes with Thrift downloads. I downloaded the src and had to compile it for Centos. This was a challenge as the documentation was so few.

The configure and build commands we had to try and test each and every time.
and finally this worked for us to compile thrift from source for java

configure --without-python --without-cpp --with-java=yes ANT=/home/ibsuser/Applns/apache-ant-1.9.2/bin/ant JAVA_HOME=/usr/java/jdk1.6.0_31

Manually set the JAVA_HOME in the main makefile and the make file for the Java component.

sudo make install

generating java client code -
 thrift -o ./test2 --gen java tutorial.thrift

We ran the calculator service and tried connecting it through the client something like this

TTransport transport;
transport = new TSocket("*********", 9090);
TProtocol protocol = new  TBinaryProtocol(transport);
Calculator.Client client = new Calculator.Client(protocol);
perform(client);

 Our calculator service was started something like this

ServerTransport serverTransport = new TServerSocket(9090);
TServer server = new TSimpleServer(new Args(serverTransport).processor(processor));

We ran a TCPMonitor(Proxy) between them and tried connecting the client and server . It failed.
It was here i came across a blog which told me to start the service as a servlet to support clinet to connect through Http protocol or http proxy. So we followed and started the service as a TServlet.

within a class that extends TServlet
   super(new Calculator.Processor(new CalculatorHandler()),new TBinaryProtocol.Factory());

And now we connected to it through an http thrift client

HttpClient thriftTrans = new THttpClient("***********");
TBinaryProtocol thriftProt = new TBinaryProtocol(thriftTrans);
 thriftTrans.open();
Calculator.Client client = new Calculator.Client(thriftProt);
perform(client);

 Now I can add the Http proxy in between them and it works. So that solved my problem. But I was hesitant going for this as http protocol is not good for large chunks of data. So we reverted to the plan of Using HAProxy and normal vanilla Thrift server and client coniguration.

And I am still reading this for furthur refining our approach

- http://en.wikipedia.org/wiki/Apache_Thrift