Semantic and GeoSPAQRL

I was looking for best practises in using geospatial queries in SPARQL and came across this .
OGC - OpenGeoSpatial Consortium has bought about standards in GeoSpatial queries and represention. For Semantics the suggested nechanism is using
GeoSPARQL queries. GeoSPARQL queries look like

The OGC GeoSPARQL standard supports representing and querying geospatial data on the Semantic Web. GeoSPARQL defines a vocabulary for representing geospatial data in RDF, and it defines an extension to the SPARQL query language for processing geospatial data.

http://www.w3.org/2001/sw/wiki/GeoSPARQL - W3C page for GeoSPARQL.There is also a jena based implementation available at
http://code.google.com/p/geospatialweb/
Sample use case are defined in - http://geosparql.org/

PREFIX co: <http://www.geonames.org/countries/#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#> 
PREFIX geo: <http://www.w3.org/2003/01/geo/wgs84_pos#>

SELECT ?link ?name ?pop  ?lat ?lon
WHERE  {
   ?link gs:within(40.157623 -74.855347 41.077281 -73.586426) .  
   ?link gn:name ?name .  
   ?link gn:population ?pop  .
   ?link geo:lat ?lat .
   ?link geo:long ?lon
}

PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>  
SELECT ?link ?label ?pop  
WHERE  {   
   ?city gn:name "Bristol" .  
   ?link gs:nearby(?city 30) .  
   ?link gn:name ?label .  
   ?link gn:population ?pop  
   FILTER ( xsd:int(?pop) < 50000)
}

I was looking for a bigdata^tm based implementation . However i could not find a native implementation . But i came across useekm a plugin

being developed under opensahara. But using them i have to sacrifice the bigdata^tm stack and should use Sail based approach. I was not willing for this

because i was already using NanoSPARQL server for my queries for scalaility. In their case they have implemented a IndexingSail indexer separatly for

improving the performance of GeoSPARQL server. This was done on the SAIL layer and when geosparql data is added they create the extra index. This could improve

the performance. but i was alittle confused.

So i raised to them in forum - And decided to do some tweaks and make use of useekm Geosparql implementation natively in bigdata^tm . As my queries where mainly like

?link gs:nearby(?city 30) . Then I started out with getting the src of useekm and finding how they added them to bigdata^tm They made use of customFunction in bigdata for

the extra function intergration. When i furthur looked into the implementation i found that the subjects need to be

http://www.opengis.net/ont/geosparql# More details are present in https://dev.opensahara.com/projects/useekm/wiki/IndexingSail.

Sample query supported in useekm
PREFIX geo: <http://www.opengis.net/ont/geosparql#>
PREFIX geof: <http://www.opengis.net/def/function/geosparql/>
SELECT DISTINCT ?result WHERE {
  ?result <http://example.org/geometry> ?geometry.
  FILTER(geof:sfWithin(?geometry, "POLYGON((0 0, 0 1, 2 1, 1 0, 0 0))"^^geo:wktLiteral))
}

Deatils about the supported function in useekm - https://dev.opensahara.com/projects/useekm/wiki/GeoReference .

Internaly in useekm they use - http://www.vividsolutions.com/JTS/JTSHome.htm for doing all spatial computations.

Map-reduce logic

Number of maps are based on the number of splits.
Default split size is that of the block size. So that data is correctly partiioned ad block boundries.
Job - > task splitting is equal to number of splits made.
Hence this cannot be controlled. Except by changing the number of splits

The number max of task that run in a node is by default 2 .
This can be changes at each node by setting the parameter - mapred.tasktracker.map.tasks.maximum

Hence if you have a 4 core machine you can force the hadoop to run more than 2 task in a node.
And also its applicable if you are not running any reduce task.
The number of reduce task to be run can be set to zero by setting  job.setNumReduceTasks(0);

Also if in your cluster you have one node a VM with a single core. you can set the max
map task to be run on it to 1or 2 by again setting the - mapred.tasktracker.map.tasks.maximum
in the mapred-site.xml of that node.

The default timeout interval for jobtrackers waiting for job completion is 600s this can be reset by adding
    <property>
          <name>mapred.task.timeout</name>
          <value>3600000</value> <!--1hr -->
    </property>

to mapred-site.xml

json custom and conditional deserializing

Problem : Need to conditionaly deserialize to string or object based on the incoming json feed :

    @JsonDeserialize(using = LocationDeserializer.class)
    public void setLocation(Location location) {

And the custom Deserializer looks Like

@Override
    public Location deserialize(JsonParser jp, DeserializationContext ctxt)
            throws IOException, JsonProcessingException {
        ObjectMapper mapper = new ObjectMapper();
        mapper.setDeserializationConfig(ctxt.getConfig());
        jp.setCodec(mapper);
        String city = null;
        Location location = null;
        if(jp.getCurrentToken() == JsonToken.VALUE_STRING) {
           city = jp.readValueAs(String.class);
           location = new Location();
           location.setCity(city);
        } else {
           location = jp.readValueAs(Location.class);
        }
        return location;
    }

Hbase

start hbase using -  start-hbase.sh
get hbase shell - bin/hbase shell
hbase(main):001:0> status
1 servers, 0 dead, 5.0000 average load

hbase(main):004:0> list
TABLE                                                                                                                                                                          
socialdata                                                                                                                                                                     
test                                                                                                                                                                           
testtable                                                                                                                                                                      
3 row(s) in 0.0380 seconds
hbase(main):002:0> create 'socialdata', 'connection','feeds','personal'
0 row(s) in 0.2930 seconds
create table with name socialdata and columnfamily  - connection feeds personal.
hbase(main):009:0> scan 'socialdata'
display contents of a file
drop 'socialdata' - remove the table
exit - exit hbase

delete a row from hbase
- delete all the columns - delete '','',':'

 delete all contents of a row -
deleteall 'socialdata','facebook:123123123'
Inorder to configure a mapreduce job for a remote hbase cluster we have configuration like :

conf = HBaseConfiguration.create(conf);
conf.set(HConstants.ZOOKEEPER_QUORUM, "athena");
conf.set(HConstants.ZOOKEEPER_CLIENT_PORT, "2222");

//
server configurations

Hbase-site.xml

      hbase.rootdir
      hdfs://******:54310/hbase/sprout
 
      hbase.zookeeper.property.clientPort
      2222

      hbase.cluster.distributed
      true

      hbase.zookeeper.quorum
      athena

      hbase.zookeeper.property.dataDir
      /home/*****/Deploy/zookeeper
Hbase needs zookeper running to manage the cluster of masters and slaves.
ALso sync the system time across the machine -The time deifference shouldnt be above .5 minute
Ubuntu
ntpdate
Inside hbase-env.sh
set - export HBASE_MANAGES_ZK=true
To tell Hbase to manage its own zookeeper ensemble and specify the zookeeper properties in the hbase-site.xml
ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications.

When connecting from Hbase client this zookeeper quorum is used in our case quorum name is athena


Connecting programmaticaly to hbase is easier. You need to add the respectibe hbase-site.xml into your base classpath and then call the following statement
Configuration conf = HBaseConfiguration.create();
That creates the configuration for you.
piping output from hbase to a text file
echo "get '*****data','facebook:123123123'"|hbase shell > test

Solving time synchronisation in VM - virtual box

From the virtualbox menu select "Install Guest Additions" in the Device menu.
Now You will see the CDROM device in your guest machine loaded with a "media containing the installation files for guest additions"
You can select it and install the guest additions in the guest machines
Once installed restart the guest machine.

Now in the Host machien go to the virtaulBox installation folder.
Their you will find a program named VboxManage.
Using this you mention the time sync latency by issuing command from here

VBoxManage guestproperty set <guest-os-name> "VirtualBox/GuestAdd/VBoxService/-timesync-interval" 1000
VBoxManage guestproperty set <guest-os-name> "VirtualBox/GuestAdd/VBoxService/-timesync-set-threshold" 1000

that should keep your vms time sync to the accuracy of 1s!

BIgdata - compressed/zip/gz reading.

my problem was that i was having a freebase dump of 8GB. Which is gz format.

But when i explode it becomes close to 60GB. that was too large to be read into the hdfs sequentialy.



idealy i needed hadoop to read in in zip format itself . A little google search took me here -

http://blog.cloudera.com/blog/2009/11/hadoop-at-twitter-part-1-splittable-lzo-compression/



I found a personal blog of - https://github.com/kevinweil/hadoop-lzo very useful. And i followed his blog completely.

I got the code from Git. And did

ant compile-java

ant compile-native



And then i copied the native files within my - hadoop-lzo-0.4.15/lib/native/Linux-amd64-64

To the hadoop native lib  folder - hadoop-1.1.0/lib/native/Linux-amd64-64

I also copied the hadoop-lzo-0.4.15.jar to  - hadoop-lzo-0.4.15/lib (* i dbt whether this is needed. anyway did it)



Now you need to add the codec to the Hadoop configuration . For this add the following to the core-site.xml

<!--added for lzo decompress support.need to move to spring-flow -->



<property>

    <name>io.compression.codecs</name>

    <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec</value>

</property>

<property>

    <name>io.compression.codec.lzo.class</name>

    <value>com.hadoop.compression.lzo.LzoCodec</value>

</property>

Now decompressed my gz and compressed it back to lzo format.



gz -d
If your file is corrupt you can do like

      gunzip < file.gz > file.txt
now i did install lzo by
      sudo apt-get install liblzo2-dev

and then i zipped into lzo format

     lzop file.txt

Now i coped the file.lzo into hdfs using :
hadoop -dfs put /lzofiles

Now i ran the the command
./hadoop jar hadoop-lzo-0.4.15.jar  com.hadoop.compression.lzo.LzoIndexer /lzofiles

It gave output like

3/03/07 09:19:04 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library

13/03/07 09:19:04 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 6bb1b7f8b9044d8df9b4d2b6641db7658aab3cf8]

13/03/07 09:19:04 INFO lzo.LzoIndexer: LZO Indexing directory /lzofiles...

Initialy i got error like

13/03/07 09:11:46 ERROR lzo.GPLNativeCodeLoader: Could not load native gpl library

java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path

    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860)

    at java.lang.Runtime.loadLibrary0(Runtime.java:845)

For this i ensured the JAVA_LIBRARY_PATH is set correctly to hadoop native lib folder.

i tried to echo'ed the variable from within the hadoop command file. and ensured thet variable is set. You can set it by putting

export JAVA_LIBRARY_PATH = in hadoop-env.sh

Also ensure that you build those libgplcompression.so files from within your machine. because these are native calls and depend on your machine architecture and os.

Thats it.

Now this is messy i need to fine tune it to suite to the spring batch way of intergration. .I shall do if time permits.till then

Happy coding.


Installing for cloudera is very simple
install the relavant repo and install the libraries

cd /etc/yum.repos.d/ && wget http://archive.cloudera.com/gplextras/redhat/6/x86_64/gplextras/cloudera-gplextras4.repo
yum install hadoop-lzo-cdh4 hadoop-lzo-cdh4-mr1

Grub2 - saving my Fedora17 - adding a new entry

I  have bought a new SSD and installed Fedora 18 into it. Enabled TRIM on it and nived hime folder and swap out to my cylindar hard disk. Everything worked as planned until i found my fedora 17 boot menu missing.

Now i am looking for a quadruple boot with Windows 7, Fedora 17 ,Fedora 18, and Windows XP. Hmmm issues is with devices i bought during various times and their compatibilty with diff OS versions. Thanls to the reckless manufacturers they dont bother upgrading the drivers of outdated devices

Coming back my plan is to rescue my fedora 17 which disappeared from the boot menu. A little search on Grub bought me here - http://www.dedoimedo.com/computers/grub-2.html#mozTocId514088 .

I found my old boot partition safe in the old disk. I just need the grub to point to there and initiate the boot process. In Olden dys this was simple add them to the menu.1st . I rember doing that during collage days -

set root=(hd0,5)
linux /boot/vmlinuz
initrd /boot/initrd.img

•  set the root variable to (hd0,5) partition
•  linux - command loads the linux kernel
• initrd - Load an initial ramdisk for a Linux kernel image, and set the appropriate parameters in the Linux setup area in memory. This may only be used after the linux command (see linux) has been run

But with Grub2 things have changed. Now you need to add the same as a script and put them into /etc/grub.d folder.

Now to get track of my harddisk location of the boot partition i lost i will have to get to the grub-cli and try out with its auto-complete feature typing root(hd0,....(presstab) .

In my case the linux kernel is
-  vmlinuz-3.7.9-104.fc17.i686
and the ram image is 
- initramfs-3.7.9-104.fc17.i686.img

So i created a script file that does this and add that script file to the /etc/grub.d/. Also chmod it to give executable persmission.

My existing scripts in /etc/grub.d/ are:
sudo ls /etc/grub.d/
00_header  10_linux  20_linux_xen  20_ppc_terminfo  30_os-prober  40_custom  41_custom    README

So i decided to a new file like :
15_fedora17 - OS that is will be in the menu after my default linux fedora 18.

Now you can update the grub.cfg by runnimg grob-makeconfig -o

TO my suprise it automaticaly found my fedora 17 inspite of my manual addition.SO finalyy my grub config had two entries for fedora 17 one made by me and another found by fedora.

When i booted i faced another problem my swap space for fedora 17 was removed..grr
SO i had to change the boot options from grub menu and rename the swap space which was mentione din the boot param rd.lvm.lv=