Installing hadoop snappy libraries in Amazon AMI

The aMAzon AMI instances are 64 bit and hence the defualt native libraries that come with hadoop distribution fail to load.

You get an exception like -
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

For me this happedned with hadoop-2.2.0
And I found that following post - http://stackoverflow.com/questions/19943766/hadoop-unable-to-load-native-hadoop-library-for-your-platform-error-on-centos

And for this I had to recompile the hadoop-src in the target machine. (Linux AMI).

If you are planning you need to compile the compression library also alongside.

• One can follow these blogs and reference sites. I basically followed them and a litle ingeuity.

•  Installing snappy compession library - https://code.google.com/p/snappy/

• Compiling hadoop with snappy - https://code.google.com/p/hadoop-snappy/

• General reference compiling hadoop  - http://vichargrave.com/create-a-hadoop-build-and-development-environment-for-hadoop/

• issues with automake - https://issues.apache.org/jira/browse/HADOOP-10110
• https://issues.apache.org/jira/browse/HADOOP-10117
• https://issues.apache.org/jira/browse/HADOOP-8580

• Main blog - http://www.ercoppa.org/Linux-Compile-Hadoop-220-fix-Unable-to-load-native-hadoop-library.htm

Once compiled copy the native libs to the respective folders and update the parameter - 

export HADOOP_OPTS="$HADOOP_OPTS  -server -Djava.net.preferIPv4Stack=true -Djava.library.path=$HADOOP_HOME/lib/native/"


For hadoop 1.21. follow the url https://code.google.com/p/hadoop-snappy/ completely