After a while - automating VM builds

2 years since the last blog.
Many things changed. Professinaly and in life.

Anywasy quick to topic
The type of work i do these days is automating- some call it devops.
But I feel its the natural progression from development to more productive tasks.
Less code more automate.

I was given task of building VM’s -. Which normally is a hated job. Who would sit and run the scripts and setup a vm . Who would love the job.
My initial thought was to export it from AWS AMI.

We already had the automation doen to create AMI’s on the fly. This was done through hashicopr packer and ansible. Where ansibe did the configurations while packer did the build.
And a quick search took me to https://aws.amazon.com/ec2/vm-import/ . A service in AWS which helps you to import and export to VM form AMI . I was taken by this feature first without realising the hidden pits . So AWS only allows to export the AMI’s that was built form imported VM’s.
So what type of VM’s they allow to import - ‘windows VMs’ . hmm tricky business. icloud interoperability is still in dark ages.

Luckily we already had our automation in packer.
I had to modify the packer scripts to build the image in VMWare.

{
"builders": [
{
"boot_wait": "2s",
"vm_name": "xxxx",
"vmdk_name": "xxxxx",
"type": "vmware-iso",
"iso_url": "/Users/test/Downloads/CentOS-7-x86_64-Minimal-1708.iso",
"iso_checksum": "aae20c8052a55cf179af88d9dd35f1a889cd5773",
"iso_checksum_type": "sha1"
"ssh_username": "root",
"ssh_password": "xxxx",
"communicator": "ssh",
"ssh_pty": "true",
"headless": false,
"disk_size": 8000,
"guest_os_type": "linux",
"vmx_data": {
"cpuid.coresPerSocket": "1",
"memsize": "8000",
"numvcpus": "1"
},
"http_directory": "http",
"boot_command": [
" <tab> text ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ks.cfg<enter><wait>"
],
"ssh_wait_timeout": "10000s",
"shutdown_command": "/sbin/halt -p"
}
],
"provisioners": [
{
"type": "shell",
"inline": [
"sleep 50",
"echo 'done'"
]
}
]
}

And i came to know about hte kickstart file in linux. sometime back i have heard about it. But never had the time to actually explore it. The kickstart file is used to automate the intstallation process in linux. Its nothing but a set of command and configurations for the linux installation to consume and carry on automatically.

So to assist the script above I had to build a kickstart file. A typical kickstart fiel looks like

install
cdrom
lang en_US.UTF-8
keyboard us
timezone UTC
network --bootproto=dhcp
rootpw --plaintext xxx123
auth --enableshadow --passalgo=sha512 --kickstart
firewall --disabled --ssh --service=ssh
selinux --disabled
bootloader --location=mbr

text
skipx
zerombr

clearpart --all --initlabel
autopart

firstboot --disabled
reboot

%packages --instLangs=en_US.utf8 --nobase --ignoremissing --excludedocs
openssh-clients

@core
%end

%post --log=/root/ks.log

%end

Some of the issue si faced

==> vmware-iso: Error starting VM: VMware error: 2017-11-29T20:58:30.791| ServiceImpl_Opener: PID 19380
==> vmware-iso: Error: The operation was canceled
==> vmware-iso: Waiting 1.384998s to give VMware time to clean up...
==> vmware-iso: Deleting output directory...
Build 'vmware-iso' errored: Error starting VM: VMware error: 2017-11-29T20:58:30.791| ServiceImpl_Opener: PID 19380
Error: The operation was canceled

This happened the first few times i tried to run the packer 'build command’ . This normally happens when VMware cannot create the vm that you have mentioned. normally its issue with the CPU, no of virtual core, RAM . In my case it was the ram which i set to 32GB and since my lappie was only 32GB it couldnt create the VM as such.

The next error i faced was

==> vmware-iso: Waiting 2s for boot...
==> vmware-iso: Connecting to VM via VNC
==> vmware-iso: Typing the boot command over VNC...
==> vmware-iso: Waiting for SSH to become available...
^C==> vmware-iso: Stopping virtual machine...
==> vmware-iso: Deleting output directory...
Build 'vmware-iso' errored: Build was cancelled.
Cleanly cancelled builds after being interrupted.

Here the screen just keep waiting for ssh connection to run the ssh provisioner. Actually the packer here waits for the os installation to complete in the VM.
You can test whats happeneing by setting the headless: false . That way when packer run it spins up the terminal in VMware where all the action happens. Now in my case the issue was that the boot_command was not proper and it dint kickstart the installation of OS. I had to murk around with the centos specicif boot command to kickstart the kickstart file.

Once that was done - it was the momnet of bliss. The entire VM creation is now automated. without a single button click. I use to rember how tedious it use to be when i first tried to install linux. Today this can be done with customisation like 3-4 times in a couple of minutes. Automation rocks :)

Hamachi on centos

create a adhoc mesh network on through hamachi web interface

yum install redhat-lsb

download the hamachi binary from - https://secure.logmein.com/labs/

rpm -ivh

sudo vi /var/lib/logmein-hamachi/h2-engine-override.cfg

Ipc.User <>
sudo /etc/init.d/logmein-hamachi start

sudo hamachi do-join
sudo hamachi go-online  

ifconfig

should give comething like 

ham0      Link encap:Ethernet  HWaddr ##.##.##.##
          inet addr:##.##.##.##  Bcast:25.255.255.255  Mask:255.0.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1404  Metric:1
          RX packets:26 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:2264 (2.2 KiB)  TX bytes:672 (672.0 b) 
That whould imply that the hamachi client is up and working. Now open all the communcaiton for hamachi through the firewall

iptables -A INPUT -i ham0 -p all -j ACCEPT

That it..now ping form the remote machine ..your machine should be reachable via the hamachi vpn..

Quick Postgres Setup Fedora

Reference  links :

https://fedoraproject.org/wiki/PostgreSQL
https://wiki.postgresql.org/wiki/First_steps
https://wiki.postgresql.org/wiki/YUM_Installation
https://community.jboss.org/wiki/InstallPostgreSQLOnFedora?_sscc=t
https://wiki.postgresql.org/wiki/First_steps


sudo yum install postgresql-server postgresql-contrib
journalctl -xn
sudo postgresql-setup initdb
postgresql-setup upgrade
sudo service postgresql start

sudo passwd postgres - set password for user postgres

su -l postgres
"now login to postgres console"
pgsql
"update the password for postgres user"
\password postgres

sudo yum install pgadmin3
sudo vi /var/lib/pgsql/data/pg_hba.conf
"update the entry to support md5 authentication"
host    all             all             127.0.0.1/32            md5
 service postgresql reload

Now open pgadmin clinet tool
And login with to database with
name - postgres
and the password you have set.

Now you may be able to connect from localmachine.
 Since the service runs in 127.0.0.1:5432
If you want to connect from remote machine it should run on 192.* address your local network address. or you external ip address

In this case the postgresal should listen to those ips.You can enable by
editing
 sudo vi /var/lib/pgsql/data/postgresql.conf

and update -
listen_addresses = '*'

 service postgresql reload

sudo firewall-cmd --permanent --add-port=5432/tcp

sudo firewall-cmd --add-port=5432/tcp

Fedora - file association and application launcher.

In Fedora to install an application and et them reflected in th launcher you need to add the entry at
~/.local/share/applications/ - if that need to be shown in the launcher of the logged in user alone
/usr/share/applications/ - if it need to shown for all users

ls ~/.local/share/applications/
AdobeReader.desktop
chrome--Default.desktop
defaults.list
mimeapps.list
SQuirreL SQL Client.desktop

This is how it looks like .

And to associate a fletype to a particular application
Add the entry at
mimeapps.list - located at the same above folders.

The mimeapps.list looks like

[Added Associations]
application/pdf=evince.desktop;
text/vnd.graphviz=gedit.desktop;
image/jpeg=shotwell-viewer.desktop;
text/plain=gedit.desktop;libreoffice-calc.desktop;
application/x-x509-ca-cert=gedit.desktop;
application/xml=gedit.desktop;
text/x-java=gedit.desktop;
application/octet-stream=firefox.desktop;
application/x-trash=gedit.desktop;
application/x-shellscript=gedit.desktop;
image/png=shotwell-viewer.desktop;
application/x-wais-source=gedit.desktop;
application/x-executable=gedit.desktop;
application/x-ica=wfica.desktop


On the left hand side is the mimetype of the file. and
on the right hand side is the entry in the applications folder.

That it.

Installing hadoop snappy libraries in Amazon AMI

The aMAzon AMI instances are 64 bit and hence the defualt native libraries that come with hadoop distribution fail to load.

You get an exception like -
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

For me this happedned with hadoop-2.2.0
And I found that following post - http://stackoverflow.com/questions/19943766/hadoop-unable-to-load-native-hadoop-library-for-your-platform-error-on-centos

And for this I had to recompile the hadoop-src in the target machine. (Linux AMI).

If you are planning you need to compile the compression library also alongside.

• One can follow these blogs and reference sites. I basically followed them and a litle ingeuity.

•  Installing snappy compession library - https://code.google.com/p/snappy/

• Compiling hadoop with snappy - https://code.google.com/p/hadoop-snappy/

• General reference compiling hadoop  - http://vichargrave.com/create-a-hadoop-build-and-development-environment-for-hadoop/

• issues with automake - https://issues.apache.org/jira/browse/HADOOP-10110
• https://issues.apache.org/jira/browse/HADOOP-10117
• https://issues.apache.org/jira/browse/HADOOP-8580

• Main blog - http://www.ercoppa.org/Linux-Compile-Hadoop-220-fix-Unable-to-load-native-hadoop-library.htm

Once compiled copy the native libs to the respective folders and update the parameter - 

export HADOOP_OPTS="$HADOOP_OPTS  -server -Djava.net.preferIPv4Stack=true -Djava.library.path=$HADOOP_HOME/lib/native/"


For hadoop 1.21. follow the url https://code.google.com/p/hadoop-snappy/ completely

Setting up Anonymous ftp server in Amazon EC2

anonymous_enable=YES
local_enable=NO
write_enable=NO
local_umask=022
anon_upload_enable=NO
anon_mkdir_write_enable=NO
anon_other_write_enable=NO
anon_world_readable_only=YES
connect_from_port_20=NO
hide_ids=YES
pasv_enable=YES
pasv_min_port=1024
pasv_max_port=1048
pasv_address=<<publlic dns name for the machine>>
pasv_addr_resolve=YES
listen_port=21
xferlog_enable=YES
ls_recurse_enable=NO
ascii_download_enable=NO
async_abor_enable=YES
one_process_model=YES
idle_session_timeout=120
data_connection_timeout=300
accept_timeout=60
connect_timeout=60
anon_max_rate=2048000
dirmessage_enable=YES
listen=YES

 

edit /etc/vsftpd/vsftpd.conf

sudo /etc/init.d/vsftpd start

 

open ports for inbound communication for 20-22

1024-1048

 

disable firewall

 

Wikipedia DBPedia and Extracting information

In semantci web a major source of information is from wikipedia. It stands as the single largest source of semantic information.Other competor is Freebase whose majority data is
retrieved from wikipedia. Wikipedia has another version known as DBPedia which allows to download the dataaset in triple format. This act as a starting point for building your knowledge graphs. But often the most difficult part is retrieving data that is relavant to you.
For instance you want to retrieve the tourist destinations in india..
The long and default way to do this is to get the all the resource instances form dpedia . check if the instance is of type hotel,meuseum etc . And then retrieve them.Then go find the latitude and longitude if given. If found then retrieve those in india alone. This tedious process needs the use of mapreduce programming and lot of iterations to finally retrieve the data.
There is a short and efficient way to do this. That is by in wikipedia every indormation is categorised. Thanks to the active content editores
So in short it results in something like

In short if you could get the contents from within the category tourism in india then you are actualy buildiing the knowledge graph about tourism in india. And this can be done!!!
You need to download the categories data from wikipedia and do the refinement on it. Basically a category description in wikipedia in triples format is :(As seen in DBPedia)

step1 ) Thats means this can be further refined to get the subcatergories as well as the subjects(topics) contained within the category tourism in india.
step 2) Once the subjects are got we can check it to be a category or not.If its a category its url will be of form resource/Category . And then we repeat the step 1
step 3) Is its not a category we take those subjects and their information from instances dump of wikipedia.

This way we would be able to extract the tourist destinations as well as the places of interest in india!!!!