Bundle up your customized server

Very obviously this makes a lot of sense! I agree that setting up a system from scratch is a lot of fun but you don't wanna do it every time you want to add your numbers !

Once you customize your environment, install your applications its best to make an image and save it for future instantiations.

We have two choices (as of this writing) to create an image: Store image in S3 or Store image in EBS.
The pricing model of Amazon still confuses me a bit. Although EBS has a lot of advantages over S3, it has to be costlier too !

Anyhow, I will describe how you can create images and store on both these services.

Create Image and Store in S3:


1. Launch the Instance : Duh !

2. Customize it to the needs : You better do this if you want to make any sense of this process :-)

3. Create a Bucket using S3Fox : You definitely should have S3Fox installed ! The whole purpose of this blog is to make the job easy using the various tools out there. Creating a Bucket beforehand is not really required. The bucket will be created if it doesn't exist in the future steps. Although this is a better way.

5. Upload the Private and Certificate keys to the /mnt folder on your instance. Use SCP for this.
(Aha .. this is why we installed cygwin...all little tools under one roof )

scp -i #keypair# #private-key# #cert# #instance-address#:/mnt

This should look like

scp -i gsg_keypair.pem pk-MF6AQ2TDUODJIZEGZ5Y7ZGNKNPEIPU4M.pem cert-MF6AQ2TDUODJIZEGZ5Y7ZGNKNPEIPU4M.pem root@ec2-37-212-46-94.compute-1.amazonaws.com:/mnt



6. Now Bundle up your image into a temporary file on your instance itself. (in the /mnt folder). The following command gets it going.

ec2-bundle-vol -d /mnt -k /mnt/#private-key.pem# -c /mnt/#cert.pem# -u #amazon-account-number# -r i386 -p #image-name#

Note that the Amazon Account number is the 12 digit number you'll find when you log into aws.amazon.com NOT your amazon id !


This command should look like

ec2-bundle-vol -d /mnt -k /mnt/pk-MF6AQ2TDUODJIZEGZ5Y7ZGNKNPEIPU4M.pem -c /mnt/cert-MF6AQ2TDUODJIZEGZ5Y7ZGNKNPEIPU4M.pem -u 111122223333 -r i386 -p imgname



7. Now once the image has bundled up. You would cut it into small pieces and upload it to your bucket !
The following command should take care of that :

ec2-upload-bundle -b #bucket-name# -m /mnt/#image-name#.manifest.xml -a #access-key# -s #secret-access-key#

This should look like : 

ec2-upload-bundle -b imgname -m /mnt/imgname.manifest.xml -a Q2T3RAKIAJJC6WLIJA2A -s 3La9QKLjk9AkjfgJJC6W/sMYrphJC8Tk9A


8. The last step is to register the image so that you can use it . Enter this command in cygwin on your LOCAL terminal NOT on the instance.

ec2-register #bucket# / #image-name#.manifest.xml

This should look like

ec2-register imgname/imgname.manifest.xml

Setting up your workstation for Amazon's Services

This post explains how to setup your local machine for smoother operation of Amazon's Services.
Let me explain my current setup so that its easier for you to put things in context.

Workstation Configuration (this is the PC used to connect to the clouds and have fun there ;-):


OS: Windows XP. (Highly recommend installing Cygwin. Whats cygwin? Basically Unix type environment on top of windows. Useful because it has many utility stuff which comes naturally with Unix but not with windows. The entry below should explain them).

SSH Client : You could use putty.exe. However I have cygwin installed which has a lot of other utility programs included along with ssh (such as scp,ftp etc).

EC2 Api Tools : This is the set of tools provided by Amazon AWS which can be used for controlling the Instances on Amazon EC2. Get the latest version of it. (At the time of writing this , the latest version was here.)
Unzip this to a folder. No need to run any scripts for installing it.

In order to properly use the tools , you would need to set some Environment Variables in Cygwin. Here they are:

1. export EC2_HOME=/cygdrive/c/amazonec2/ec2-api-tools
2. export PATH=$PATH:/cygdrive/c/amazonec2/ec2-api-tools/bin
3. export EC2_PRIVATE_KEY=/cygdrive/c/amazonec2/keys/
4. export EC2_CERT=/cygdrive/c/amazonec2/keys/
5. export JAVA_HOME=/cygdrive/C/Program\ Files/Java/jdk1.6.0_18
6. export PATH=~/cygdrive/c/amazonec2/ec2-api-tools/bin:$PATH

(Notes: Notice the escape character in 5 .If you let Java install in its normal path in window, this escape is necessary)

Amazon EC2's Management Console is very good at most of the Management work you would need for playing with your instance but its good to have a command line option as well.

Using Cloud Computing in real life !

I have started this blog to record my experiences wrestling and having fun with the whole "Cloud computing" buzz.

A little bit of my background with Cloud Computing:

1. During my Master's study, I had taken a course "Web Scale Computing" which talked about using networked commodity hardware and a distributed framework on top of it. This cheap and reliable setup could scale up some really big data crunching tasks. We played with Amazon EC2 and Hadoop Framework.
2. After Graduation (yup...finally did it ), I decided to pursue the cloud computing foray and see how it can be used to apply to real life business problems/processes. I am working on a project right now which again involves huge data crunching tasks. This time my role is to help the development team scale the process up to the task by providing infrastructure support.

This blog aims to record my experience while I enjoy doing my work :-)