As part of all of this research into Amazon cloud computing I've started to program against the API's to see how they work.  Eventually I'd like to build a client that does a lot more than what can be done with ElasticFox, which is a great tool, but falls short when you try to do things that require multiple steps.  For example, one scenario that I'm already repeating all the time is the following:

1) Launch AMI
2) Mount 2 volumes to it (one step for each)

This isn't very difficult to do, but imaging if I'm building out managed services on behalf of other clients of mine, and the # of clients gets large, such that this will actually start to take a lot of time in the aggregate.  I'd like to have a tool that can do this automatically, and it is possible to program this to be done through the web services.

Another scenario that I'm thinking will be useful is to be able to do automated backups of AMI's and / or volumes, and perhaps also roll back AMI's and / or volumes on a regular schedule.  Again, not too hard with ElasticFox, but again a manual process that doesn't scale well.

So, I've started programming a WPF client using one of the C# api's available on the Amazon site, the C# Library for EC2.  It's been pretty easy to use so far, and I wanted something C# based instead of pure SOAP / REST.  There are deficiencies in the library that I will address later, but I wanted to show what I've done so far even though it's really just proving to myself that this works.

The following is a control that I put together to manipulate S3 buckets.  It's a listbox that is databound to the results from the 'listAllMyBuckets' method on the AWSAuthConnection class in the C# library:

Indeed, these are the two buckets that I have right now.

For reference, I include the XAML and the code behind for the control:

While doing all of this blogging on cloud computing lately, it came to my realization that I should probably put together a template/outline for a complete set of posts to cover everything that I'd like to discuss, and to perhaps serve as an outline for a book or at least a series of white papers.  What I've come up with so far is below, and I'll modify this list as I go along refining the topics:

Working title: Building Cloud Applications with .NET, Azure, and Amazon Services

I.                    Introduction

1.       Overview

a.       What is cloud computing?

b.      What problems does cloud computing solve?

c.       Business Benefits of Cloud Computing

d.      Components of a cloud application

                                       i.      Table based storage

                                     ii.      File based storage

                                    iii.      Service bus

                                   iv.      Workflow

                                     v.      Web front ends

                                   vi.      Dynamic processing allocation

                                  vii.      Queue based messaging

                                viii.      Web and worker roles

                                   ix.      Mesh Services

                                     x.      Data services

                                   xi.      Geographic diversity

                                  xii.      Elastic addresses

e.      Where are we going in this book?

2.       Cloud Computing Architectural Patterns

a.       Overview

b.      Example Architectures

                                 i.      Processing Pipelines

                               ii.      Batch Processing Systems

                              iii.      Websites

c.       Characteristics of Cloud Systems

                                 i.      Loosely Coupled

                               ii.      Self scalable

                              iii.      Highly Parallel

                             iv.      Must be reboot and relaunch resilient

                               v.      Distributed storage and computing supporting failover

II.                  Cloud Services: Azure and Amazon

3.       Windows Azure Services

a.       Overview

b.      Comparison to reference architecture

c.       Components of Azure

                                 i.      Internet Service Bus (.NET Services)

                               ii.      Azure web applications

                              iii.      Mesh

                             iv.      SQL Data Services

4.       Amazon Services

a.       General overview

b.      S3

                                 i.      Buckets,

                               ii.      …

c.       EC2

                                 i.      AMI’s

                               ii.      Kernels

                              iii.      Security Groups

d.      SQS

e.      SimpleDB

5.       Comparison of Azure and Amazon Cloud Services

a.       Similarities and Differences

b.      Using the two together

III.                Programming the Cloud with .NET

6.       Web front ends

a.       ASP.Net applications for Azure

b.      ASP.NET on Amazon

7.       The Internet Service Bus: Connecting Desktops and Servers to the Cloud and to each other

a.       Overview of ISB

b.      Connecting to the Cloud to request Services

c.       Integrating with Partners through the ISB

8.       Storing and Retrieving Data in the Cloud

a.       Blob storage: Azure blobs and S3 storage

b.      Table data storage: Azure tables and Amazon SimpleDB

9.       Databases

a.       SQL Data Services

b.      Connecting to your SQL Server

c.       Amazon SimpleDB SQL

10.   Queuing: using Azure and Amazon Queues

11.   Workflow: Orchestrating the Cloud

a.       MS Workflow

b.      Self implemented workflow

12.   Mesh Services: Replicating Data

13.

IV.                Deploying, Operating and Optimizing Cloud Applications

13.   Application Packaging:

a.       Azure packages

b.      Amazon AMI’s

14.   Deployment to the Cloud:

a.       Provisioning Azure applications

b.      Deploying AMI’s at Amazon

15.   Geodiversity

a.       Using Azure Geodiversity

b.      Amazon Availability Zones

c.     Amazon Cloud Front

16.   Data partitioning

a.       Why Partition the Data?

b.      Partitioning Azure Table Data

c.       Similar for Amazon

17.   Monitoring

a.       Logging

b.      Self monitoring in the cloud

c.       Remote cloud monitoring

d.      Talking back to OSS systems

V.                  Closing

18.   Best Practices for Cloud Computing

a.       Storage

b.      Queueing

c.       Table data manipulation

d.      Start multiple instances at once

e.      Plan for failure

19.   Cloud Computing Futures

a.       Automatic Demand Based Allocation

b.      Better Monitoring

c.       Windows Live Services

d.      Advanced Amazon Services

                                 i.      Mechanical Turk

                               ii.      Payment Services

                              iii.      Search

In my previous post I discussed how to use ElasticFox to start a windows instance in EC2.  In that post in order to keep it simple I avoided discussing security groups.  Security groups in EC2 are basically just instructions to EC2 on how to open firewall ports to allow communications to your system.

In ElasticFox, open the Security Groups tab and you will see something similar to the follow, although this is showing a security group called '42Spikes' that I've already added and run this site in:

My last post mentioned that I was trying to move my blog over to Amazon EC2.  I have that completed (if you are reading this, it came out of the EC2 cloud), and as promised I am now writing up how to do it.

First, this was not without challenge, of which the primary one is just how to operate EC2.  Specifically, here are the challenges I came across and which I will explain (although not all in this single post):

• Creating a key pair,
• Finding a windows server AMI to run,
• Getting an elastic IP,
  
• Setting up security zones,
• Starting the ami,
• Getting the server password,
  
• Getting remote access to the server,
• Configuring the server,
• Oh, terminating and rebooting starts over from scratch with the image and therefore blows away all of your config,
• How to bundle your OS into your own AMI, hence saving your configuration, but
• That only saves the C drive, which on an Amazon machine is fairly small,
• How to create your own volumes and attach them to your system,
  
• Using the volumes to store your application data (hence using it as persistent storage in S3), and
• Restarting your AMI and attaching the volumes

This is a fairly non-trivial list and documentation is pretty sparse on how to do all of this, and hence this blog entry.

When I got started, the best resource I found was Tim Heuer's blog which showed how to get started with EC2.  I therefore give him credit here and you can read that post here.

The first thing to do is get an already existing tool to work with EC2.  The one that Tim recommended and that I am using at this point is ElasticFox, an extension to Firefox.  The reason for needing a tool like this is that Amazon does not provide a client for managing EC2.  They only provide a web service API, which is actually quite great as you can programmatically do anything with EC2 from outside (or inside) the Amazon cloud.  But it does leave you wondering how to get going quickly, and if you are a Firefox user like I am just go and get ElasticFox.

note: ElasticFox is a great tool and you can do most things with it, but it does fall short for automating a sequence of tasks, which I can see will be quite useful.  For example, say you want to start one of your AMI's, attach volumes, assign the elastic IP?  You need to do these all manually.  Hence, I'm working on a WPF application to manage EC2, which will also allow you to do workflow.  I'll be posting that to codeplex soon and I'll let you all know when I do this.

One you have ElasticFox installed, open firefox and select Tools -> ElasticFox.  Since this is the first time, you will need to enter your Amazon EC2 information.  To do this, press the 'Credentials' button:


At which point you will see a form like this (albeit empty if it is your first time):


Enter your account name, access key and secret access key, and press 'Add'.  This information will be used by elasticfox to identify you to them and to access your resources.

When that is completed, ElasticFox will present to you similar to the following:

What you see here is a list of AMI's (virtual machines in the amazon library), which is also filtered to just show only those with windows in the title.  The naming convention of this is somewhat convoluted, but you will get used to it.  Notice that I have one selected, and that is the AMI I'm using to run my blog.  It is a windows 2003 R2 instance with IIS and SQL Server 2005 express installed.  In the 'Your Instances' list, you can see the virtual systems that you have created and their status.  Here you see the one that I am currently running the blog upon.

Now before we can do anything (like starting an instance), you must now create a keypair.  This keypair is used in the web service call that launches the instances and the private key is used by ssh to authenticate.  The keypair can easily be created in elastic fox, and it will pass it to the service calls for you.  The easiest way to create one is to use the 'KeyPairs' tab create a keypair specifying a name and then save the returned private key in a safe place on your file system.

So, open the keypair tab and you will see something similar to the following, although if this is your first time the list will be empty:


Now press on the green key button, and you will be presented with the following form:


I've entered a name in the field, but name yours anything you want.  When you press 'OK', elastic fox passes the name to amazon, and it returns a '.pem' key file to elastic fox, and elasticfox will ask you to save the file somewhere.  Please do this, as this keypair / file will be used later (specifically to start an instance)

Fundamentally, you can just right click on any AMI and select 'Launch Instance(s) of this AMI' and you will see a new instance start, go through initialization, and eventually reach the 'Running" state.  The running state is when the system is, well, running, and you can RDP in to the box (if your security zone is configured to allow RDP - I'll explain this later).

To get the system into the running state, it will take several minutes of initialization time.  I don't know the specifics, but what I can guess from experience is going on is the following:

• Allocation of the AMI to a physical server,
• Sys prepping the AMI so it has a unique system name and IP configurations,
• Changes made to the Amazon firewalls to allow access into the system
• Database entries created to allow manipulation of the virtual system via the web services

To be honest, I skipped a few things here, specifically the security zone configuration and the launch instance dialog.  Security zone info I'll cover in another post, and I'll briefly explain the launch instance dialog now.  When you select 'Launch Instance(s) of this AMI' you will be presented with this dialog asking you some information that EC2 needs to know to start the instance:

IMHO, at this point, the most important thing to select is the Security Groups to run the instance in.  I'll cover them in another post, but in short they are instructions to EC2 on how to open (or keep closed) tunnels through the firewall.  The default security group will basically shut down all access, even RDP.  But, if you try to connect to the system from ElasticFox, it will see you don't have port 3389 open and will ask you if it is ok if it is ok for it to add 3389 to the defautl zone.  So, leaving default here will allow you to RDP to the system (after ElasticFox confirms with you opening 3389), but will block everything else.  A good test of this is to try (since this instance by default installs IIS) to connect to the public DNS name of the instance with HTTP.  This will be blocked.  You can open port 80 later by creating a new security group.

Another thing of importance here is the specification of the keypair.  You can see here that I've selected one of my existing keypairs.  Again, I'm not sure of the details, but some investigation leads me to believe that firefox passes the .pem file associated to the key back to EC2 when you start the instance.  Note that I believe the .pem file is sent is because if you either don't save the file, OR your delete/move the file, you can not start an instance.

There is other information asked here, like the instance type, min and max # of instances, your key pair, availability zone, (I'll comment on these other options in another post) ...  but to keep it simple at this point you can at this point just press launch and you will get a machine that is getting spun up (I'll explain all these in other posts):

I've moved my blog over to a windows virtual system hosted on Amazon EC2.  Part of the process was also to update to the latest DasBlog, which I did.  It was an interesting experience and I'll write it up over the holidays. 

I'm still wondering whether or not I should move the whole blog over to Wordpress.  I'm going to work on that too.  Overall I'm real happy with EC2, except for the price of a windows system which I figure will be over $100 per month.  That's quite good for a business, but too much for a personal blog.

I do like having overall control of the blog though.  I think I noticed somewhere that the windows instance in EC2 was actually a linux machine virtualizing windows 2003.  I might be good if I can just virtualize an ubuntu instance, and then run windows 2003 inside of it (from my own licenses).  The linux boxes are much less expensive than the windows ones.

Just came across this today: VirtualBox.  I'm going to give it a try.  I'm a heavy vmware and parallels user, but a nice open source solution would also be really nice.  I'll let everyone know what I think.  Installing now...

Tags: virtual, cloud