Module: Cloud Computing

Project: Choose one of the projects below

You need to implement an application using Amazon Web Services.

Project 1: Amazon Web Services

In this part, you will launch and offer access to a simple batch processing service through
Amazon using EC2, SQS, and S3. As an extra challenge (not required), you can create a
public facing web page for your application.

There are many ways of doing this. We have introduced a variety of interfacing
possibilities during the lecture, including the management console, command-line tools
(EC2 command line was demonstrated), the REST and SOAP APIs, etc. There are libraries
in many languages (boto in Python is quite active), or SDKs, including for Java (there is an
AWS plugin for Eclipse users). You can use any tool you want (including those not
mentioned here) to interact with AWS and prototype the application. You are expected
to be able to demonstrate and explain the coding and functioning of your system.

You will have to set up an AWS account. This is a free account and can be done at
(http://aws.amazon.com/free/)1.
1. Use the AWS management console to do the initial set up of your environment
deployment.
2. Launch a micro instance (using the classic wizard, choose the basic Linux AMI –
the star next to it indicates it is free).
3. Create a key pair and configure a security group (a firewall defining permissions
and accesses).
4. Allow ssh, http and https traffic.
5. Now do the necessary key exchange set up, and ssh into your instance.
Next:
6. Set up Eclipse (or whatever tool, library or SDK you are using), on the client side,
i.e., your machine. Note that you need the EE version of Eclipse (dependencies
restrictions!). Have a look at video tutorials on http://aws.amazon.com to do that.
Note that you need to generate and provide credentials to access the services.
Few sample applications are provided, have a look at them. The boto website also
includes a set of examples and use cases if you are using this library.
7. Write and run simple examples, which perform basic services calls (refer to the
documentation).

Batch processing on Amazon:

1 This needs a credit card number, but it will not be charged if you only lunch micro
instances under the ‘free usage tier’ (http://aws.amazon.com/free/).
Your application will consist of two main components that monitor the messages and
jobs as they move through the computational and storage processes. These are the client
application (run locally on the client side) and the worker process (run on the instance).

The system includes three SQS queues (input, output, and error). The client application
initiates the batch process. This, primarily, consists of input files required by the job. For
simplicity, you can specify a simple worker code (a simple ‘grep’ for instance) to avoid
any pre-required technology stack on the worker. You can still allow the user to specify
the code he or she wants to execute (this is a more advanced task). Or alternatively, any
worker code can also be attached to the instance at the set up time.

The client then constructs the ‘work unit’, which details the work to be performed, input
and output (the ‘grep’ command line for instance). Then, the client uploads the input files
to S3, and creates a message, which encompasses the work unit and the details of the
input file location. Finally, it inserts this new message into the SQS input queue. The client
should be able to repeat this process for new jobs.

The worker process is responsible of managing the jobs flow, executing the worker code,
and uploading output files to S3. It first retrieves a message from the input queue and
processes it, i.e. downloads the required files from S3 and executes the job specified in
the work unit. The output files and any log files are then uploaded to the appropriate S3
bucket. If the job is completed successfully, a result message is placed in the SQS output
queue. If the job resulted in an error, a message is placed in the SQS error queue.

The general architecture is as follows:

SQS output queue

Client Worker
SQS input queue

SQS error queue

S3

8. (As a more advanced task), create a web interface for your application. It would
be a simple front-end page.
9. Another more advance task would be to add an elasticity process that monitors
the input queue, and launch an additional worker instance according to different
scaling metrics (e.g., the number of messages).
10. You can add any additional functionality. You will get bonus points for this or any
original idea.
Finally:
11. Write a short report describing the design and implementation of the system
applications, as well as the uses cases. Include a discussion at the end commenting on
your experience with this project, what you have learnt, and any issues or open
questions.
Submitted material
 You will create an archive of all your source code, documentation, and the short
report, before submission.
 We will arrange for 10/15 minutes slots for everybody to demo their work.
 If you have any questions please post them on the forum.
 Note that we will not provide technical assistance, i.e. configuration support or coding,
debugging, etc.