• View
• Edit
• History
• Print

CloudComputing-infra-sp11

Introduction

The goal of this project is to learn more about the infrastructure supporting cloud computing. You will implement map-reduce for a set of workstations in our department.

Details

You should implement map reduce to run on workstations, sort of like a limited version of Condor. You can assume that data to be processed is stored in AFS.

A user should submit a job that specifies the map task, reduce task, and input/output locations. Your service should execute this job across a set of machines (configured statically through a file in AFS).

Given that the machines available here are "volunteers", you should make your map-reduce implementation adapt to a heterogeneous machines that may not always be available.

In addition, you should work with our department file system: AFS may be a bottleneck, so you should investigate the best ways of communicating data between nodes in the system.

Applications

You should implement a few simple applications, even as easy as sort or 'grep to test your service.

Extensions

If you have a basic map-reduce function working, here are some extensions to consider:

• Idleness: only schedule jobs on idle workstations
• Locality: if a map and reduce task run on the same machine, it may be more efficient to store intermediate results locally in /tmp or /scratch, rather than through AFS
• Security: it is dangerous to accept a job from anybody and execute it. Consider how to limit the risk of running jobs, for example by executing map and reduce tasks in a virtual machine.

What to turn in

You should turn in a paper describing your implementation and evaluating your system.