Compute Cluster Project

2009-04-09 @10:27pm

I've started working on a low priority project to build an experimental four node compute cluster to power a specialized computer chess application. One option would simply be to purchase a fast multicore processor machine - however, I wouldn't have four actual machines with individual IP addresses, unless of course I used virtualization. Another approach might have been to purchase Mini-ITX boards or PC104 boards. Truth be told - I already have most of what I need and I'm doing this just for kicks!

On the hardware side of the project I'm planning on utilizing four of the eight machines donated by my friend Kevin Lew. Kevin and I co-wrote a research paper with Colin Frayn for the 2006 IEEE Symposium on Computational Intelligence and Games. The machines I'm using for parts were a small part of that research effort.

I have several goals on the hardware side, such as removing the use of the 40GB drives, CD Rom drives and somehow eliminating (or at least reducing) the use of the four power supply units. A CD Drive won't be part of each node. It's only used during a one time install of Linux. Thus far I've eliminated the 40GB drive by replacing it with a 1GB Flash Module from Transcend.

The over arching goal is to stack the four motherboards into a custom enclosure.

Each compute node will feature an ATX board with onboard, NIC, Video, 1.8 GHz processor, 2GB RAM and 1GB IDE Flash Module running Damn Small Linux.

Having limited experience with hardware I certainly feel much more comfortable with the software side of the project. A few weeks ago I started seriously investigating the use of Damn Small Linux (DSL). DSL seemed ideal because it's only a 50MB distro and designed to run on live CDs and USB thumb drives. My first and obvious concern was to ensure that DSL fully supported the hardware I have available. Fortunately, DSL worked like a charm.

At this point in time I'm only getting started. But I'm happy with the results thus far. In later phases of the project I'll need to recompile my open source msgCourier product for use under DSL. msgCourier is a light-weight web/application server ideal for use with embedded Internet enabled solutions. I intend to use msgCourier as a monitor / controller for the chess engine that runs on each compute node. MsgCourier will essentialy allow me to view the status of each node and to otherwise control each node as an Internet appliance - in the same way that Cable/DSL routers are configured via a web browser.