Hercules Parallel Computing Cluster

• Pedigree

  The cluster is modelled after the Avalon cluster at Los Alamos National Laboratory, which in turn is offspring of their Loki cluster. Loki is an early example of the Beowulf implementation of commodity computers in a MIMD configuration to solve super-computer-class problems.

• Architecture

  The cluster, hosted off-site in two 19" racks, consists of:
  • 15 headless nodes (soon to be 23):
    • Microway Screamer 533 MHz LX
    • Alpha CPU 21164A
    • 2MB cache
    • 64MB memory
    • 3GB IDE hard drive
    • Floppy (used for booting)
    • Kinston DEC DC21140-based fast-ethernet cards
  • 1 full node: same + CDROM, keyboard, video and SCSI disk
  • 3Com SuperStack II 3900, 24-port 100Mbps ethernet switch
  • Dumb terminal: used for rare out-of-band interaction with headless nodes
  
  

• Software

  • MPICH Message-Passing Interface
  • Fortran: Compaq, NDP, and egcs-1.1.1 g77 compilers
  • C: egcs-1.1.1 gcc
  • custom MPI port of Swift/Symba-based N-body dynamical integrator
  • custom MPI port of Smooth Particle Hydrodynamics (SPH) -->
  
  

• Operating System

  RedHat Linux, kernel versions 2.0.35, 2.1.125, and 2.2.5

• Challenges

  Overall, the implentation of this cluster was surprisingly easy thanks to the well-packaged/documented efforts of the creators of Avalon, MPICH, RedHat, and the larger Linux community. Our key challenge was very poor ethernet performance with the initial 2.0.x series Linux kernels. Later kernels vastly improved this performance and we are currently experimenting with further reducing latencies based on tips gleaned from the MPICH site. We have elected to boot the headless nodes off an SRM boot floppy and are interested in migrating to network boots for easier kernel maintenance. Of course, effective use of a parallel cluster of this sort requires appropriate software; we are currently porting several dynamics codes and have recently completed porting an implementation of SPH. That code approaches Cray T3E performance on a similar number of processors (see figure above), proving that the lack of shared memory architecture is not critical for workstation clusters of this scale.

• Personnel

  • Hal Levison, PI
  • Peter Tamblyn
  • Martin Duncan
  • Robin Canup
  • Erik Asphaug
  • Craig Agnor

• Related information

  Please check the sites listed in the Pedigree section. We have also made extensive use of Peter S. Pacheco's book, Parallel Programming with MPI