This page describes the compute node setup. The configurations and many of the scripts for the nodes had to be custom written. Below is some of the customizations that were made to convert the machines from user boxes into HPC cluster nodes.


The following is the fstab of the compute nodes. This is a modified version of the default script that is run at startup to mount filesystems.  /                   nfs     nolock          0 0
none                    /dev/pts                devpts  gid=5,mode=620  0 0
none                    /proc                   proc    defaults        0 0
none                    /dev/shm                tmpfs   defaults        0 0       /usr/etc                nfs     nolock          0 0   /usr/include            nfs     nolock          0 0   /usr/libexec            nfs     nolock          0 0     /usr/local              nfs     nolock          0 0     /usr/share              nfs     nolock          0 0       /usr/src                nfs     nolock          0 0          /home                   nfs     nolock          0 0         /short                  nfs     nolock          0 0          /apps                   nfs     nolock          0 0
/dev/sda1               /mnt/localhdd           ntfs-3g rw              0 0

The first entry is the root filesystem. For each node, this line contains the path to its individual root filesystem on iserver2 ( The nodes root filesystems are named after their IP addresses on iserver2 so each node can connect to it’s respective filesystem.

The next 3 entries are system mounts that are required by the operating system.

All the mounts are from bc247. These mounts contain the programs and libraries that are used by users. Only one copy of these directories exist on the servers, this is so that updating and changing user programs is as easy as modifying one directory rather than the hundreds of individual node directories.

Finally, the last line mounts the first partition of the local hard drive of the node. This is a Windows NTFS Partition that is mounted using the ntfs-3g driver. Mounting this drive allows a local swapfile to be created and used (see below).

Swap Space

The following script file is located at /etc/rc.d/init.d/ntfsswap on the root filesystem of all compute nodes. A symbolic link is then made to the script at /etc/rc3.d/S98ntfsswap.

# Startup script for creating swap space on NTFS partition on compute nodes. Written 25 June 2012 by Stuart Buckingham
echo -n "Making room for the 12 secret herbs and spices: "
#Check that the hdd has first been mounted, fail if it hasn't
if [ -z "`cat /proc/mounts | grep /mnt/localhdd`" ]; then
        echo WARNING: /mnt/localhdd is not mounted, unable to create swap on it
        exit 1
#Create the swapfile if it does not already exist
if [ -x /mnt/localhdd/swapfile ]; then
        echo swapfile exists
        dd if=/dev/zero of=/mnt/localhdd/swapfile bs=1M count=30000
mkswap /mnt/localhdd/swapfile
swapon /mnt/localhdd/swapfile

Swap space acts like backup RAM. Because the speed of hard drives is much slower than the speed of on-board RAM, the swap space is only used once the RAM is full. For memory intensive programs, the amount of on-board RAM may be insufficient, and so pages may “spill over” into the swap space. Without the swap space, a kernel panic would occur when the RAM ran out. Although swapping to disk is much slower than RAM, it is a handy backup incase the RAM becomes full. All swapping is managed by the kernel. The kernel prioritizes pages and chooses what to swap. This becomes helpful when nearing the RAM capacity, as unused pages (e.g. those belonging to background processes) may be swapped out, thus freeing more memory for important processes.

Each time a compute node is booted into the Linux Operating System, the node checks for the file /mnt/localhdd/swapfile. The /mnt/localhdd/ directory is the Windows “C:” Drive of the local computer. If the swap file does not exist, it is created by taking 30000MB (30GB) of emptiness from /dev/zero and putting it in a file. This creates an empty 30GB file to be used as swaps space. If the swap file already exists when booted, the this file is used. The swap file is then converted to a swap filesystem using the mkswap command. The swap file can then be mounted as swap space using the swapon command, thus giving the system that extra buffer of memory if needed.

During this process, “Making room for the 12 secret herbs and spices: ” is displayed on screen. This is from a previous joke with a colleague.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: