This page describes the compute node setup. The configurations and many of the scripts for the nodes had to be custom written. Below is some of the customizations that were made to convert the machines from user boxes into HPC cluster nodes.
The following is the fstab of the compute nodes. This is a modified version of the default script that is run at startup to mount filesystems.
10.1.0.2:/nodes/10.60.18.6 / nfs nolock 0 0 none /dev/pts devpts gid=5,mode=620 0 0 none /proc proc defaults 0 0 none /dev/shm tmpfs defaults 0 0 10.1.0.1:/usr/etc /usr/etc nfs nolock 0 0 10.1.0.1:/usr/include /usr/include nfs nolock 0 0 10.1.0.1:/usr/libexec /usr/libexec nfs nolock 0 0 10.1.0.1:/usr/local /usr/local nfs nolock 0 0 10.1.0.1:/usr/share /usr/share nfs nolock 0 0 10.1.0.1:/usr/src /usr/src nfs nolock 0 0 10.1.0.1:/home /home nfs nolock 0 0 10.1.0.1:/short /short nfs nolock 0 0 10.1.0.1:/apps /apps nfs nolock 0 0 /dev/sda1 /mnt/localhdd ntfs-3g rw 0 0
The first entry is the root filesystem. For each node, this line contains the path to its individual root filesystem on iserver2 (10.1.0.2). The nodes root filesystems are named after their IP addresses on iserver2 so each node can connect to it’s respective filesystem.
The next 3 entries are system mounts that are required by the operating system.
All the 10.1.0.1 mounts are from bc247. These mounts contain the programs and libraries that are used by users. Only one copy of these directories exist on the servers, this is so that updating and changing user programs is as easy as modifying one directory rather than the hundreds of individual node directories.
Finally, the last line mounts the first partition of the local hard drive of the node. This is a Windows NTFS Partition that is mounted using the ntfs-3g driver. Mounting this drive allows a local swapfile to be created and used (see below).
The following script file is located at /etc/rc.d/init.d/ntfsswap on the root filesystem of all compute nodes. A symbolic link is then made to the script at /etc/rc3.d/S98ntfsswap.
#!/bin/bash # Startup script for creating swap space on NTFS partition on compute nodes. Written 25 June 2012 by Stuart Buckingham echo -n "Making room for the 12 secret herbs and spices: " #Check that the hdd has first been mounted, fail if it hasn't if [ -z "`cat /proc/mounts | grep /mnt/localhdd`" ]; then echo WARNING: /mnt/localhdd is not mounted, unable to create swap on it exit 1 fi #Create the swapfile if it does not already exist if [ -x /mnt/localhdd/swapfile ]; then echo swapfile exists else dd if=/dev/zero of=/mnt/localhdd/swapfile bs=1M count=30000 fi mkswap /mnt/localhdd/swapfile swapon /mnt/localhdd/swapfile
Swap space acts like backup RAM. Because the speed of hard drives is much slower than the speed of on-board RAM, the swap space is only used once the RAM is full. For memory intensive programs, the amount of on-board RAM may be insufficient, and so pages may “spill over” into the swap space. Without the swap space, a kernel panic would occur when the RAM ran out. Although swapping to disk is much slower than RAM, it is a handy backup incase the RAM becomes full. All swapping is managed by the kernel. The kernel prioritizes pages and chooses what to swap. This becomes helpful when nearing the RAM capacity, as unused pages (e.g. those belonging to background processes) may be swapped out, thus freeing more memory for important processes.
Each time a compute node is booted into the Linux Operating System, the node checks for the file /mnt/localhdd/swapfile. The /mnt/localhdd/ directory is the Windows “C:” Drive of the local computer. If the swap file does not exist, it is created by taking 30000MB (30GB) of emptiness from /dev/zero and putting it in a file. This creates an empty 30GB file to be used as swaps space. If the swap file already exists when booted, the this file is used. The swap file is then converted to a swap filesystem using the mkswap command. The swap file can then be mounted as swap space using the swapon command, thus giving the system that extra buffer of memory if needed.
During this process, “Making room for the 12 secret herbs and spices: ” is displayed on screen. This is from a previous joke with a colleague.