The compute nodes are all equipped with two Ethernet cards. The on-board card is connected to the Monash network, and is the only active networking device on the Windows image that is run during the daytime. This network connection gives students access to university resources as well as the World Wide Web. Each node also has a GigEthernet PCI card connected to the “private” network. On a normal Windows boot, this device is disabled, and is prevented from activation by Windows user permissions. During Linux boot, this device is enabled and becomes the method of communication between head-nodes and sister-nodes.
The architecture of the “private” network is as follows:
The network represents a classical tree topology. This is not the ideal layout for an HPC cluster (more efficient topologies such as meshes, toroids and hyercubes are described here), however it is quite apt for most of the jobs that are frequently run on this cluster.
The two servers at the top of the diagram are iserver2 (left) and bc247 (right). These two servers are connected to the outside world (the cloud) through one interface, and the bc247 network through another interface. iserver2 has the added feature of using iptables masquerade table to perform Port Address Translation. This allows the nodes to communicate to the outside world (e.g. for requesting software licenses) while also filtering outside requests to the nodes (e.g. security threats).
Each computer lab consist of 24 computer connected to one GigE switch. In this way, the computers in one room communicate with each other in a simple star network. This topology lends itself to fast transfers between exclusive nodes simultaneously. That is, computer A and B could be communicating while computer C and D could be communicating through the switch at the same time without delaying A and B. This is in contrast to a bus topology using a layer 1 device (a hub) and having to deal with delays due to packet collisions or token passing.
Proposed updates for the network being considered include creating inter-switch connections between the second level switches, therefore creating a star topology amongst the switches. Also link bundling between he servers and switches would allow a higher throughput along these important connections, thereby allowing faster boots and file I/Os. Another option could be to remove the top level switch, and install direct links from the servers to each of the four room switches, thereby decreasing the number of hops from server to node. In order to decrease the latency caused by simultaneous traffic from different nodes, link bunching could be used from the server two each switch, thus allowing non-blocking access by multiple nodes. Unfortunately the resources required to modify the network topology are not easily available, and the cost of installing new cabling may be prohibitive.