Specs & Hardware Configuration
All machines, except for dam, are located in the Erlangen server room.
Cluster
Infrastructure servers
| Node | Enclosure | Sockets/Mainboard | Serial/SysGen Asset | Comment |
|---|---|---|---|---|
hn-1 | 2 HU 12LFF | 2 × SP3 H11DSU-iN | 0321.1174-01 | Qlustar head node (boot, LDAP, slurm), hosts login VM |
sn-1 | 4 HU 36LFF | 2 × 3647 X11DPH-T | 0321.1174-02 | |
fire | 2 HU 24SFF SC216BE1C-R920LPB | 2 × 2011-3 X10DRi-LN4+ | 1115.3920-03 | web/SaaS server |
dam | 4 HU 36LFF SC847BE1C-R1K28LPB | 2 × 2011-3 X10DRi-LN4+ | 1115.3920-02 | Located in the Nürnberg/AEG 1 server room |
Compute Nodes
| Queue (µArch) | Nodes | Enclosure Mainboard | CPUs @ base clock Core count @ TDP | RAM/Node (RAM/Core) L3 cache/core | Net Weight / Enclosure | Power Supply / Enclosure |
|---|---|---|---|---|---|---|
| haswell12 192 cores | sun-[01-04] sun-[05-08] | SYS-6028TP-HTFR X10DRT-PIBF | Intel Xeon E5-2680 v3 @ 2.50GHz 12 @ 120W | 32GB (1.3GB) 2.5MB | 32,7kg | 2 × 2000W |
| haswell 448 cores | sun-[09-12] sun-[13-16] sun-[17-20] sun-[21-24] | SYS-6028TP-HTFR X10DRT-PIBF | Intel Xeon E5-2695 v3 @ 2.30GHz 14 @ 135W | 64GB (2.3GB) 2.5MB | 32,7kg | 2 × 2000W |
| gold (Skylake) 288 cores | sun-[25-28] sun-[29-32] | SYS-6029TP-HTR X11DPT-PS | Intel Xeon Gold 6140 @ 2.30GHz 18 @ 140W | 96GB (2.7GB) 1.38MB | 32,7kg | 2 × 2200W |
| rome (previously “epic”) 3584 cores | sun-[33-36] sun-[37-40] sun-[41-44] sun-[45-48] sun-[49-52] sun-[53-56] sun-[57-60] | AS-2124BT-HNTR H12DST-B | AMD EPYC 7662 64-Core 2.0Ghz 64 @ 225W | 256GB (2.0GB) 4MB | 24,7kg | 2 × 2200W |
| gpu (Broadwell) 72 cores | sun-gpu-1 sun-gpu-2 | SYS-7048GR-TR X10DRG-Q | Intel Xeon CPU E5-2695 v4 @ 2.10GHz 18 @ 120W | 64GB (1.8GB) 2.5MB | 20,9kg | 4 × 2000W |
External & internal layout: Every 4 nodes share one single 2 HU “twin-square” enclosure with redundant power supply units (PSU). Each node contains two CPUs with SMT/Hyperthreading disabled.
GPUs: Both GPU nodes have each 4 × NVIDIA Titan V with a TDP of 225W and a recommended Power Supply of 650W.
Storage: All compute nodes are logically diskless (network boot, unionfs root filesystem fed with Qlustar images & chroot-over-NFS). The first CPU nodes sun-[01-08] have 1 × SAMSUNG MZ7WD240 SSD (240 GB) each (for /var and scratch space).
Other Components
Switches
- Infiniband: HDR 200Gbps (AMD rack)
- Infiniband: FDR 56Gbps Mellanox SX6025 (Intel rack)
- HPe 1820-48G Switch (Intel IPMI (?))
- Netgear 48-port switches (Intel cluster-LAN (?), AMD cluster-LAN (?) + IPMI (?))
Access to external public LAN (JuNet) via single ethernet cables (black, blue, yellow, green) that are plugged into the NET354 switch in the network rack.
See
Terminal
(
currently not operational, should display/control hn-1, sn-1, fire, and a spare cable to plug into a node on an ad-hoc basis)
Rack
water cooling via Rittal LCP