The 2023 hardware + OS update for wind (web server)

Machine: Supermicro SYS-2015CS-TNR (EPYC 9354P Genoa, 128 GB RAM, 2 system SSDs, 4 × 2 TB data SSDs, 2 × 10GbE & 2 × GbE NICs), SysGen Auftrags Nr. A1123.4535, eBanf 42342153

Due to limited rack space, I set up the machine next to the server rack, and ducted power + LAN cables through the roof of the Intel rack.

BMC: Update BMC LAN Configuration

• Configuration Address source: Static
• Station IP Address: 192.168.064.234 (MAC 3C-EC-EF-E3-E0-EE)

⇒ IPMI access works. I immediately change the BMC credentials to the cluster standard.

• 10GbE first (leftmost) jack: 7C-C2-55-9F-FF-D2 (Linux kernel: enp65s0f0)
• Network: Primary installation carried out in the private network of the cluster 192.168.62.0/24 with static IP 192.168.62.201, gateway 192.168.62.254 (hn-1) ⇒ network access works
• Partition layout: system SSD 1: 1 GB for ESP, system SSD 2: 1 GB for swap, rest as “physical volume for RAID”; unfortunately, it seems like the Debian partitioner can only either partition mdraid devices on bare disks, or use full mdraid device as a whole, if the mdraid bases on partitions.

After reboot, UEFI found the ESP, so no boot device reordering was necessary.

After GRUB, the OS needs about 2 minutes (displaying just mdadm info on tty1), but then goes online just fine.

We maintain the list of installed packages with debfoster(8) . See /var/lib/debfoster/keepers on this machine (and the other servers with manual OS setup).

In preparation of port 22 exposed to GitLab, we have to obfuscate the SSH port of the host OS, again to Port 2202 in /etc/ssh/sshd_config.

timedatectl set-timezone Europe/Berlin (it's an old Debian thing that the installer will only let you select US time zones after you have chosen US keyboard layout.)

Remember: the threat model for disk encryption is NOT against theft/malicious actions, but to ease disposal/RMA of broken SSDs.

1. Run /root/keys/prepare_disk.sh (convenience wrapper around blkdiscard and cryptsetup luksFormat) on the four data SSDs. blkdiscard takes surprisingly long (ca. 1 min per SSD).
2. Assemble zpool: zpool create aeolia raidz2 sdb.luks sdd.luks sde.luks sdf.luks (naming lore)
3. zfs volumes:
   • aeolia/config/{hosted,docker,secret} holds configuration files for bare-metal web server, docker, and SSL
   • aeolia/hosted/{service1,service2,…} One ZFS filesystem per web service (storage place), for both bare-metal (e. g., wiki) and dockerified (e. g., GitLab) servers
   • aeolia/docker to be arbitrarily polluted by the ZFS storage backend of Docker
   • aeolia/scratch with zfs set mountpoint=/scratch for temporary junk

Provisionally integrate regular NICs of wind-old, wind-new and fire into the IPMI network, i. e. stretch additional cables from the three machines to the IPMI switch, and configure them with manual IPs.

The last remaining cables were removed on 2024-02-28 (alongside the static IP in /etc/network/interfaces).

Storage backend (see also above):

• (Our) configurations are collected in /aeolia/config/docker/.
• User data of containers is stored in subvolumes of aeolia/hosted (included into docker-compose files via the standard volumes: /aeolia/hosted/service/volume:/internal/path/to/dir)
• Docker needs a machine-readable filesystem backend. It can do whatever it wants, within aeolia/docker. To do so, set the ZFS backend: zfs set mountpoint=/var/lib/docker aeolia/docker, and enable ZFS backend in /etc/docker/daemon.json.

As of 2024-01, containers for basilic and guacamole are hosted on fire, and this won't change any time soon.

Database. Runs bare-metal (Debian package mariadb-server). Used as the database backend of NextCloud and Basilic publication database.

Client applications are following in their respective sections.

We are using NextCloud in the Docker Apache flavour, i. e. a Docker container with the web server inlcuded, serving HTTP locally. Our bare-metal Apache only ProxyPasses the connections into the network, and adds SSL.

Following the upstream migration docs, including the steps about backup/restore cycles.

To create the database in the first place, open the MariaDB commandline, and

create database nextcloud CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;

(The charset bits are important to be able to use Unicode file names.)

Copy performance: Migration away file from HDDs to SSDs comes with a decent benefit. Be wary, though, that filling SQL databases is terribly slow (compared to reading), even on cutting-edge hardware.

In the future, update this table every once in a while, to keep some track of disk space usage by the cloud (especially if it gets more popular).

Still transitional on fire, resume™ update cascade at some point.

SSH forwarding (iptables)

Using Debian's Apache

wiki (as of 2024-02, testing LDAPS )

Apache modules and PHP plugins

apt-get install php-fpm                    # interconnect between web server and Docker backend
apt-get install php-curl,zip,gd,mbstring   # runtime dependencies
a2enmod proxy_http rewrite ssl             # Modules for general interaction with Apache/vHosts etc.
a2enmod proxy_fcgi setenvif                # Modules for NextCloud
a2enconf php8.2-fpm
systemctl reload apache2
sed -i 's/listen = .*$/listen = 127.0.0.1:9000/' /etc/php/8.2/fpm/pool.d/www.conf
systemctl restart php8.2-fpm

Apache: global config needs the <Directory /aeolia/hosted> section.

Apache sites: Place symlinks from /aeolia/config/hosted/sites-available/ to /etc/apache/sites-available/ and enable like locally configured

Wiring of dockerified web services via ProxyPass directives (local/port-based, or remote/DNS-based)

1. Get certificate via standard procedure
2. Place Certificates in /aeolia/config/secret/ssl/
3. In Apache VirtualHost files, use the certificate (*.cert) as SSLCertificateFile, and its passphrase *.key as SSLCertificateKeyFile
4. Remember to apply for a certificate at least several weeks before the old one runs out.

Prerequisite from FZJ to improve control over machines exposed to the internet. Tenable Nessus docs point to DEB package downloads (unfortunately, no proper repo, so I have to watch out for updates and install them manually).

(2024-02-26) I downloaded the DEB via web browser, scp'ed it to the server, installed and activated it:

apt install ./NessusAgent-10.5.1-debian10_amd64.deb
/opt/nessus_agent/sbin/nessuscli agent link --key=asdfasdfasdf --name=wind.iek.fz-juelich.de --groups="IEK-11" --host=nessus-manager.fz-juelich.de --port=8834

FZJ/JSC contact wrt Nessus: E. Gerzen, S. Linner

How much here, how much at the webserver site?