Python on sun

We (Qlustar 12 human users, LB3D pipelines) used an installation of Anaconda 2021.05, presumably installed by Manuel for the Erlangen cluster relaunch in August 2021, with the use of EasyBuild. It has quickly become unmaintainable (by the proverbial sluggishness of conda, and on top of this, people unwittingly installing stuff via pip in system mode).

Installation path: /apps/local/software/Anaconda3/2021.05/, enabled via module load Anaconda3

All as normal user:

1. Download Anaconda3-2023.03-1-Linux-x86_64.sh (861 MB) from their website
2. Install into /apps/local/software/Anaconda3/2023.03/, skip conda init
3. Adapt /apps/local/modules/all/Anaconda3/2021.05.lua to /apps/local/modules/all/Anaconda4/2023.03.lua (bump version, strip EasyBuild internals)
4. Add conda-forge to the default package list: conda config --append channels conda-forge Contrary to mamba documentation, the conda-forge channel is already enabled by default.
5. Install mamba: module load Anaconda4 ; conda install mamba -n base -c conda-forge (on sun-60: takes 4m32.552s, most of it for calculating dependencies)
6. Updates! mamba update --all
7. Install custom packages: gmsh vtk (important to be installed first), then ford f90nml jupyterhub{,-base} jupyter-server-proxy cython TODO jupyterlab-slurm???
8. … More to follow upon user request. First one: wordcloud
9. Another round of updates: mamba update --all
10. Compare conda list output between installations to spot manually installed packages: vimdiff <(cat $previous_conda_list | awk '{ print $1 }' | sort) <(conda list | awk '{ print $1 }' | sort)
11. Remove packages that are unnecessary and/or interfere with other stuff in the toolchain: mpi mpich
12. Write an e-mail to everyone that they should test Anaconda4 and report if things are missing or misbehaving.
13. Remove write privileges to everyday users (make it harder for myself to mess up the installation). I have chown -R softadm /apps/local/software/Anaconda4’d the installation, but thanks to a 002-ish umask, users that are members of the softadm group (like JH7r) are still not write-protected.

Minimum-invasive forward-port of Jupyter Hub from MZ (including his semi-successful Slurm queue efforts). Config files copied over, $PATH adapted.

Get the Jupyter Lab running:

mamba install batchspawner "nodejs=12*" configurable-http-proxy %%--%%channel conda-forge
jupyter lab build           # Pointer from <https://stackoverflow.com/a/60606367>. Takes a few minutes to finish.

Then stop the jupyterhub.service on login, re-wire the systemd service file, start the new server, and test it (import sys; print(sys.executable)).

Finally, replace the Anaconda3 module by a symlink to the Anaconda4 module.

• 2023-10-16: trimesh (for stl2rocks.py: mamba install trimesh --channel conda-forge (also updates openssl and ca-certificates)
• 2023-11-06: lcov (try to move from Spack to Anaconda, in an attempt to avoid Gitlab CI to rely on Spack): mamba install lcov --channel conda-forge (also updates openssl and perl)
• 2023-11-10: qiskit (MSc k.haas): mamba install qiskit --channel conda-forge (also updates mpc and mpfr)

1. Download Anaconda3-2023.09-0-Linux-x86_64.sh (1.1 GB)
2. Install into /apps/local/software/Anaconda3/2023.09/ (don't create the directory beforehand!), skip conda init
3. Set up /apps/local/modules/all/Anaconda4/2023.09.lua; set up a symlink ln -s 2023.03.lua default to avoid accidental loading while in internal beta. Load the module.
4. Snapshot the ZFS filesystem zfs snapshot home/apps@anaconda2023.09vanilla
5. Updates (conda level): time conda update –all, takes 2m1.668s
6. conda config --show channels shows us that conda-forge is already in there, so we don't need to enable it.
7. time conda install mamba, takes 1m34.330s, but leaves mamba in an unusable state (argument COMMAND: conflicting subparser: repoquery; this seems to be known upstream, and a fix is on the way)

The current situation is broken. We cannot use mamba on fresh installations, so we are left with using conda on either the old or the new Anaconda4. (Needless to say that this comes with the expected drawbacks like absurd performance, killing conda install numba after 23 minutes of miserable attempts to resolve dependencies.)

For the bold future:

1. mamba install gmsh vtk
2. mamba install ford f90nml jupyterhub{,-base,-server-proxy} cython wordcloud trimesh lcov qiskit --channel conda-forge