Linux Environment Modules at the D-MATH
With Environment Modules or simply called Modules it is possible to dynamically expand the active user environment and to remove it again without closing the shell.
At the D-MATH modules are used for:
- providing applications in different versions,
- customising the personal library environment,
- setting environment presets for institutes and groups.
Modules are grouped by their functionality in different folders. These module collections are referenced by their MODULEPATH.
Default collections
Every bash-shell starts with two collections.
main
The “main” collection provides applications, libraries and other additional collections:
- applications: matlab, mathematica, maple, sage, teTeX, etc.
- libraries: CUDA
- collections: Python, Distro, EasyBuild, userdata, etc.
environs
The “environs” collection involves institution dependent adjustments. The main thing is the mapping of the existing teTeX installation but also trying to reproduce the environment before modules:
- hg-*: default customisations for the D-MATH users
- sfs-*: customisations for the SfS users
- ifor-*: customisations for the IFOR users
- and some more
Additional collections
Additional collections are found in the “main” collections under modules.
Python2, Python3.6, Python3.7
These collections provide python modules for a specific version of python which do not exist in the standard installation or are in the wrong version.
The following command loads the Python3.6 or Python3.7 collection depends on the active python interpreter:
$ module load modules/Python3
We were very unhappy that everybody had to setup their own big “tensorflow” environment, which in the case of tensorflow-cpu/gpu happened twice. Now we can easily switch between different versions and settings.
(Note: “tensorflow” works only on computers with Intel-CPUs.)
For instance, to run the code in CPU-mode, execute
$ module load modules/Python3; module load tensorflow
and to run the same code in GPU-mode (on a supported computer), execute
$ module unload tensorflow; module load tensorflow-gpu
dist
Loading the “dist” collection will provided all the modules installed with the operating system, like for instance mpich, openmpi, carat, etc.:
$ module load modules/dist
EasyBuild
EasyBuild is a software build and installation framework that we use to provide in particular older versions of the GNU Compiler Collection (c++, gcc, gfortran, …):
$ module load modules/EasyBuild
userdata
This module gives you the possibility to create your own collection:
$ module load modules/userdata
The first time the directory /userdata/$USER/app/modules will be created for you and this MODULEPATH will be added to your environment. You have now the possibility to make your own collection. Read more on the Modulefile documentation.
Other collections
For a full list of collections check this:
$ module desc modules
The most important commands
List all available modules:
$ module avail
List only the default version of all modules in long format (alias for “module avail -l -d”):
$ module av
List only the default version of all modules with a short description (selfmade alias):
$ module desc
List loaded modules:
$ module list
List loaded modules in long format (alias for “module list -l”):
$ module li
Load modules into the shell environment:
$ module load [MODULE ...]
Unload modules from the shell environment:
$ module unload [MODULE ...]
Add module to the personal initialisation file to load it persistently:
$ module initadd [MODULE ...]
List modules loaded persistently from your personal initialisation file:
$ module initlist
Module has even more subcommands; see “man module” for a full reference:
$ module switch | show | purge | whatis | apropos | help | save...
Examples
List all available modules:
$ module av
- Package/Alias -----------------------.- Versions -.- Last mod. -------
/usr/local/share/dmath/modules/main:
CUDA/9.0-local default 2019/10/02 14:29:43
gcc/8.3/8.3.1-test-fc30 2019/10/02 14:36:54
julia/1.1.0 default 2019/10/02 14:21:22
maple/2017.2 2019/10/02 14:17:50
mathematica/11.3.0 default 2019/10/02 14:16:27
matlab/R2019a default 2019/10/02 14:17:23
modules/dist/1.0 2019/10/02 14:12:42
modules/EasyBuild/fc30 default 2019/10/18 14:13:39
modules/environs/1.0 2019/11/03 21:06:28
List all available versions of a module:
$ module avail -l matlab
- Package/Alias -----------------------.- Versions -.- Last mod. -------
/usr/local/share/dmath/modules/main:
matlab/R2016b 2019/09/26 23:53:33
matlab/R2017b 2019/09/26 23:53:33
matlab/R2018a 2019/09/26 23:53:33
matlab/R2018b 2019/09/26 23:53:33
matlab/R2019a default 2019/10/02 14:17:23
Load a module:
$ module load matlab
Load a module in a specific version that is not the default:
$ module load mathematica/11.2.0
List of all loaded modules:
$ module li
- Package -----------------------------.- Versions -.- Last mod. -------
Currently Loaded Modulefiles:
hg-texmf 2019/10/31 00:57:44
matlab/R2019a default 2019/10/02 14:17:23
mathematica/11.2.0 2019/09/26 23:53:33
Make “tensorflow” to be loaded persistently:
$ module initadd modules/Python3 tensorflow
References
- Full environment modules documentation:
https://modules.readthedocs.io/en/latest/ -
Environment modules instructions from the ETH Zurich scientific computing group:
https://scicomp.ethz.ch/wiki/Setting_up_your_environment
Final words
I am not completely happy with this page. But I hope that this information will help you. You can always ask us directly if you have further questions.
Please contact us:
- If you can’t use it. Maybe it is some bug of us.
- If you miss something. Maybe it is possible to integrate.
- And so on…
Thanks for helping us making IT greater.
