{"id":597,"date":"2020-01-24T11:08:09","date_gmt":"2020-01-24T10:08:09","guid":{"rendered":"https:\/\/wpethzprd.ethz.ch\/isgdmath\/?p=597"},"modified":"2025-05-09T08:12:22","modified_gmt":"2025-05-09T06:12:22","slug":"modules","status":"publish","type":"post","link":"https:\/\/blogs.ethz.ch\/isgdmath\/modules\/","title":{"rendered":"Linux Environment Modules at the D-MATH"},"content":{"rendered":"\n<p>With <strong>Environment Modules<\/strong> or simply called <strong>Modules<\/strong> it is possible to dynamically expand the active user environment and to remove it again without closing the shell.  <\/p>\n\n\n\n<!--more-->\n\n\n\n<p>At the D-MATH modules are used for: <\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>providing applications in different versions,<\/li>\n\n\n\n<li>customising the personal library environment,<\/li>\n\n\n\n<li>setting environment presets for institutes and groups.<\/li>\n<\/ul>\n\n\n\n<p>Modules are grouped by their functionality in different folders. These module collections are referenced by their MODULEPATH.  <\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Default collections<\/h2>\n\n\n\n<p>Every bash-shell starts with two collections.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">main<\/h3>\n\n\n\n<p>The &#8220;main&#8221; collection provides applications, libraries and other additional collections:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>applications:<\/strong>\u00a0matlab, mathematica, maple, sage, teTeX, etc.<\/li>\n\n\n\n<li><strong>libraries:<\/strong>\u00a0CUDA<\/li>\n\n\n\n<li><strong>collections:<\/strong>\u00a0Python, Distro, EasyBuild, userdata, etc.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">environs<\/h3>\n\n\n\n<p>The &#8220;environs&#8221; collection involves institution dependent adjustments. The main thing is the mapping of the existing teTeX installation but also trying to reproduce the environment before modules:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>hg-*<\/strong>: default customisations for the D-MATH users<\/li>\n\n\n\n<li><strong>sfs-*<\/strong>: customisations for the SfS users<\/li>\n\n\n\n<li><strong>ifor-*<\/strong>: customisations for the IFOR users<\/li>\n\n\n\n<li>and some more<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Additional collections<\/h2>\n\n\n\n<p>Additional collections are found in the &#8220;main&#8221; collections under modules.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Python2, Python3.6, Python3.7<\/h3>\n\n\n\n<p>These collections provide python modules for a specific version of python which do not exist in the standard installation or are in the wrong version.<\/p>\n\n\n\n<p>The following command loads the Python3.6 or Python3.7 collection depends on the active python interpreter:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ module load modules\/Python3<\/pre>\n\n\n\n<p>We were very unhappy that everybody had to setup their own big &#8220;tensorflow&#8221; environment, which in the case of tensorflow-cpu\/gpu happened twice. Now we can easily switch between different versions and settings.<\/p>\n\n\n\n<p>(Note: &#8220;tensorflow&#8221; works only on computers with Intel-CPUs.)<\/p>\n\n\n\n<p>For instance, to run the code in CPU-mode, execute<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ module load modules\/Python3; module load tensorflow<\/pre>\n\n\n\n<p>and to run the same code in GPU-mode (on a supported computer), execute<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ module unload tensorflow; module load tensorflow-gpu<\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">dist<\/h3>\n\n\n\n<p>Loading the &#8220;dist&#8221; collection will provided all the modules installed with the operating system, like for instance mpich, openmpi, carat, etc.:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ module load modules\/dist<\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">EasyBuild<\/h3>\n\n\n\n<p>EasyBuild is a software build and installation framework that we use to provide in particular older versions of the GNU Compiler Collection (c++, gcc, gfortran, &#8230;):<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ module load modules\/EasyBuild<br>$ module avail<br>$ module load GCC\/12.3.0<br>$ gcc --version<br>==&gt;<br>gcc (GCC) 12.3.0<\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>userdata<\/strong><\/h3>\n\n\n\n<p>This module gives you the possibility to create your own collection:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ module load modules\/userdata<\/pre>\n\n\n\n<p>The first time the directory \/userdata\/$USER\/app\/modules will be created for you and this MODULEPATH will be added to your environment. You have now the possibility to make your own collection. Read more on the <a href=\"https:\/\/modules.readthedocs.io\/en\/latest\/modulefile.html#\">Modulefile documentation<\/a>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Other collections<\/h3>\n\n\n\n<p>For a full list of collections check this:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ module desc modules<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">The most important commands<\/h2>\n\n\n\n<p>List all available modules:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ module avail<\/pre>\n\n\n\n<p>List only the default version of all modules in long format <em>(alias for &#8220;module avail -l -d&#8221;):<\/em><\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ module av<\/pre>\n\n\n\n<p>List only the default version of all modules with a short description <em>(selfmade alias):<\/em><\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ module desc<\/pre>\n\n\n\n<p>List loaded modules:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ module list<\/pre>\n\n\n\n<p>List loaded modules in long format <em>(alias for &#8220;module list -l&#8221;):<\/em><\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ module li<\/pre>\n\n\n\n<p>Load modules into the shell environment:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ module load [MODULE ...]<\/pre>\n\n\n\n<p>Unload modules from the shell environment:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ module unload [MODULE ...]<\/pre>\n\n\n\n<p>Add module to the personal initialisation file to load it persistently:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ module initadd [MODULE ...]<\/pre>\n\n\n\n<p>List modules loaded persistently from your personal initialisation file:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ module initlist<\/pre>\n\n\n\n<p>Module has even more subcommands; see &#8220;man module&#8221; for a full reference:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ module switch | show | purge | whatis | apropos | help | save...<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Examples<\/h2>\n\n\n\n<p>List all available modules:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ module av<br>- Package\/Alias -----------------------.- Versions -.- Last mod. -------<br>\/usr\/local\/share\/dmath\/modules\/main:<br>CUDA\/9.0-local                          default      2019\/10\/02 14:29:43<br>gcc\/8.3\/8.3.1-test-fc30                              2019\/10\/02 14:36:54<br>julia\/1.1.0                             default      2019\/10\/02 14:21:22<br>maple\/2017.2                                         2019\/10\/02 14:17:50<br>mathematica\/11.3.0                      default      2019\/10\/02 14:16:27<br>matlab\/R2019a                           default      2019\/10\/02 14:17:23<br>modules\/dist\/1.0                                     2019\/10\/02 14:12:42<br>modules\/EasyBuild\/fc30                  default      2019\/10\/18 14:13:39<br>modules\/environs\/1.0                                 2019\/11\/03 21:06:28<\/pre>\n\n\n\n<p>List all available versions of a module:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ module avail -l matlab<br>- Package\/Alias -----------------------.- Versions -.- Last mod. -------<br>\/usr\/local\/share\/dmath\/modules\/main:<br>matlab\/R2016b                                        2019\/09\/26 23:53:33<br>matlab\/R2017b                                        2019\/09\/26 23:53:33<br>matlab\/R2018a                                        2019\/09\/26 23:53:33<br>matlab\/R2018b                                        2019\/09\/26 23:53:33<br>matlab\/R2019a                           default      2019\/10\/02 14:17:23<\/pre>\n\n\n\n<p>Load a module:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ module load matlab<\/pre>\n\n\n\n<p>Load a module in a specific version that is not the default:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ module load mathematica\/11.2.0<\/pre>\n\n\n\n<p>List of all loaded modules:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ module li<br>- Package -----------------------------.- Versions -.- Last mod. -------<br>Currently Loaded Modulefiles:<br>hg-texmf                                             2019\/10\/31 00:57:44<br>matlab\/R2019a                           default      2019\/10\/02 14:17:23<br>mathematica\/11.2.0                                   2019\/09\/26 23:53:33<\/pre>\n\n\n\n<p>Make &#8220;tensorflow&#8221; to be loaded persistently:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ module initadd modules\/Python3 tensorflow<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">References<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Full environment modules documentation:<br><a href=\"https:\/\/modules.readthedocs.io\/en\/latest\/\">https:\/\/modules.readthedocs.io\/en\/latest\/<\/a><\/li>\n\n\n\n<li><br><p>Environment modules instructions from the ETH Zurich scientific computing group:<br><a href=\"https:\/\/scicomp.ethz.ch\/wiki\/Setting_up_your_environment\">https:\/\/scicomp.ethz.ch\/wiki\/Setting_up_your_environment<\/a><\/p><br><\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Final words<\/h2>\n\n\n\n<p>I am not completely happy with this page. But I hope that this information will help you. You can always ask us directly if you have further questions.<\/p>\n\n\n\n<p>Please contact us:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>If you can&#8217;t use it. Maybe it is some bug of us.<\/li>\n\n\n\n<li>If you miss something. Maybe it is possible to integrate.<\/li>\n\n\n\n<li>And so on&#8230;<\/li>\n<\/ul>\n\n\n\n<p>Thanks for helping us making IT greater.<\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>With Environment Modules or simply called Modules it is possible to dynamically expand the active user environment and to remove it again without closing the shell.<\/p>\n","protected":false},"author":43853,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[198216,759],"tags":[],"class_list":["post-597","post","type-post","status-publish","format-standard","hentry","category-computing","category-linux"],"_links":{"self":[{"href":"https:\/\/blogs.ethz.ch\/isgdmath\/wp-json\/wp\/v2\/posts\/597","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.ethz.ch\/isgdmath\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.ethz.ch\/isgdmath\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.ethz.ch\/isgdmath\/wp-json\/wp\/v2\/users\/43853"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.ethz.ch\/isgdmath\/wp-json\/wp\/v2\/comments?post=597"}],"version-history":[{"count":1,"href":"https:\/\/blogs.ethz.ch\/isgdmath\/wp-json\/wp\/v2\/posts\/597\/revisions"}],"predecessor-version":[{"id":1695,"href":"https:\/\/blogs.ethz.ch\/isgdmath\/wp-json\/wp\/v2\/posts\/597\/revisions\/1695"}],"wp:attachment":[{"href":"https:\/\/blogs.ethz.ch\/isgdmath\/wp-json\/wp\/v2\/media?parent=597"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.ethz.ch\/isgdmath\/wp-json\/wp\/v2\/categories?post=597"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.ethz.ch\/isgdmath\/wp-json\/wp\/v2\/tags?post=597"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}