Managing the Software Environment on LI-red and Seawulf

As with most High Performance Computing systems, Seawulf and Li-red employ Environment Modules for managing development environments and providing software to users. Typically users initialize their environment when they log in by setting environment information for every application they will reference during the session. The Environment Modules package is a tool that simplifies shell initialization and lets users easily modify their environment during the session with modulefiles.

Audience: Faculty, Researchers and Staff

This KB Article References: High Performance Computing
This Information is Intended for: Faculty, Researchers, Staff
Last Updated: October 11, 2017

Modules

Each modulefile contains the information needed to configure the shell for an application. For example, there are many different flavors of MPI available, each with compiler wrappers: mpicc, mpicxx, mpif77 and mpif90 and launchers: mpirun, mpiexec, and mpiexec.hydra, which may have differences depending on the flavor of mpi. The commands specific to each flavor can be accessed by loading the respective module, i.e.:

Intel MPI:

module load intel/mpi/64/2017/0.098

or MVAPICH2:

module load mvapich2/gcc/64/2.2rc1

Using Modules

After logging on to Seawulf or Li-red, users setup their environment by loading modules as needed into the current shell. While many users will code this information directly into their .bashrc or .bash_profile, we tend to discourage users from doing this in order to fully utilize a modular environment. 

The module command is the primary tool used to setup your environment. To get an overview, execute:

module help

The most frequently used commands include:

module avail:  returns a list of the available modules

module load (or add):  adds module to the current environment

module list:  returns a list of all the currently loaded modules

module unload (or remove):  removes a module from the current environment

module purge: removes all loaded modules

 

IMPORTANT: There are two directories where modules are stored, shared and local. Local modules are local to the login node while shared modules are used throughout the entire system. The local module is loaded by default by the scheduler but shared is not. Before loading any modules, also load the shared module:

module load shared

Recommended (Intel) Software Stack

While the software stack in your environment will change depending on your needs, we recommend a simple Intel based software stack when possible:

 

SeaWulf LIred
module load shared 
module load torque/6.0.2 
module load intel/compiler/64/2017/17.0.0 
module load intel/mkl/64/2017/0.098 
module load intel/mpi/64/2017/0.098
module load shared 
module load torque/5.1.0 
module load intel/compiler/64/16.0.2/2016.2.181 
module load intel/mkl/64/11.3.2/2016.2.181 
module load intel/mpi/64/5.1.3/2016.2.181

 

This is composed of shared and torque/6.0.2, for accessing shared modules and submitting jobs to our queue systemThe Intel Developer Toolchain (Intel Parallel Studio 2017 - Cluster Edition) through intel/compiler/64/2017/17.0.0 and intel/mkl/64/2017/0.098, and Intel's implementaion of MPI through intel/mpi/64/2017/0.098.


Alternative (GCC) Software Stack

If the open source GNU Toolchain is preferred, we recommend using the software stack below:

SeaWulf LIred
module load shared
module load torque/6.0.2
module load gcc/6.1.0
module load mvapich2/gcc/64/2.2rc1
module load shared 
module load torque/5.1.0 
module load gcc/5.1.0 
module load mvapich2/gcc/64/2.1

 

This includes shared and torque/6.0.2, as in the previous example, gcc/6.1.0 compilers, and MPI implemented through mvapich2/gcc/64/2.2rc1. One notable difference between this and our recommended software stack is the omission of Intel's optimized math libraries, MKL. Generally using MKL will provide your application with some performance benefit, and we recommend its use when possible. However, without a full understanding of your applications intricacies, this can lead to many issues. As an alternative Netlib reference BLAS, BLACS, LAPACK, and scaLAPACK are provided as modules. 

Additional Information


There are no additional resources available for this article.

Getting Help


The Division of Information Technology provides support on all of our services. If you require assistance please submit a support ticket through the IT Service Management system.

Submit A Ticket

Supported By


IACS Support System