Abstract:
Providing runtime dependencies for computational workflows in shared environments,
like HPC clusters, requires appropriate management efforts from users
and administrators. Users of a cluster define the software stack required for a
workflow to execute successfully, while administrators maintain the mechanisms
to offer libraries and applications in different versions and combinations for the
users to have maximum flexibility. The Environment Modules system is the tool
of choice on bwForCluster BinAC for this purpose. In this paper, we present a
solution to execute a workflow which relies on a software stack not available via
Environment Modules on BinAC. The paper describes the usage of a containerized,
user-defined software stack for this particular problem using the Singularity
and Docker container platforms. Additionally, we present a solution for the reproducible
provisioning of identical software stacks across HPC and non-HPC
environments. The approach uses a Docker image as the basis for a Singularity
container. This allows users to define arbitrary software stacks giving them the
ability to execute their workflows across different environments, from local workstations
to HPC clusters. This approach provides identical versions of software
and libraries across all environments.