Unified Container Environments for Scientific Cluster Scenarios

DSpace Repositorium (Manakin basiert)


Dateien:

Zitierfähiger Link (URI): http://hdl.handle.net/10900/87666
http://nbn-resolving.de/urn:nbn:de:bsz:21-dspace-876668
http://dx.doi.org/10.15496/publikation-29052
Dokumentart: Konferenzpaper
Erscheinungsdatum: 2019-04
Sprache: Englisch
Fakultät: 7 Mathematisch-Naturwissenschaftliche Fakultät
Fachbereich: Informatik
DDC-Klassifikation: 004 - Informatik
Schlagworte: Hochleistungsrechnen
Freie Schlagwörter: bwHPC Symposium
containers
reproducibility
unified container environments
docker
Singularity
machine learning
HPC
computational workflow
Zur Langanzeige

Abstract:

Providing runtime dependencies for computational workflows in shared environments, like HPC clusters, requires appropriate management efforts from users and administrators. Users of a cluster define the software stack required for a workflow to execute successfully, while administrators maintain the mechanisms to offer libraries and applications in different versions and combinations for the users to have maximum flexibility. The Environment Modules system is the tool of choice on bwForCluster BinAC for this purpose. In this paper, we present a solution to execute a workflow which relies on a software stack not available via Environment Modules on BinAC. The paper describes the usage of a containerized, user-defined software stack for this particular problem using the Singularity and Docker container platforms. Additionally, we present a solution for the reproducible provisioning of identical software stacks across HPC and non-HPC environments. The approach uses a Docker image as the basis for a Singularity container. This allows users to define arbitrary software stacks giving them the ability to execute their workflows across different environments, from local workstations to HPC clusters. This approach provides identical versions of software and libraries across all environments.

Das Dokument erscheint in: