Coinciding with the Containers at TACC training taking place today via webinar, here is a method I have used successfully to build our version of SWAN v41.10 spectral wave model in order to run with MPI on the University of Texas’ Stampede2 cluster.
The following Dockerfile will build the parallel mpi implementation of SWAN, the single-core serial SWAN model, and the SWAN hcat concatenation utility. The containerized model should be able to be used for 2D non-stationary, 1D or stationary simulations.
# TACC base image with MPI compatible with the Stampede2 network fabric
FROM tacc/tacc-ubuntu18-mvapich2.3-psm2
RUN apt-get update
RUN apt-get install -qq -y build-essential gcc gfortran
RUN apt-get install -qq -y libcurl4-gnutls-dev
RUN apt-get install -qq -y libcr-dev
RUN apt-get install -qq -y bash at perl unzip sed
# Build HDF5
WORKDIR /build/build
ADD hdf5-1.8.13.tar.gz /build/build
WORKDIR /build/build/hdf5-1.8.13
RUN ./configure --prefix=/usr/local --enable-shared --enable-hl --enable-fortran --enable-cxx
RUN make
RUN make install
RUN ldconfig
# Build NetCDF4
WORKDIR /build/build
ADD netcdf-4.3.2.tar.gz /build/build
WORKDIR /build/build/netcdf-4.3.2
ENV LDFLAGS -L/usr/local/lib
ENV CPPFLAGS -I/usr/local/include
RUN ./configure --enable-netcdf-4 --enable-dap --enable-shared --prefix=/usr/local --disable-doxygen
RUN make
RUN make install
RUN ldconfig
WORKDIR /build/build
ADD netcdf-fortran-4.4.0.tar.gz /build/build
WORKDIR /build/build/netcdf-fortran-4.4.0
RUN ./configure --enable-netcdf-4 --enable-dap --enable-shared --enable-f90 --prefix=/usr/localadd_client_regions()
RUN ln -s /build/swan4110 /build/adcirc_swan/swan
RUN ls /build/adcirc_swan
RUN ls /build/adcirc_swan/swan
# Build
WORKDIR /build/adcirc_swan/work
RUN mkdir -p /build/adcirc_swan/swan/bin
WORKDIR /build/adcirc_swan/swan
RUN make config
RUN sed -i -e "s/NETCDFROOT =/NETCDFROOT=\/usr\/local/g" macros.inc
RUN sed -i -e "s/-lnetcdf /-lnetcdf -lhdf5 /g" macros.inc
RUN sed -i -e "s/-lnetcdff /-lnetcdff -lhdf5_fortran/g" macros.inc
RUN cat macros.inc
RUN make mpi
RUN echo "##########"
RUN ls *exe
RUN echo "#########"
RUN mv swan.exe /build/adcirc_swan/swan/bin/swan.mpi
RUN make clean
RUN make clobber
RUN cat macros.inc
RUN make ser
RUN mv swan.exe /build/adcirc_swan/swan/bin/swan.ser
RUN make hcat
RUN ls *exe
RUN mv hcat.exe /build/adcirc_swan/swan/bin/hcat
ENV PATH $PATH:/build/adcirc_swan/swan/bin
To convert the local Docker image to a Singularity/Apptainer sif file, we do the following using the appropriate image for the Singularity version we are targeting. The output sif can be copied to the HPC cluster for use in the batch queue scripts.
docker run --rm -ti -e SINGULARITY_NOHTTPS=1 \
-v /your/folder:/your/folder quay.io/singularity/singularity:v3.7.2 \
build swan-41_10-mpi.sif \
docker://youruser/swan:41.10-mpi-tacc
We have a set of bash scripts that run numerical models in a way that fits into our workflow and includes verifying inputs and doing automated QC and validation of the outputs of simulations. I had to add the following module commands to correctly run the Singularity/Apptainer image built from the above Dockerfile using the tacc/tacc-ubuntu18-mvapich2.3-psm2 base image.
module load mvapich2
module load tacc-singularity
module unload xalt
...
ibrun singularity run swan-41_10-mpi.sif swan.mpi
Of issues I have previously had with Lonestar5’s proprietary networking fabric, the subsequent Lonestar6 HPC cluster supports a non-proprietary networking fabric with the recent upgrade: There is some confirmation from TACC that tacc/tacc-ubuntu18-mvapich2.3-ib works, so a single MPI image starting from that base image could potentially be used on Frontera, Maverick2, and Lonestar6.