Foreword

Singularity is a technology to run software containers (think Docker containers) in an High-Performance Computing environment. The most striking difference in comparison to Docker is that running these singularity containers do not need root rights.

Singularity is a software system that is in development. SURFsara aims to keep the version of singularity on our systems at the latest stable release. To keep our documentation up to date we decided to give some guidelines and do not provide in-depth generic information, but there are links to find this information. We guarantee no backward compatibility when singularity is upgraded.

Taken from the official website: http://singularity.lbl.gov

Singularity enables users to have full control of their environment. This means that a non-privileged user can “swap out” the operating system on the host for one they control. So if the host system is running RHEL6 but your application runs in Ubuntu, you can create an Ubuntu image, install your applications into that image, copy the image to another host, and run your application on that host in it’s native Ubuntu environment!

Requirements

• Access to Snellius, Grid or Lisa system
• Access to a Linux system with root rights

Build and install Singularity

In case you want to use Singularity on your Fedora (Linux) workstation, you can then install Singularity with the command sudo dnf install singularity. Ubuntu does not have Singularity included in the main software repositories. For manual builds, you can check the documentation. This can be done for example on your own machine, or on a virtual environment, either at your local computer, at SURF HPC Cloud, or at another Cloud resource provider. Administrative privileges are necessary to install the software. However, you can also use the container build nodes on Snellius.

Container build nodes (Snellius)

On Snellius we have a Singularity build partition called "cbuild". The nodes in this partition are configured to allow the building of containers without root privileges.

Get and run an image from Docker Hub

It is possible to convert a Docker container to the Singularity format, make sure you use the "cbuild" partition on Snellius for this. Otherwise you will run into errors related to missing privileges. E.g. to pull in an image from Docker Hub with singularity pull docker://godlovedc/lolcow and then run it with singularity run lolcow.sif

To run a container interactivly, pull in an image with e.g. singularity pull docker://fedora:latest And then enter the container with singularity shell fedora_latest.sif Or run a single command with e.g. singularity run fedora_latest.sif cat /etc/os-release

By default, the $HOME directory will be bound to your image. Changes made inside the image to the files that are bound also take effect outside the container. Deleting, creating and modifying files in your image's home directory removes also the files from the host itself. Changes inside the image made to the image itself (not to the directories that are bound) will not be saved and you will not have root rights inside the container. To make permanent adjustments continue reading in the post hoc adjustments section.

Build an image with a Singularity definition file

In case you need a customized image and want to build your own. You can then check the documentation to see the lolcow example as a Singularity definition file. In order to build the lolcow image without root privileges, use the nodes in the "cbuild" partition on Snellius.

When your Singularity definition file is ready to be build, you must use the --fakeroot switch, e.g. singularity build --fakeroot lolcow.sif lolcow.def. Once the build is finished, you can run the container with singularity run lolcow.sif.

Convert from local existing Docker images.

Docker is not installed or used on Snellius and Lisa. The following is possible when Docker and Singularity are installed on a host. Current local Docker images can be shown with sudo docker images where you can find the ID or IMAGE ID

$ sudo docker images
Password:
REPOSITORY                            TAG                 IMAGE ID            CREATED             SIZE
tensorflow/tensorflow                 latest              0bb45d441a4b        6 days ago          1.15 GB
singularityware/docker2singularity    latest              9a621f249838        3 weeks ago         101 MB
asciinema2gif                         latest              386c8b5977de        3 weeks ago         56.2 MB

To convert the image you need to set up a local Docker registry

sudo docker run -d -p 5000:5000 --name registry registry:2

tag the wanted images and push it to the registry

sudo docker image tag tensorflow/tensorflow  localhost:5000/mytensorflow
sudo docker push localhost:5000/mytensorflow

Now you can use singularity to pull the images from your private local registry. The registry has no encryption and we must tell singularity to ignore lack of https with the prefix "SINGULARITY_NOHTTPS=true"

sudo SINGULARITY_NOHTTPS=true singularity build mytenserflow.img docker://localhost:5000/mytensorflow

Stop the docker registry and clean up

sudo docker container stop registry && docker container rm -v registry

Post hoc image adjustments

When you need to install additional software or change some settings you can execute commands using shell commands inside the image. Keep in mind that you will need root permissions for these operations.

First step is to convert the compressed image into a uncompressed format in a folder structure (called a sandbox)

sudo singularity build --sandbox  sandbox ubuntu-latest.simg

To keep the image persistent use the --writeable option. e.g.

sudo singularity shell --writable sandbox/

Exiting the image can be done by exiting the shell with the exit command.

After editing you compress the sandbox to ensure portability and ease of use with

sudo singularity build myubuntu.simg sandbox/

Fine-tune for NIKHEF systems

To ensure you can reach data in scratch on NIKHEF systems while working on the grid you need to create a directory with the same name inside your image. This is also done with the following one-liner:

sudo singularity exec --writable example.img mkdir -p /tmpdir /cvmfs

Upload your image to our systems

After bootstrap has been completed on your system and tests have been done locally it is time to move the image to one of the SURFsara systems. The various systems have different best practices on where to put your image.

Snellius

From your local system, you can do a SCP to Snellius, with the following command the image will be placed in your home directory.

scp example.img username@snellius.surf.nl:~/

Grid

Or while using the Grid distribute it via Softdrive (cvmfs):

scp example.img softdrive.grid.sara.nl:/cvmfs/softdrive.nl/<username>/.

and publish with

ssh softdrive.grid.sara.nl publish-my-softdrive

LISA

From your local system you can do a scp to LISA with the following command the image will be placed in your home directory.

scp example.img username@lisa.surfsara.nl:~/.

Use your singularity image

Snellius

First create a directory on the scratch-shared part of the scratch file system ( NOTE: of course there are a lot of other ways to use your image, we just give one example ).

SCRATCH_SHARED_DIR=$(mktemp -d -p /scratch-shared)

Copy the image to the newly created scratch shared directory.

cp /place/where/you/store/image.img ${SCRATCH_SHARED_DIR}

Go into the newly created directory

cd ${SCRATCH_SHARED_DIR}

Start an interactive container sesion (you need to be on a compute node in order to access singularity).

singularity shell --pwd $PWD  ${SCRATCH_SHARED_DIR}/test.img

LISA

Jump into the scratch directory

cd $TMPDIR

copy your image to the scratch space

cp ~/example.img .
singularity exec --pwd $PWD example.img echo "hello world"

Grid

When you start a job you start by default in the scratch dir ($TMPDIR) and there is no need to switch to another directory. Images are automaticly cached by the cvmfs filesystem and there is no need to copy them to the scratch ($TMPDIR)

singularity exec --pwd $PWD /cvmfs/softdrive.nl/<username>/example.img echo "hello world"

FAQ

How do I set the $PATH variable?

The $PATH variable is taken from host environment. You can add a path to $PATH with

export PATH=/test/:$PATH
sudo singularity exec example.img echo $PATH

To make the PATH persistent in your image, add the export PATH line to "/environment" inside the container

My image is too large, can I make it smaller?

This is most likely caused by the software that was needed to build the software. Think of compilers, development headers and source code.

To make the image smaller you can best uninstall these packages and source code compress the package. Hereby an example which works on a ubuntu based container.

Detect and remove the largest packages

Before we can edit the image we need to convert it to an editable sandbox format

 sudo singularity build --sandbox  sandbox ubuntu-latest.simg 

Then bash inside your sandbox with sudo rights:

sudo singularity shell --writable  sandbox 

To detect the largest packages we run the following one-liner which prints package size and sort them by size (from small to big)

dpkg-query -W --showformat='${Installed-Size;10}\t${Package}\n' | sort -k1,1n

Then we select the big package, which are not needed at runtime. This depends on your software stack but in general, it is save to remove gcc, clang, and *-dev packages.

For instance:

sudo apt remove gcc *-dev

Also, remove the unneeded old packages and removing the cache of apt might help to clean the container.

sudo apt autoremove
sudo apt clean

Compress the image

After cleaning the container can converted back to a compressed format with

sudo singularity build myubuntu-small.simg sandbox/ 

Why is the --pwd $PWD option necessary?

By default, Singularity makes the current working directory in the container the same as on the host. For resolving the current working directory, Singularity looks up the physical absolute path (see man pwd for more info). However, many directories on our systems are symbolic links and the current working directory would resolve different than expected. This results that your files are not were you expected them to be (combined with some warning messages).

I do not like Docker, is there an alternative?

Yes, there is! You can write a Singularity bootstrap file (or convert Dockerfile to Singularity bootstrap). 

A singularity bootstrap file is a recipe to create a singularity image (or Singularities counterpart of a Dockerfile).

Information about writing a Singularity bootstrap file can be found at https://www.sylabs.io/guides/3.0/user-guide/definition_files.html.

You can use Singularity build  command to convert containers between the formats supported by Singularity. 

On Lisa, I get an LD_PRELOAD error

Please check https://servicedesk.surfsara.nl/wiki/display/WIKI/9.+LD_PRELOAD+error+when+running+a+container