Welcome to Cartesius! This guide is intended to provide you with the absolute basics required to get you connected to Cartesius and get your first jobs running.

Once you've grokked the basics, we strongly encourage you to read the user guide. This in-depth guide will teach you everything you need to know to make optimal and efficient use of Cartesius and its resources.

Connecting to Cartesius

Back to top

Access to Cartesius is provided through a command line interface only, using Secure Shell (SSH). To connect to the system you will need to use a terminal. On macOS and Linux-based operating systems, these are installed by default; on Windows you will need to install one yourself (e.g. PuTTY or mobaXterm).

On Linux/macOS, login by issuing the following command (replacing <username> with the username provided to you):

ssh <username>@cartesius.surfsara.nl

On windows, find a tab named 'Session' or similar, and enter cartesius.surfsara.nl in the 'host' field. Enter your Cartesius username in the 'user' or 'login' field, and if asked for a port number enter '22'.

Hit 'enter' or click 'connect' to open your connection. If all goes well, you should be presented with a command line interface. On your first connection, you will be presented with Cartesius's RSA fingerprint: please verify it matches the fingerprint on the page Cartesius Hostkey Fingerprints, and type 'yes' to continue.

Congratulations! You just connected successfully to Cartesius. More extensive login options and troubleshooting info can be found here and on the documentation page on interactive usage.

Getting around on the CLI

Back to top

Since Cartesius is navigated through the command line, it is imperative that you familiarise yourself with some essential CLI commands. For reference, some of the basic commands are:

• ls
• mkdir [mydir]
• cd [mydir]
• rm [file]
• who
• date
• top
• logout

If you are not familiar with the Linux command line, we suggest you look into a UNIX tutorial. The UNIX Tutorial for Beginners is a good starting point; if you want to get proficient with bash, our login shell, try the Advanced Bash-Scripting Guide.

Remember that if you want to interrupt a running command, you can always use Ctrl+c to force-terminate it.

Transfering your files

Back to top

There are two ways to transfer files between Cartesius and your PC: via the scp command in the terminal, or using an FTP file browser.

If you open a terminal on your local machine (i.e. without logging in to Cartesius), you can use scp to copy files. The syntax is

scp [source] [destination]

For example, to copy my_file from the current directory on your local machine to the directory destination_dir on Cartesius:

scp my_file <username>@cartesius.surfsara.nl:destination_dir

Alternatively, you can install and use an SFTP browser such as Filezilla (available for Windows, MacOS and Linux), Cyberduck (Winodws, MacOS) or WinSCP (Windows). MobaXterm (Windows) also has an integrated SFTP browser. To connect, you need to set the hostname (cartesius.surfsara.nl), connection protocol (SCP, SSH2, SFTP or similar) and port number (22) in your SFTP client.

File systems

Back to top

There are various file systems available on Cartesius, each with their specific use cases. The following file systems are available:

After logging in to Cartesius you'll find yourself in your home directory, on the home filesystem. This is the directory where you can store your job scripts, source code, data sets (temporarily) etc. You can always access your home directory through the $HOME environment variable (e.g. ls $HOME or cd $HOME).

Scratch is a high performance, parallel file system designed to meet the I/O demands of massively parallel jobs. The storage quota is much higher than Home, but there is no backup - you are responsible for making sure your data is safe, by copying it to either your home directory, the SURFsara archive, or off-site. Keep in mind that any file on scratch older than 14 days will be automatically removed.

The scratch file system is presented in two ways: /scratch-local/ behaves like a local disk to each node (but is still on a shared file system!); /scratch-shared/ is presented as a shared folder between all nodes.

The project file system can be used

1. If you need additional storage space, but do not require a backup.
2. If you need to share files within a collaboration.

It can be accessed at /projects/0/<project_name>. You can check how much free space you have using df -h /projects/0/project_name>. To share files on the project file system, you need to make sure to write files with the correct file permissions. Take a look at the in-depth page on file systems if you don't know how to do that yet.

The archive file system is intended for long term storage of large amounts of data. Most of this data is (eventually) stored on tape, and therefore accessing it may take a while. The archive is accessible at /archive/<username>. The archive system is designed to only handle large files efficiently. If you want to archive many smaller files, please compress them first in a single tar file, before copying it to the archive. Never store a large amount of small files on the archive: they may be scattered across different tapes and it will put a large load on the archive to retreive all those files if you need them at a later stage. See __this section__ of the User Guide for more information on using the archive appropriately.

It is always a good idea to keep track of your usage of the file systems, and stay under quota. To check your quota, use the quota command.

More in-depth information on our file systems can be found here.

Submitting a job script

Back to top

The usual way of running your jobs on Cartesius is to submit a job script (or batch script) to the job scheduler. The scheduling system used on Cartesius is called SLURM (official website), and to submit jobs we use a utility called sbatch. A basic batch script might look like this:

#!/bin/bash
#SBATCH -J my_first_job
#SBATCH -n 480
#SBATCH -t 01:00:00

srun -n 480 ./my_mpi_code

Since batch scripts are essentially shell scripts, the first line is required to invoke the right shell (usually bash). The following lines, starting with #SBATCH, are parameters to the scheduler. In this example, in order, we give the name my_first_job to this task (for easy identification); we request 480 cores (SLURM automatically calculates the required number of nodes); and we set the maximum wall clock time to 1 hour - if the job runs longer, it will get terminated. Finally, we use srun to call our MPI-aware code with 480 MPI tasks. Again, SLURM ensures these tasks are distributed over the allocated nodes.

This is only a very basic example of a job submission script, and many more options and refinements are possible and, most likely, desirable. A very detailed guide to submitting batch jobs, as well as running commands interactively, can be found on this page.

The job queue

Back to top

Since Cartesius is a busy system, your job will most likely not run immediately after submitting you job script. Instead, it will enter the job queue where it will remain until it is scheduled to run by the SLURM scheduler. When your job will run is dependent on how many resources it requires, and the execution time requirement specified in the job script.

To check the current state of the queue, the squeue command can be used:

squeue [-u $USER]

Running just squeue will report the state of the entire job queue, while running it with the -u $USER parameter will display only the status of your own jobs.

If you want to remove one of your jobs from the queue, use scancel:

scancel -j <jobid>

where <jobid> is the job ID reported by sbatch when submitting the job. If you have lost this ID, it is also shown in the squeue output.

More advanced queue operations can be found on the SLURM website.

Job output

Back to top

In an interactive session, the terminal shows two types of output streams: the standard output and standard error streams. Regular output (e.g. the result of a calculation) that a program wants to show in the terminal is generally written to the standard output stream, while error output (e.g. your program reports it is missing an argument) is commonly written to the standard error stream.

By default, SLURM willl redirect all output to a file called <jobid>.out, which will be placed in the directory from where you submitted your job. This file contains both the standard output and error streams, combined. If you want to send these two streams to different files, you can achieve this by setting certain sbatch parameters; this is described in the official sbatch documentation.

Budget (core hours)

Back to top

Running on Cartesius is charged in System Billing Units (SBUs), and charging is based on the wall clock time of a job. On fat and thin nodes, an SBU is equal to using 1 core for 1 hour (a core hour), or 1 core for 20 minutes on a GPU node. Since compute nodes are allocated exclusively to a single job at a time, you will be charged for all cores on that node - even if you are using less. There are several utilities available on Cartesius to help you keep track of your usage and remaining budget; accuse, accinfo and budget-overview. Detailed instructions on how to use these tools can be found at Accounting on Cartesius.