Welcome to Cartesius! This guide is intended to provide you with the absolute basics required to get you connected to Cartesius and get your first jobs running.
Once you've grokked the basics, we strongly encourage you to read the user guide. This in-depth guide will teach you everything you need to know to make optimal and efficient use of Cartesius and its resources.
When the account obtained for Cartesius is activated, personal logins will be created as needed to do work on Cartesius under this particular account. Users that receive a new personal login associated with the account are notified by e-mail about the login name and will receive an initial, temporary, password. With these credentials new users must login to the portal, at https://portal.surfsara.nl, to:
When you have accepted the usage agreement and have set a valid password for using the system, in principle you are all set to start your first interactive session. Note however that for Cartesius interactive nodes it matters from where the connection attempt is made.
The login nodes accept (GSI-)SSH connections only from known, socalled "white-listed", IP address ranges. If your connection attempt is from a Dutch academic site or research institute, chances are that you can simply login without any further ado, because many of the IP address ranges used by these sites are already white-listed. But if you try to login from abroad, or from home, you might find that the connection is denied even before you can enter your login credentials. In that case you can still login via the SURFsara doornode. For IP ranges that you frequently use to make the connection from, you also might want to file a request at helpdesk@surfsara.nl to add them to the whitelist. N.B.: See the Cartesius FAQ, under "Help, I can't login!", on how to file such a request, what information it must contain, and how to obtain that information).
Access to Cartesius is provided through a command line interface only, using Secure Shell (SSH), or in some cases, a certificate-based version of Secure Shell (GSI-SSH). To connect to the system you will need to use a terminal. On macOS and Linux-based operating systems, these are installed by default; on Windows you will need to install one yourself (e.g. PuTTY or mobaXterm).
On Linux/macOS, login by issuing the following command (replacing <username> with the username provided to you):
ssh <username>@cartesius.surfsara.nl
On windows, find a tab named 'Session' or similar, and enter cartesius.surfsara.nl in the 'host' field. Enter your Cartesius username in the 'user' or 'login' field, and if asked for a port number enter '22'.
Hit 'enter' or click 'connect' to open your connection. If all goes well, you should be presented with a command line interface. On your first connection, you will be presented with Cartesius's RSA fingerprint: please verify it matches the fingerprint on the page Cartesius Hostkey Fingerprints, and type 'yes' to continue.
Congratulations! You just connected successfully to Cartesius. More extensive login options and troubleshooting info can be found here and on the documentation page on interactive usage.
Since Cartesius is navigated through the command line, it is imperative that you familiarise yourself with some essential CLI commands.
If you are not familiar with the Linux command line, we suggest you look into a UNIX tutorial. The UNIX Tutorial for Beginners is a good starting point; if you want to get proficient with bash, our login shell, try the Advanced Bash-Scripting Guide.
Remember that if you want to interrupt a running command, you can always use Ctrl+c to force-terminate it.
There are two ways to transfer files between Cartesius and your PC: via the scp command in the terminal, or using an FTP file browser.
If you open a terminal on your local machine (i.e. without logging in to Cartesius), you can use scp to copy files. The syntax is
scp [source] [destination]
For example, to copy my_file from the current directory on your local machine to the directory destination_dir on Cartesius:
scp my_file <username>@cartesius.surfsara.nl:destination_dir
Alternatively, you can install and use an SFTP browser such as Filezilla (available for Windows, MacOS and Linux), Cyberduck (Winodws, MacOS) or WinSCP (Windows). MobaXterm (Windows) also has an integrated SFTP browser. To connect, you need to set the hostname (cartesius.surfsara.nl), connection protocol (SCP, SSH2, SFTP or similar) and port number (22) in your SFTP client.
There are various file systems available on Cartesius, each with their specific use cases. The following file systems are available:
File system | Quota | Speed | Expiration | Backup |
---|---|---|---|---|
Home | 200 GiB | Normal | None | Daily incremental |
Archive | N.A. | Slow | None | Daily incremental backup of metadata, continuous migration of data to tape |
Scratch | 8 TiB | fast (parallel I/O) | At most 14 days | No |
Projects | Varies per project | fast (parallel I/O) | Project duration | No |
After logging in to Cartesius you'll find yourself in your home directory, on the home filesystem. This is the directory where you can store your job scripts, source code, data sets (temporarily) etc. You can always access your home directory through the $HOME
environment variable (e.g. ls $HOME
or cd $HOME
).
Scratch is a high performance, parallel file system designed to meet the I/O demands of massively parallel jobs. The storage quota is much higher than Home, but there is no backup - you are responsible for making sure your data is safe, by copying it to either your home directory, the SURFsara archive, or off-site. Keep in mind that any file on scratch older than 14 days will be automatically removed.
The scratch file system is presented in two ways: /scratch-local/
behaves like a local disk to each node (but is still on a shared file system!); /scratch-shared/
is presented as a shared folder between all nodes.
The project file system can be used
It can be accessed at /projects/0
/<project_name>
. You can check how much free space you have using df -h /projects/0/project_name>
. To share files on the project file system, you need to make sure to write files with the correct file permissions. Take a look at the in-depth page on file systems if you don't know how to do that yet.
The archive file system is intended for long term storage of large amounts of data. Most of this data is (eventually) stored on tape, and therefore accessing it may take a while. The archive is accessible at /archive/<username>
. The archive system is designed to only handle large files efficiently. If you want to archive many smaller files, please compress them first in a single tar file, before copying it to the archive. Never store a large amount of small files on the archive: they may be scattered across different tapes and it will put a large load on the archive to retreive all those files if you need them at a later stage. See __this section__ of the User Guide for more information on using the archive appropriately.
It is always a good idea to keep track of your usage of the file systems, and stay under quota. To check your quota, use the quota
command.
More in-depth information on our file systems can be found here.
The usual way of running your jobs on Cartesius is to submit a job script (or batch script) to the job scheduler. The scheduling system used on Cartesius is called SLURM (official website), and to submit jobs we use a utility called sbatch
. A basic batch script might look like this:
#!/bin/bash #SBATCH -J my_first_job #SBATCH -n 480 #SBATCH -t 01:00:00 srun -n 480 ./my_mpi_code
Since batch scripts are essentially shell scripts, the first line is required to invoke the right shell (usually bash
). The following lines, starting with #SBATCH
, are parameters to the scheduler. In this example, in order, we give the name my_first_job
to this task (for easy identification); we request 480 cores (SLURM automatically calculates the required number of nodes); and we set the maximum wall clock time to 1 hour - if the job runs longer, it will get terminated. Finally, we use srun
to call our MPI-aware code with 480 MPI tasks. Again, SLURM ensures these tasks are distributed over the allocated nodes.
This is only a very basic example of a job submission script, and many more options and refinements are possible and, most likely, desirable. A very detailed guide to submitting batch jobs, as well as running commands interactively, can be found on this page.
Since Cartesius is a busy system, your job will most likely not run immediately after submitting you job script. Instead, it will enter the job queue where it will remain until it is scheduled to run by the SLURM scheduler. When your job will run is dependent on how many resources it requires, and the execution time requirement specified in the job script.
To check the current state of the queue, the squeue command
can be used:
squeue [-u $USER]
Running just squeue
will report the state of the entire job queue, while running it with the -u $USER
parameter will display only the status of your own jobs.
If you want to remove one of your jobs from the queue, use scancel
:
scancel -j <jobid>
where <jobid>
is the job ID reported by sbatch
when submitting the job. If you have lost this ID, it is also shown in the squeue
output.
More advanced queue operations can be found on the SLURM website.
In an interactive session, the terminal shows two types of output streams: the standard output and standard error streams. Regular output (e.g. the result of a calculation) that a program wants to show in the terminal is generally written to the standard output stream, while error output (e.g. your program reports it is missing an argument) is commonly written to the standard error stream.
By default, SLURM willl redirect all output to a file called <jobid>.out
, which will be placed in the directory from where you submitted your job. This file contains both the standard output and error streams, combined. If you want to send these two streams to different files, you can achieve this by setting certain sbatch parameters; this is described in the official sbatch documentation.
Running on Cartesius is charged in System Billing Units (SBUs), and charging is based on the wall clock time of a job. On fat and thin nodes, an SBU is equal to using 1 core for 1 hour (a core hour), or 1 core for 20 minutes on a GPU node. Since compute nodes are allocated exclusively to a single job at a time, you will be charged for all cores on that node - even if you are using less. There are several utilities available on Cartesius to help you keep track of your usage and remaining budget; accuse
, accinfo
and budget-overview
. Detailed instructions on how to use these tools can be found at Accounting on Cartesius.
ls
mkdir [mydir]
cd [mydir]
rm [file]
who
date
top
logout
The SURFsara Data Archive allows the user to safely archive up to petabytes of valuable research data.
Persistent identifiers (PIDs) ensure the findability of your data. SURFsara offers a PID provisioning service in cooperation with the European Persistent Identifier Consortium (EPIC).
B2SAFE is a robust, secure and accessible data management service. It allows common repositories to reliably implement data management policies, even in multiple administrative domains.
The grid is a transnational distributed infrastructure of compute clusters and storage systems. SURFsara is active as partner in various...
Spider is a dynamic, flexible, and customizable platform locally hosted at SURF. Optimized for collaboration, it is supported by an ecosystem of tools to allow for data-intensive projects that you can start up quickly and easily.
The Data Ingest Service is a service provided by SURFsara for users that want to upload a large amount of data to SURFsara and who not have the sufficient amount...
The Collaboratorium is a visualization and presentation space for science and industry. The facility is of great use for researchers that are faced with...
Data visualization can play an important role in research, specifically in data analysis to complement other analysis methods, such as statistical analysis.