Cartesius is a bullx system extended with one Bull sequana cell. It is a clustered SMP (Symmetric Multiprocessing) system built by Atos/Bull.
Cartesius consists of a large number of batch nodes and a small number of special purpose nodes. For the batch nodes we differentiate between so-called thin nodes, fat nodes, GPU nodes and Xeon Phi nodes. Thin nodes constitute the majority of the available batch nodes. GPU nodes use GPGPUs to accelerate the computations. The Xeon Phi nodes have Intel Many-Core CPUs. Fat nodes have more memory (256 GB) than the thin nodes and more physical cores (32) than most thin nodes.
Special nodes are the two interactive front end nodes and five so-called service nodes. The interactive front end nodes provide the ssh login service and are meant for interactive work. The service nodes are meant for data staging (transfers to and/or from external contexts, such as the SURFsara archive facility or a remote site.).
Currently, Cartesius consists of the following compute nodes:
Node Type | Number | Cores | CPU | Clock | Memory | SBU/node-hr |
---|---|---|---|---|---|---|
broadwell | 177 | 32 | E5-2697A v4 | 2.6 GHz | 64 GB | * 32 |
thin | 1080 | 24 | E5-2690 v3 | 2.6 GHz | 64 GB | * 24 |
thin | 540 | 24 | E5-2695 v2 | 2.4 GHz | 64 GB | * 24 |
fat | 32 | 32 | E5-4650 | 2.7 GHz | 256 GB | * 32 |
gpu | 64 | 16 | E5-2450 v2 | 2.5 GHz | 96 GB | * 48 |
knl | 18 | 64 | 7230 | 1.3 GHz | 96 GB | * 64 |
The current system comprises:
This totals up to:
For the batch nodes we have introduced the concept of islands. Inside the bullx islands, the interconnect has a fully non-blocking fat-tree topology. Inside the sequana cell, the interconnect has a pruning factor of 2 : 1. Between islands, there is a pruning factor of 3.33 : 1 and the inter-island latency is 3 µs.
Every bullx node has a Mellanox ConnectX-3 or Connect-IB (Haswell thin nodes) InfiniBand adapter providing 4 × FDR (Fourteen Data Rate) resulting in 56 Gbit/s inter-node bandwidth, The GPGPU nodes have two ConnectX-3 InfiniBand adapters: one per GPGPU.
Every sequana node has a Mellanox ConnectX-4 InfiniBand adapter providing 4 × EDR (Enhanced Data Rate) resulting in 100 Gbit/s inter-node bandwidth.
We have 180 TB for home file systems and we have some 7.7 PB Lustre (parallel) file systems for scratch and project space.
Each node runs under one operating system (bullx Linux, which is a Linux distribution compatible with Red Hat Enterprise Linux) and shows a single memory image to the user programs.
The current (as of December 2016 ) theoretical peak performance of Cartesius is:
All performance figures we present throughout this description are based on 64-bit floating point operations. (The 32-bit floating point performance is only given for completeness / as a reference.)
Below we present benchmark results for subsets of the entire system. We only present the rankings of first appearance in the respective benchmarks.
Please note that Cartesius is designed to be a well-balanced system instead of being a TOP500 killer. This is to ensure maximum usability for the Dutch scientific community.
The SURFsara Data Archive allows the user to safely archive up to petabytes of valuable research data.
Persistent identifiers (PIDs) ensure the findability of your data. SURFsara offers a PID provisioning service in cooperation with the European Persistent Identifier Consortium (EPIC).
B2SAFE is a robust, secure and accessible data management service. It allows common repositories to reliably implement data management policies, even in multiple administrative domains.
The grid is a transnational distributed infrastructure of compute clusters and storage systems. SURFsara is active as partner in various...
Spider is a dynamic, flexible, and customizable platform locally hosted at SURF. Optimized for collaboration, it is supported by an ecosystem of tools to allow for data-intensive projects that you can start up quickly and easily.
The Data Ingest Service is a service provided by SURFsara for users that want to upload a large amount of data to SURFsara and who not have the sufficient amount...
The Collaboratorium is a visualization and presentation space for science and industry. The facility is of great use for researchers that are faced with...
Data visualization can play an important role in research, specifically in data analysis to complement other analysis methods, such as statistical analysis.