System Type

Cartesius is a bullx system extended with one Bull sequana cell. It is a clustered SMP (Symmetric Multiprocessing) system built by Atos/Bull.

System Overview

Cartesius consists of a large number of batch nodes and a small number of special purpose nodes. For the batch nodes we differentiate between so-called thin nodes, fat nodes, GPU nodes and Xeon Phi nodes. Thin nodes constitute the majority of the available batch nodes. GPU nodes use GPGPUs to accelerate the computations. The Xeon Phi nodes have Intel Many-Core CPUs. Fat nodes have more memory (256 GB) than the thin nodes and more physical cores (32) than most thin nodes.

Special nodes are the two interactive front end nodes and five so-called service nodes. The interactive front end nodes provide the ssh login service and are meant for interactive work. The service nodes are meant for data staging (transfers to and/or from external contexts, such as the SURFsara archive facility or a remote site.).

Currently, Cartesius consists of the following compute nodes:

Detailed Node Description

The current system comprises:

• 1 thin node island, a so-called Bull sequana X1000 cell (as of December 2016):
  • 177 sequana X1110 thin nodes
    • 2 × 16-core 2.6 GHz Intel Xeon E5-2697A v4 (Broadwell) CPUs/node
    • 64 GB/node
  • 18 sequana X1210 Xeon Phi nodes
    • 1 × 64-core 1.3 GHz Intel Xeon Phi 7230 (Knights Landing) CPU/node
    • 96 GB/node
• 3 thin node islands (as of November 2014), consisting of 360 bullx B720 thin nodes each
  • 2 × 12-core 2.6 GHz Intel Xeon E5-2690 v3 (Haswell) CPUs/node
  • 64 GB/node
• 2 thin node islands (as of June 2013), one consisting of 360 bullx B710 thin nodes and one consisting of 180 bullx B710 thin nodes
  • 2 × 12-core 2.4 GHz Intel Xeon E5-2695 v2 (Ivy Bridge) CPUs/node
  • 64 GB/node
• 1 accelerator island (as of May 2014) consisting of 66 bullx B515 GPGPU accelerated nodes
  • 2 × 8-core 2.5 GHz Intel Xeon E5-2450 v2 (Ivy Bridge) CPUs/node
  • 2 × NVIDIA Tesla K40m GPGPUs/node
  • 96 GB/node
• 1 fat node island (as of April 2013) consisting of 32 bullx R428 E3 fat nodes
  • 4 × 8-core 2.7 GHz Intel Xeon E5-4650 (Sandy Bridge) CPUs/node
  • 256 GB/node
• 2 bullx R423-E3 interactive front end nodes
  • 2 × 8-core 2.9 GHz Intel Xeon E5-2690 (Sandy Bridge) CPUs/node
  • 128 GB/node
• 5 bullx R423-E3 service nodes
  • 2 × 8-core 2.9 GHz Intel Xeon E5-2690 (Sandy Bridge) CPUs/node
  • 32 GB/node

Nodes Summary

This totals up to:

• 47,776 cores + 132 GPUs: 1.843 Pflop/s (theoretical peak performance)
  • 25,920 Haswell cores (thin nodes): 1.078 Pflop/s
  • 12,960 Ivy Bridge cores (thin nodes): 249 Tflop/s
  • 5,664 Broadwell cores (thin nodes): 236 Tflop/s
  • 1,056 Ivy Bridge cores + 132 K40m GPUs (GPGPU nodes): 210 Tflop/s
  • 1,152 Knights Landing cores: 48 Tflop/s
  • 1,024 Sandy Bridge cores (fat nodes): 22 Tflop/s
• 130 TB memory (CPU + GPGPU + HBM)

Interconnect

For the batch nodes we have introduced the concept of islands. Inside the bullx islands, the interconnect has a fully non-blocking fat-tree topology. Inside the sequana cell, the interconnect has a pruning factor of 2 : 1. Between islands, there is a pruning factor of 3.33 : 1 and the inter-island latency is 3 µs.

Every bullx node has a Mellanox ConnectX-3 or Connect-IB (Haswell thin nodes) InfiniBand adapter providing 4 × FDR (Fourteen Data Rate) resulting in 56 Gbit/s inter-node bandwidth, The GPGPU nodes have two ConnectX-3 InfiniBand adapters: one per GPGPU.

Every sequana node has a Mellanox ConnectX-4 InfiniBand adapter providing 4 × EDR (Enhanced Data Rate) resulting in 100 Gbit/s inter-node bandwidth.

Disk Space

We have 180 TB for home file systems and we have some 7.7 PB Lustre (parallel) file systems for scratch and project space.

Operating System

Each node runs under one operating system (bullx Linux, which is a Linux distribution compatible with Red Hat Enterprise Linux) and shows a single memory image to the user programs.

TOP500, Green500 and HPCG ranking

The current (as of December 2016 ) theoretical peak performance of Cartesius is:

• 1.843 Pflop/s (64-bit floating point)
• 3.874 Pflop/s (32-bit floating point)

All performance figures we present throughout this description are based on 64-bit floating point operations. (The 32-bit floating point performance is only given for completeness / as a reference.)

Below we present benchmark results for subsets of the entire system. We only present the rankings of first appearance in the respective benchmarks.

• Cartesius Phase 2 - all thin nodes subset. In November 2014, Cartesius Phase 2 entered the TOP500 at position 45 and the Green500 at position 80. These entries involved all thin nodes, both Haswell and Ivy Bridge, resulting in a (heterogeneous) LINPACK performance (Rmax) of 1,052.5 Tflop/s at an energy consumption of 1,471.20 Mflop/s/W.

• Cartesius Phase 2 - Haswell thin nodes subset. In November 2014, the Cartesius Haswell thin nodes subset entered the Little Green500 at position 49 with a LINPACK performance (Rmax) of 847.7 Tflop/s at an energy consumption of 1,802.13 Mflop/s/W. Also in November 2014, the same Cartesius Haswell thin nodes subset entered the HPCG at position 17 with a Conjugate Gradients performance of 0.019 Pflop/s.

• Cartesius Accelerator Island. In June 2014, the Cartesius Accelerator Island entered the TOP500 at position 421 with a LINPACK performance (Rmax) of 153.6 Tflop/s and the Green500 at position 4 with an energy consumption of 3,459.46 Mflop/s/W. In November 2014, the Cartesius Accelerator Island entered the HPCG at position 25 with a Conjugate Gradients performance of 0.003 Pflop/s.

Please note that Cartesius is designed to be a well-balanced system instead of being a TOP500 killer. This is to ensure maximum usability for the Dutch scientific community.