Juniors: Prototypes for novel exascale I/O concepts

H. El-Harake, S. El Sayed, U. Fischer, S. Graf, M. Hennecke, W. Homberg, K. Kutzer, J. Lauritsen, O. Mextorf, P. Morjan, D. Pleiter, H. Schick, G. Schwarz, M. Stephan

To meet future demands of exascale systems new I/O concepts are required since compute power and performance of storage technologies are developing at different speed. With PCIe flash cards a new fast, persistent storage technology is emerging which can bridge the performance gap between volatile main memory and persistent disk-based storage devices.

Prototype I: Servers with x86 Processors

Cluster of 8 IBM xSeries servers 10 GbE interconnect Different PCIe flash devices: Fusion-IO Duo 320 GByte TSM Ramsan 450 and 900 GByte Status: Operational
ssh juniors.fz-juelich.de

Prototype II: Based on Blue Gene/Q Technology

Setup description 4 Blue Gene/Q I/O drawers 8 I/O nodes per drawer Flash card devices: TMS Ramsan 450 GByte 10 GbE adapter (for external connectivity) Status: Initial unit tests successful

Juniorsm (.115.66)
x3650M3

Management Net (134.94.115.64/27)

Juniors1 (.115.67) x3650M3 / FusionIO Juniors2 (.115.68) x3650M3 / FusionIO Juniors3 (.115.69)
x3650M3 / FusionIO

Fusion-IO Duo 320 GBytes

Read BW [Gbyte/s] Write BW [Gbytes/s] Read IOPs

Nexus 7k Switch

1.5 1.5 261,000 262,000 4x

TMS Ramsan 450 GBytes 1.25 0.9 300,000 220,000 8x

Write IOPs PCIe GEN2 bus width

Mitglied der Helmholtz-Gemeinschaft

juniors4 (.115.70) x3650M3 / FusionIO Juniors5 (.115.71)

x3650M3 / FC

Flash Storage Managed by Filesystem

Nexus 7k Switch

DCS3700 / FC 2TB HDDs (RAID6) DCS3700 / SAS 2TB HDDs (RAID6)

Juniors6 (.115.72)
x3650M3 / FC

Juniors7 (.115.73)
x3650M3 / SAS

Juniors8 (.115.74)
x3650M3 / SAS

Concept verified on prototype I: GPFS pools on flash storage and disk Use GPFS policy engine to manage placement and migration Example: Migration of oldest files triggered by LOW_SPACE event:
RULE MIGRATE FROM POOL 'flash' THRESHOLD(60,20) WEIGHT(CURRENT_TIMESTAMP-ACCESS_TIME) TO POOL 'disk'

I/O Benchmarks
IOR: Parallel benchmark supporting different I/O interfaces like POSIX, MPI-IO, HDF5 FIO: Parallel benchmark with multiple engines for generating synchronous and asynchronous I/O requests Others: IOzone, mdtest and application benchmarks
ioengine=libaio, bs=1M, iodepth=64
1.3 1.2

Prototype I

ioengine=libaio, bs=1M, iodepth=64

1.3 1.2

Prototype II

GByte/s

1.1 1 0.9 0.8

GByte/s

read write

1.1 1 0.9 0.8

read write

Preliminary performance results using FIO

Bandwidth and IOP rates at or close to vendor specification Performance of GPFS using 2 Fusion-IO cards similar to raw device access Performance on lower-clocked many-core devices will improve using interfaces and devices which enable highly concurrent access to flash memory

0.7

N thread

64

0.7

N thread

64

400 350 300

ioengine=libaio, bs=4K, iodepth=512

Prototype I

150

ioengine=libaio, bs=4K, iodepth=512

Prototype II

100

kIOPS

200 150 100 50 1 8

kIOPS
50

250

random read random write N thread

64 0 1 8

random read random write N thread

64

Supported by