XtreemStore –

A SCALABLE STORAGE MANAGEMENT

SOFTWARE WITHOUT LIMITS
YOUR DATA. YOUR CONTROL
Software Produkt Portfolio

New Products

Product Family

Scalable sync & share solution for - Software for high performance
secure data exchange for data archiving
organizations and enterprises - Parallel file system with
integrated archive

Established Products

FileLock

Software WORM solution for Windows and Linux based HSM

audit-proof long-term data software for long-term data
archiving on disk systems archiving up to petabyte scale
Archive Manager - the Basis for XtreemStore

PACS
Scientific VIDEO
DMS Email / Files Others
Data PrePress
CAD/CAM

NFS / CIFS

TCP/IP (remote) Archive Server SAN (remote)

Incl. Performance Disk

Tape Tape
Archive Archive
Disk Disk „Tape Media“ „Tape Media“
Archive Archive
“Disk Media” “Disk Media”
Client – Server Architecture

Advantages of different configurations

Windows/Linux Clients
Keep it simple Several clients for
for seamless integration
“All in one box” high throughput
into CIFS or NFS world

CIFS NFS
Client 1 Client 2 Client n Windows
Client Linux Client
Client
Server GAM Server Win/Linux GAM Server Win/Linux

Disk Tape Disk Tape

Disk Tape
Several Disk and Tape Pools

UNIX WINDOWS

LAN

Archive
Manager
Remote Copy

Disk Tape Tape

Media
Media Media

Partitioning for physical

separation of data
Key Features

 Online access to „unlimited content“

 Multiple media strategy
− Secure and cost effective archiving on tape and / or removable disk
− fast access to disk
 Partitioning for different applications („data separation“)
 Remote copy functionality
 Several levels of backup / recovery options
 Free selection of archival hardware (server, disk, tape libraries)
 Direct access through file system (additional API optional)
 Native support for CIFS (Active Directory) & NFS
XtreemStore
HPC – Posix Archive
YOUR DATA. YOUR CONTROL
Need for high performance POSIX Archive

 POSIX interface is very common to all applications

 But Standard File System have limitations
 ArchiveManager inherits those limitations when working with Standard file systems
 Major limits:
− The amount of files within one File System
− The throughput of one single File System
− Throughput for huge files (10TB and more)

 To digest large data volumes from parallel sources GAM needs an embedded
adapted parallel file system ( P/HSM-FS ) - „Everything needs to be parallel“

 GAM + P/HSM-FS  XtreemStore

 P/HSM-FS is an adaption of BeeGFS
 New product: XtreemStore

 ArchiveManager (GAM) with built in Parallel HSM File System

 XtreemStore

 Storage Management Software (HSM; Archive; Backup) for high

performance applications.

 Parallel access through a Meta File System with POSIX interface to

unlimited amount of storage on disk and tape.

 Grid structure build on standard PC hardware

Software Architecture XtreemStore

NFS / CIFS NFS / CIFS NFS / CIFS

Parallel HSM File System

XtreemStore GAM Client

GAM Server

Driving an unlimited amount of standard hardware

Overview XS-GAM

 BeeGFS is adapted to
BeeGFS BeeGFS BeeGFS *n work with GAM
Client Client Client
Meta Data
 Each storage node of
Server
BeeGFS becomes a
BeeGFS
HSM-node
Storage Storage Storage
Server Server Server *n  This works OK as an
extremely fast
archive or as target
GAM Client GAM Client GAM Client for HPC-HSM
Primary Disk Primary Disk Primary Disk
 Some high speed
Raid 5/6 Raid 5/6 Raid 5/6
operations of BeeGFS
GAM Server GAM Server GAM Server are slowed down
Secondary Secondary Secondary
 For HPC-
Disk Disk Disk
improvements the
and Tape and Tape and Tape
architecture is in
rework process.
(later chart)
Near Future: Seamless integration of HPC and HSM

BeeGFS BeeGFS New architecture:

Client Client
Linux Window  Pools of nodes
Meta Data s
Server  Nodes may have
BeeGFS different
characteristics
Storage Node
storage nodeNode
Storage storage node
Storage Node Long Term
storage node storage node  Life cycle of data
for for includes several
for for GAM Client
high-speed high-speed
medium
fast
high-speed high-speed GAM Client types of storage
i.e.
i.e.SSD i.e. SSD
i.e. SAS
Primary Disk
i.e.SSD
SSD i.e. SSD Primary
Raid 5/6Disk
Raid 5/6  Data end up on tape
GAM Server
GAM Server
Secondary  All data are
Secondary
Disk managed in one
Disk
and Tape
and Tape
name space
Storage Management for Lustre: Overview

Lustre Clients & Server

Robinhood
Policy Lustre Server
Engine
HSM API

XtreemStore
Parallel Data Mover
Storage Adapter

Backend
Technologies
Object NAS p/XS/FS
Store
XtreemStore HSM
S3, Isilon GRAU
etc. ArchiveManger
etc.
(Future) Disk + Tape
from any vendor
Use Case 1: Standard Throughput

Lustre Lustre Lustre Lustre Lustre Lustre Lustre Lustre

Client Client Client Client Client Client OST OST

Lustre
Client
HSM-API
PDM

NAS – Device (single path) i.e.

ArchiveManager
Use Case 2: High Throughput

Lustre Lustre Lustre Lustre Lustre Lustre Lustre Lustre

Client Client Client Client Client Client OST OST

Lustre Lustre Lustre

Client Client Client
HSM-API
PDM PDM PDM

NAS – Device with parallel ingest

i.e. XtreemStore HSM / Isilon / . . .
XtreemStore
Parallel Data Mover (XS-PDM)
for HPC
YOUR DATA. YOUR CONTROL
XS - Parallel Data Mover (currently POSIX only)

Data Data Data Data Data Data

Mover Mover Mover Mover Mover Mover

 The XS - Parallel Data Mover is just software

 It may run on
− the source machine
− the target machine
− or on dedicated computer nodes
 The amount of streams running in parallel is not limited
Possible Environment

 In theory PDM works in any environment.

Source Source Source  Throughput depends only on hardware
Node Node Node
infrastructure.
 Grid structure environment seems best
matching to those requirements.
Data Data Data
Mover Mover Mover  To scale to full speed, it needs source and
target nodes that can deliver and ingest
the required amount of data and a
network without bottlenecks.
Target Target Target  The amount of streams running in parallel
Node Node Node is not limited.
How the PDM works

Source File Target File

1 5 9 D Data Mover 1 5 9 D

2 6 A E Data Mover 2 6 A E
0
3 7 B F Data Mover 3 7 B F

4 8 C G Data Mover 4 8 C G

 Large files are copied chunk-wise

− Each PDM node copies one set of chunks
− The number of threads and the chunk sizes are freely configurable
− The throughput capacity of target and source nodes should be
matched
 Small files are copied single-threaded, the threshold is configurable
Customer Installations
Customer Installation samples

 Lustre installation at University of Rijeka

 BeeGFS with continuous backup functionality at

Automotive Supplier

 Installation at Intel lab Lustre to Lustre

Lustre + XtreemStore at University of Rijeka

 Standard architecture
 About 20 Lustre client nodes
 4 Lustre storage nodes
 4 HSM nodes for throughput
− 2 physical machines
− 2 HSM nodes each
 1 Tape library with 400 tapes and 4 drives LTO-6
 Productive since November 2015
Scratch File System gets Archive + Continuous Backup

Lustre Lustre Lustre

Clients Storage Node Storage Node

Parallel Data Data Data Data Data Scratch File System

Master Mover Mover Mover Mover
Mover
Primary Storage

HSM for Archive and Backup XtreemStore for

Archive and Backup
GAM Client

GAM Server
Rijeka Functionality

 Lustre clients are working for HPC applications

 HSM – functionality
− “Old” data are moved to XtreemStore
− Storage on Lustre is released
− Files are transparently accessible through Lustre
− Limited read access at archive level
BeeGFS + XtreemStore at Automotive Supplier

 Scale out NAS device

 3 BeeGFS storage nodes
 2 Xtreemstore HSM nodes
− 2 physical machines
− 2 virtual machines each

 2 Tape libraries with 50 tapes and 2 drives

 Productive since February 2016
 Installation – Automotive Supplier: Continuous Backup

 All Data are

Win Win Win Linux synchronized from
Clients Linux Additional
*n BeeGFS
Linux clients BeeGFS into GAM –
CIFS CIFS CIFS BeeGFS
Client HSM
BeeGFS optional
Client
External Client
 Currently all data are
LAN 1GB kept in two file
systems..
SAMBA Linux SAMBA Linux
BeeGFS BeeGFS GAM  GAM behaves as
Meta Data Client Client Backup for the disk
Server BeeGFS Archive.
BeeGFS Client
Sync-  Inactive data may be
Storage Storage Storage Tool removed from BeeGFS
Internal Server Linux 1 Server Linux 2 Server Linux 3 *n and are available on
LAN 10GB BeeGFS BeeGFS BeeGFS Disk GAM only.

i.e. i.e. i.e. T T  Everywhere the

15 TB Disk 15 TB Disk 15 TB Disk A A capacity may be
Raid 5/6 Raid 5/6 Raid 5/6 P P extended on demand.
E E
1 2
250 TB – X PB
Functionality

 BeeGFS offers scale out file system

− NFS & CIFS (through Samba )
 HSM – functionality
− “Old” data are moved to HSM nodes
− Storage in primary nodes is released
 Backup – functionality
− “Important” data are copied to HSM nodes
− Files are kept on primary storage nodes
Scratch File System gets Archive + Continuous Backup

Lustre Lustre Lustre

Clients Storage Node Storage Node

XtreemStore Data Data Data Data

Scratch File System
Master Mover Mover Mover Mover
Parallel Data
Mover Primary Storage

Lustre Lustre Lustre

Clients Storage Node Storage Node

Secondary Storage
Lustre + PDM + Lustre at Intel Lab

 Primary Lustre clients are working for HPC applications

 Secondary Lustre acts as scale out NAS device
 Backup – functionality
− “Important” data are copied to secondary Lustre
− Files are kept on primary storage nodes
− In regular mode files are “read only” on secondary Lustre
− In disaster situations operation may switch to secondary
level Lustre
− Primary Lustre may be rebuild from secondary Lustre
YOUR DATA. YOUR CONTROL

www.graudata.com