Oracle Exadata Deep Dive:

Architecture and Internals

Kothanda Umamageswaran
Vice President, Exadata Development
Gurmeet Goindi
Technical Product Strategist

X5-2

Copyright 2015, Oracle and/or its affiliates. All rights reserved.

X5-8

Exadata Vision
Dramatically Better Databases on Smart Full-Stack Platforms
Smart Hardware - Scale-out, Database optimized compute,
networking, and storage for fastest performance and lowest costs
Smart Software Unique full-stack algorithms for fastest and most
efficient OLTP, Analytics, Consolidation, and In-Memory DB
Smart Integration - Full-stack optimization, automation, and
support to reduce operations costs

Copyright 2015, Oracle and/or its affiliates. All rights reserved.

Exadata Momentum: Thousands of Deployments

Worlds #1 Selling Database Machine
Petabyte Warehouses
Business Applications

9 of the top 10 Banks, 7 of the top 10

Retailers, 10 of the top 10 Telcos use Exadata

SAP, Oracle, Siebel, PSFT,

Online Financial Trading

E-Commerce Sites
Massive DB Consolidation
Public SaaS Clouds
Oracle Fusion Apps, NetSuite,
Salesforce, Intuit, SAS

Copyright 2015, Oracle and/or its affiliates. All rights reserved.

Exadata Hardware

Copyright 2015, Oracle and/or its affiliates. All rights reserved.

Smart Hardware Architecture for Database

Highest Performance - Always Available

Smart Server Architecture

Smart Network Architecture

Starts Small, Scales Huge

Scale-Out Servers, Fastest CPUs

Unified Ultra-fast InfiniBand

Smart Storage Architecture

Fully Scale-Out Storage

Smart Offload Architecture

Application Co-Processors in Storage

Smart Flash Architecture

Ultra-fast PCIe Flash Cards

Smart Tiering Architecture

Tier Flash & Ultra-Capacity SAS Disks

Smart Cost Architecture

Industry Standard CPUs, Flash, Disks

Eliminates Legacy Anchors: Fibre Channel, Flash behind controllers and networks, Dumb Disk Trays
Slow Cluster Interconnect, Low Volume Components, Proprietary Storage OS
Copyright 2015, Oracle and/or its affiliates. All rights reserved.

New Hardware Introduced This Year

Fastest Database Servers
X5-2 with 18-core Intel E5 Haswell 2socket
X5-8 with 18-core Intel E7 Haswell 8socket

Largest Capacity
High-Capacity
Storage Servers

Fastest Storage

8 TB SAS Disks Helium

1.3 PETABYTES per rack

Elastic Configurations

1.6TB PCIe Flash Cards in High

Capacity Servers
All-Flash Exadata Storage Server
4 Million OLTP IOPs to SQL
263 GB/sec Analytics bandwidth

Database Server
Extreme Flash Storage
High-Capacity Storage

Copyright 2015, Oracle and/or its affiliates. All rights reserved.

Start Small
Add compute or storage
servers as needed
Optimize for OLTP, DW,
In-Memory, Consolidation
6

Exadata X5-2: Jan 2015 Release | Oct 2015 8TB Disks

Scale-Out Database Servers
2 socket x86 processors
36 cores New
256-768 GB DRAM

Fastest Internal Fabric

40 Gb/s InfiniBand
Ethernet external connectivity

Scale-Out Intelligent Storage

High-Capacity Storage Server
New
Extreme Flash Storage Server
Copyright 2015, Oracle and/or its affiliates. All rights reserved.

Exadata X5-2 Database Server

New 18-core Haswell-EP CPUs, DDR4 Memory expandable to 768 GB
Fastest Intel Haswell Xeon Processor
2 Eighteen-Core Intel Xeon E5-2699 v3 Processors
(2.3GHz)
256 GB DDR4 (8 x 32GB) Expandable to 768GB (24 X
32GB)

On-board 4 x 1GbE/10GbE Base-T Ethernet, 2 x

10GbE SFP+ PCIe Ethernet card
Dual-port InfiniBand QDR (40 Gb/s) HCA
4 x 2.5 600GB 10K RPM Disk Drives
Disk Controller HBA with 1GB Cache No More
Batteries
Redundant Power Supplies, Fan, ILOM
Copyright 2015, Oracle and/or its affiliates. All rights reserved.

Exadata X5-2 High Capacity Storage Server Hardware

New 8-core Haswell-EP CPUs, Larger NVMe Flash Cards, 8TB Disk Drives
2 Eight-Core Intel Xeon E5-2630 v3 2.4 GHz
96 GB DRAM (4 x 8 GB + 4 x 16 GB)

Fastest PCIe Flash

4 x 1.6 TB Flash Accelerator F160 PCIe Card
Non Volatile Memory Express (NVMe) protocol

12 x 3.5 8TB SAS drives 96 TB Raw Capacity

Dual-port InfiniBand QDR (40Gb/s) HCA all
ports active
Disk Controller HBA with 1 GB Cache and
Super Capacitors No More Batteries
Redundant Power Supplies, Fans, ILOM
Copyright 2015, Oracle and/or its affiliates. All rights reserved.

New: Exadata X5-2 Extreme Flash Storage Server Hardware

New 8-core Haswell-EP CPUs, NVMe Flash Drive, No Spinning Drive
2 Eight-Core Intel Xeon E5-2630 v3 2.4
GHz
64 GB DRAM (8 x 8 GB)

Fastest PCIe Flash Drives

8 x 1.6 TB 2.5-inch Flash Accelerator F160 PCIe
Drives
Non Volatile Memory Express (NVMe) protocol
Hot Swappable

Dual-port InfiniBand QDR (40Gb/s) HCA all

ports active
Redundant Power Supplies, Fans, ILOM
Copyright 2015, Oracle and/or its affiliates. All rights reserved.

10

Announcing: Exadata X5-8: Oct 2015 Release

Scale-Out Database Servers
8 socket x86
processors
New 144 cores
2-6 TB DRAM

Fastest Internal Fabric

40 Gb/s InfiniBand
Ethernet external connectivity

Scale-Out Intelligent Storage

Same Networking and Storage as X5-2

High-Capacity Storage Server

Extreme Flash Storage Server

Copyright 2015, Oracle and/or its affiliates. All rights reserved.

11

Announcing: Exadata X5-8 Database Server

18-core Haswell-EX CPUs, Up to 3X More Memory
8 Eighteen-Core Intel Xeon E7-8895 v3
Processors (2.6GHz)
2 TB (64 x 32GB) - Expandable to 6TB (192 X
32GB)
4 x Dual-port 10GbE PCIe ports
4 x Dual-port InfiniBand QDR (40 Gb/s) Express
modules
8 x 2.5 600GB Disk Drives
Disk Controller HBA with 1 GB Cache No More
Batteries
Redundant Power Supplies, Fans, ILOM
Copyright 2015, Oracle and/or its affiliates. All rights reserved.

12

X5-2 Elastic Configurations

Optimize Exadata for any Workload
Qtr Rack

Full Rack

Configuration Examples
DB In-Memory Extreme Flash OLTP Data Warehousing
Machine
Machine
Machine

Database Server
Extreme Flash Storage
High-Capacity Storage

Start with

2 Database Servers
3 Storage Servers

Add Servers
Any Kind
Any Quantity

15 DB Servers
5 Storage Servers

11 DB Servers
11 Storage Servers

8 DB Servers
14 Storage Servers

576 DB Cores

396 DB Cores
8 TB RAM

512 Cores
90 TB Flash Cache

13.3 TB RAM
480 TB Disk

140 TB Flash

Copyright 2015, Oracle and/or its affiliates. All rights reserved.

1,344 TB Disk

13

X5-8 Elastic Configurations

Very Large Scale Compute and Memory
Half Rack

Full Rack
Database Server

Configuration Examples
Consolidation
Extreme Flash OLTP
Machine
Machine

Extreme Flash Storage

High-Capacity Storage

Start with

2 Database Servers
3 Storage Servers

Add Servers
Any Kind
Any Quantity

4 DB Servers
7 Storage Servers

2 DB Servers
14 EF Storage Servers

576 DB Cores
24 TB RAM
672 TB Disk

288 DB Cores
12 TB RAM

Copyright 2015, Oracle and/or its affiliates. All rights reserved.

179 TB Flash
14

Exadata X5 Fastest Database Machine

Extreme Flash
Storage Server

High-Capacity
Storage Server

All-Flash

Disk + Flash Cache

State-of-the-art NVMe PCIe flash

Consistently Low Response Times
Optimized InfiniBand I/O Protocols
Exadata Storage Server Software
Smart Scan (SQL Offload)
Smart Flash Cache
I/O Resource Management
Hybrid Columnar Compression

Performance

Extreme Flash

High-Capacity

Analytic Scans

263 GB/s

140 GB/s

OLTP Reads (8K)

4.14 M IOPS

4.14 M IOPS

OLTP Writes (8K)

4.14 M IOPS

2.69 M IOPS

Flash Latency

0.25 ms @ 2M
IOPS

0.25 ms @ 1M
IOPS

Capacity

Extreme Flash

High-Capacity

16

16

Disk (per server)

96 TB

Flash (per server)

12.8 TB

6.4 TB

Disk (full rack)*

1,344 TB

Flash (full rack)*

179.2 TB

89.6 TB

Cores (for SQL offload)

* Full Rack : 8 DB servers, 14 storage servers

Copyright 2015, Oracle and/or its affiliates. All rights reserved.

15

Exadata Cloud Service

Copyright 2015, Oracle and/or its affiliates. All rights reserved.

Exadata Cloud Service

Single Cloud Service for ALL Database Workloads

OLTP

Deploy Mission Critical Enterprise Databases

with a few clicks

Disaster Recovery

100% Database Compatibility enables easy

migration and hybrid deployments

Data Warehousing

Oracle Experts Manage Infrastructure

DB Consolidation

Pay per Use

In-Memory Analytics

Complete Isolation of tenants with no

overprovisioning

App Certification
Copyright 2015, Oracle and/or its affiliates. All rights reserved.

17

Exadata Cloud Service: Scalable, Available, Secure, Compatible

Decades of Database Innovation - Proven at Millions of Mission Critical Deployments

Multitenant

Offload SQL to Storage

In Memory Database

InfiniBand Fabric

Real Application
Clusters

Smart Flash Cache,

Smart Flash Log

Active Data Guard

Partitioning
Advanced
Compression
Advanced Security,
Label Security,
Database Vault

All Oracle
Database
Innovations

All Exadata
Platform
Innovations

PCI Flash

Columnar Flash
Cache
Hybrid Columnar
Compression
I/O Resource
Management

HCC

10:1
I/O

I/O

I/O

Network Resource
Management

Real Application
Testing

In-Memory Fault
Tolerance

OLAP, Analytics,
Spatial and Graph

Exafusion Direct-toWire Protocol

Management Packs
Copyright 2015, Oracle and/or its affiliates. All rights reserved.

18

Smart Exadata Software

Copyright 2015, Oracle and/or its affiliates. All rights reserved.

Exadata Unique Smart Platform Software

Smart Analytics

Smart Storage

Move queries to storage, not storage

to queries
Automatically parallelizes queries
across all storage servers

Hybrid Columnar Compression

reduces space usage by 10x
Database aware Flash Caching gives
speed of flash with capacity of disk

100x Faster Analytics

Smart OLTP

Smart Consolidation

Special InfiniBand protocol for best

OLTP latency : 250us IOs @ 2MM IOPs
Ultra-fast transactions using DB
optimized flash logging algorithms

Workload prioritization from CPU to

Network to Storage ensures QOS
4X more Databases in same hardware

Copyright 2015, Oracle and/or its affiliates. All rights reserved.

20

Dozens of Additional Smart Platform Capabilities

Smart Analytics

Storage Index data skipping

Storage offload for min/max
operations
Data Mining Storage Offload
Storage offload for LOBs and
CLOBs
Auto Flash Caching for Table
Scans
Reverse Offload to DB servers
Offload Index Fast Full Scans
Offloads Scans on Encrypted
Data, with FIPS compliance
Active Bonding of InfiniBand
Instant data file creation

Smart OLTP

Smart network packet

prioritization
I/O Prioritization by DB, User, or
workload to ensure QOS
Active AWR includes storage
stats for end to end monitoring
Write-back Flash Cache
Cell-to-Cell Rebalance preserving
Flash Cache
Secure disk and flash erase
Database scoped security
Full-stack security scanning
Exachk full-stack validation
NVMe flash interface for lowest
latency IO

Smart Availability

In-Memory Fault Tolerance

Offload backups to storage servers
Prioritize rebalance of critical files
Elimination of false drive failures
Flash and disk life cycle
management alert
Avoid reading Predictive failed
disks
Cell software transparent restart
I/O hang hardening
Prevent shutdown if mirror server
is down
Confinement of poor performing
drives

Copyright 2015, Oracle and/or its affiliates. All rights reserved.

21

Smart Platform Capabilities Introduced This Year

Smart Consolidation

Smart Analytics
5X faster using Columnar Flash Cache
3X faster JSON/XML using storage
offload

VM

VM

Smart OLTP
3X faster OLTP messaging - DB talks
directly to InfiniBand, bypassing OS
Instant detection of Node Failure
Sub-second Capping of I/O Latency

FIN VM
SALES
VM

DB
Vault
Option
Spatial
& Graph
Option

Zero Overhead Xen VMs

Snapshots for Test/Dev
InfiniBand Partitioning

Smart Licensing
Capacity-on-Demand reduces
license cost by disabling cores
Trusted Partitions limit license
scope of specialized options

Copyright 2015, Oracle and/or its affiliates. All rights reserved.

22

Smart Availability: Fast Mirror Resynchronization

Resync Time Across Power Levels
90

Minutes to Resync Cell

80

With 12.1.0.2 Grid Infrastructure,

resync of an offline disk is treated
just like a rebalance

81

70

One Hour Savings

60
50

39

40

HC Cell

30
20

EF Cell

21

18
10

10

11

6 4

6 4

32

64

Increased power level can cut down

resync time by over an hour
Improvement multiplies by #cells for
rolling upgrades

Recommend upgrade to 12.1.0.2 GI

0
1

16

ASM Power Limit

Copyright 2015, Oracle and/or its affiliates. All rights reserved.

23

Smart Management: RESTful Service and Exacli

RESTful Management of storage servers
No need to log onto storage servers and run cellcli
Attribute level security
Use exacli from the compute node
Use curl or web browser to send commands to the storage server

Lockdown the storage server and disable ssh access

Upcoming Exadata Cloud Service will allow customers to manage and
monitor storage
[root@compute08 ~]# exacli cell11
cell11/Username:scott
Password:**************
cell11> list griddisk
CELL-06015: Current user does not have privileges to run this command.
cell11> list griddisk attributes name, size
data_CD_00_scam07celadm11
2.6954193115234375T
data_CD_01_scam07celadm11
2.6954193115234375T
...
Copyright 2015, Oracle and/or its affiliates. All rights reserved.

24

Additional Features Enabled

X5-8 server with up to 25% improvement in performance
InfiniBand Partitioning for security and isolation in both virtual and
bare metal environments
IPv6 support for management, ILOM, client access ethernet
networks
Fixed Allocations for databases in the flash cache
Databases cannot over run their flash cache size and starve other
databases in a consolidated environment

Up to 15% elapsed time speedup due to faster reverse offload

Cell to Cell offload of resilver, resync, rebalance preserves flash
cache
Exadata Snapshot Management for pluggable and non-pluggable
databases in Enterprise Manager 12c Release 5
Copyright 2015, Oracle and/or its affiliates. All rights reserved.

25

Conclusion: Smart Platforms are the Future

Dramatic Performance and Cost Advantages

In-Memory Fault Tolerance

Direct-to-wire Protocol
Columnar Flash Cache
JSON and XML offload
Oracle Database Platforms
I/O latency capping
Smart by Design
Instant failure detection
Network Resource Management
Smarter Every Year
Multitenant Aware Resource Mgmt
Prioritized File Recovery
Software-in IO Priorities
Silicon
Data Mining Offload
Tiered Disk/ Flash
Offload Decrypt on Scans
PCIe Flash
Database Aware Flash Cache
Storage Indexes
Application Co-Processors
Columnar Compression
Smart Scan
Unified InfiniBand
InfiniBand Scale-Out
Scale-Out Storage

Smart Phone
or
Basic Phone

Scale-Out Servers

Copyright 2015, Oracle and/or its affiliates. All rights reserved.

26

Copyright 2015, Oracle and/or its affiliates. All rights reserved.

27