Storage in AWS

Storage on AWS
Name, Title
Date
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
• Introduction
• Storage Primer
• Block Storage
• Shared File Systems
• Object Store
• On-Premises Storage Integration
Introduction: Why choose AWS for storage
Compelling Speed,
Economics Easy to Use Reduce risk Agility, Scale
Pay as you go Self service Durable and Secure Reduce time to
administration market
No upfront Avoid risks of
investment SDKs for simple physical media Focus on your
No commitment integration handling business, not your
infrastructure
No risky capacity
planning
No need to
provision
for redundancy
or overhead
0
Storage Primer
Block vs File vs Object
Block Storage
Raw Storage
Data organized as an array of unrelated blocks
Host File System places data on disk
e.g.: Microsoft NTFS, Unix ZFS
File Storage
Unrelated data blocks managed by a file (serving) system
Native file system places data on disk
Object Storage
Stores Virtual containers that encapsulate the data, data attributes, metadata and Object IDs
API Access to data
Metadata Driven, Policy-based, etc
Storage - Characteristics
Some of the ways we look at storage
Durability Availability Security Cost Scalability Performance Integration
Measure of Measure of Security Amount per Upward Performance Ability to
expected expected measures in storage flexibility metrics interact with
data loss downtime place unit, e.g. $ /
GB
AWS has a variety of storage options
Amazon EBS (Elastic Block Storage)
Amazon Elastic File System (EFS)
Amazon EC2 Instance Store (Ephemeral Volumes)
Amazon S3 (Simple Storage Service)
Amazon Glacier
ew!
AWS Storage Gateway: File Gateway N
ew!
Amazon Snowball & Snowball Edge N
ew!
AWS Snowmobile N
1
Block Storage
Amazon EBS
• Persistent block level storage for EC2

• Pay only for what you provision
• Native redundancy and write cache
• Consistent and low-latency performance
• Optimized for random I/O
• Native support for encryption at rest (data volumes)
Amazon EBS
Network attached block device

• Independent data lifecycle
• Virtual disks
• Multiple volumes per EC2 instance
• Only one EC2 instance at a time per volume
• Can be detached from an instance and attached to a different one
Raw block devices
• Unformatted block devices
• Ideal for databases, filesystems
Available in multiple types

AWS EBS Features
Durable Secure Scalable
Designed for five Identity and Unlimited
9’s reliability Access Policies capacity when
Redundant storage Encryption you need it
across multiple devices Easily scale up
within an AZ and down
Performance Backup
Low-latency SSD Point-in-time Snapshots
Consistent I/O Performance Copy snapshots across AZ
Stripe multiple volumes for and Regions
higher I/O performance
Amazon EBS
Highly available block storage for all types of data
Internet-scale storage Built-in redundancy Low price per GB Benefit from AWS’s
Grow without limits Designed for per month massive security
99.999% availability No commitment investments
No up-front cost
EBS Volume Types Comparison
Magnetic General Purpose Provisioned IOPS
(SSD) (SSD)
Performance Lowest Cost Burstable Predictable
Use Cases Infrequent Data Boot volumes I/O Intensive
Access Small to Medium Relational & NoSQL
DBs
Dev & Test
Media Magnetic (HDD) SSD SSD
Max IOPS 100 on average with Baseline 3 IOPS/GB Consistently performed
the ability to burst Burstable to 3,000 at provisioned level, up
to hundreds of IOPS IOPS to 32,000 IOPS
Price $.05/GB/Month $.10/GB/Month $.125/GB/Month
$.05/million I/O I/O Operations - $.065/provisioned IOPS
Free
EBS Provisioned IOPS
(SSD) (SSD)
Perform Lowest Cost Burstable Predictable

ance
Use Infrequent Data Boot volumes I/O Intensive

Cases Access Small to Medium Relational & NoSQL
DBs
Dev & Test
•EBS Optimized Instances

Max 100 on average Baseline 3 Consistently performed
IOPS with the ability to IOPS/GB at provisioned level, up
burst to hundreds Burstable to 3,000 to 32,000 IOPS
of IOPS IOPS

• Dedicated storage throughput

Free
•Predictable Performance
• 100-32000 IOPS per volume
• Single digit millisecond latency
•Performance Design
• Deliver within 10% of PIOPs, 99.9%
of the time
Enhanced Throughput for PIOPS & GP2 Volumes
(SSD) (SSD)

ance

DBs
Dev & Test

of IOPS IOPS

• Maximum attainable throughput to each volume now

Free
at 500 MB/s read or write traffic (on instance that

supports r3.8xl or x1.32xl)
• An I/O request of up to 256 KB is now counted as a single I/O
operation (IOP)
• Capable of dramatically reducing your storage costs
Amazon EBS at 32,000 IOPS
(SSD) (SSD)

ance

DBs
Dev & Test

Provisioned IOPS (SSD)

of IOPS IOPS

Free
• Max Volume 16 TB
• Max I/O rate 32,000 IOPS
• Max throughput 500 MB/s
General Purpose (SSD)

• Max Volume 16 TB
• Max I/O rate 10,000 IOPS
• Max throughput 160 MB/s
EBS Snapshots
AWS Cloud
EC2 Availability Zone Amazon S3
Create Snapshot
EBS EBS EBS EBS EBS EBS
EBS Snapshot
Clone From EBS Snapshot
Snapshot EBS Snapshot
EC2 EC2 EC2 EBS Snapshot
EBS Snapshot
Internet
How Do Snapshots Work?
Time
X
Snapshot 1 Snapshot 2 Snapshot 3
S3
EBS Volume
Block 11
Chunk
Block 22
Chunk
Block 33
Chunk
Block 44
Chunk
EC2 Instance Store (Ephemeral Volumes)
Free with your EC2 instance

• SAS and SSD options
• Size/type based on instance type
Local, direct attached resource
Consistent sequential reads and writes
Use only for non-persistent data
2
Shared file system
Elastic File System (EFS)
• Fully managed file system for EC2 instances

• Provides standard file system semantics
• Works with standard operating system APIs
• Sharable across thousands of instances
• Elastically grows to petabyte scale
• Delivers performance for a wide variety of workloads
• Highly available and durable
• NFS v4–based
• Accessible from on-premise servers New!
1 Amazon EFS is Simple
Fully managed
- No hardware, network, file layer
- Create a scalable file system in seconds!
Seamless integration with existing tools and apps
- NFS v4.1—widespread, open
- Standard file system access semantics
- Works with standard OS file system APIs
Simple pricing = simple forecasting
2 Amazon EFS is Elastic
• File systems grow and shrink

automatically as you add and remove
files
• No need to provision storage capacity or
performance
• You pay only for the storage space you
use, with no minimum fee
3 Amazon EFS is Scalable
• File systems can grow to petabyte scale

• Throughput and IOPS scale
automatically as file systems grow
• Consistent low latencies regardless of
file system size
• Support for thousands of concurrent
NFS connections
Highly Durable and Highly Available
• Designed to sustain AZ offline

conditions
• Resources aggregated across multiple
AZ’s
• Superior to traditional NAS availability
models
• Appropriate for Production / Tier 0
applications
Example use cases
Big Data Analytics

Media Workflow Processing
Web Serving
Content Management
Home Directories
EFS – Mounting
EFS
EC2 EC2 EC2 EC2 EC2 EC2
EFS DNS Name

availability-zone.file-system-id.efs.aws-region.amazonaws.com
Mount on machine
sudo mount -t nfs4 mount-target-DNS:/ ~/efs-mount-point
3
Object Stores
• Web accessible object store

• Pay for exactly what you use
• Highly durable (99.999999999% design)
• Limitlessly scalable
• Natively online
• Two flavors:
• Standard Storage - $0.023 * per GB / mo
• Standard – Infrequent Access Storage (min size 128KB) – $0.0125* per GB / mo + Data
retrieval cost
• * (US East (N Virginia) pricing)
• Parallel I/O for max speed (Multipart Upload, Ranged GETs)

• Resource-level IAM permissions
• Bucket Policies & ACLs
• Direct access through APIs
• Server Side Encryption
• Static Website Hosting
• Data Lifecycle Rules
• Amazon Athena – New
• Interactive Query Service that makes it easy to analyze data in Amazon S3
using standard SQL
Object Storage Tiering
Data tiering using S3 Life Cycle Policies
S3 Standard S3 - IA Glacier
Primary data File sync and share Deep/offline

Big Data Analytics Active Archive archives
Small objects Enterprise backup Tape vaulting
Temporary scratch Media transcoding replacement
space Geo-redundancy/DR WORM-compliant
data
Object Storage Use Cases
Temporary &
Cloud Big Data Content Primary
Small
Applications Analytics Distribution Data
Objects
S3
File Sync & Active Enterprise Media Disaster Recovery /

Share Archive Backup Transcoding Geo Redundancy
S3-IA
Deep / WORM
Tape Vaulting
Offline Compliant
Replacement
Archives Data
Glacier
Storage Tiered To Your Requirements
Durable
99.999999999%
“Hot” Data Starts at $0.023 / GB per month > 0K ≥ 0 Days

Active and/or
Temporary Data Available
S3
S3: 99.99%
L S3-IA: 99.9%
i
f Performant
Low Latency
“Warm” Data $0.0125 / GB per month ≥ 128K ≥ 30 Days
e High Throughput
Infrequently
Accessed Data c
$0.01/GB retrieval
S3-IA y Scalable
Elastic capacity
c No preset limits
l
e
“Cold” Data $0.004 / GB per month > 0K ≥ 90 Days
Archive and
Compliance Data
1-5 mins 3–5 hrs 5–12 hrs
Glacier Expedited Standard Bulk
$0.03 / GB $0.01 / GB $0.0025 / GB
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 3 new retrieval options
S3 Storage Management Features New!
S3 Object Tagging
manage and control access for Amazon S3 objects.
S3 Analytics, Storage Class Analysis

Analyze storage access patterns and transition the right data to the right storage class.
S3 Inventory
Simplify and speed up business workflows and big data jobs
S3 CloudWatch Metrics
Understand and improve the performance of your applications that use S3
Amazon Glacier
•Low-Cost Archival Storage
•Secure
• SSL & AES-256
•Durable
• Designed for 99.999999999% durability
•Optimized for data archiving and backup
• Suitable for RTO measured in hours
• Includes storage costs and retrieval costs
•Three retrieval options: Expedited, Standard, Bulk
•As little as $0.004 per GB/Month (US East pricing)
•Integrated with S3
Amazon CloudFront
Easy-to-use Content Delivery Network (CDN)
Pay-as-you-go pricing
Multiple origins: S3, EC2, on-premise
Worldwide network of 70+ edge locations and growing

• Video streaming
• Geo Restriction
• Custom SSL Certificates
• Dynamic Content
• Supports POST/PUT
4
On-Premises Storage
Integration
Storage Gateway hybrid storage solutions
Enables using standard storage protocols to access AWS storage services
Files Amazon S3
Volumes Amazon Glacier
Amazon EBS
Tapes AWS Storage
snapshots
Gateway
Amazon AWS Identity and Access

CloudWatch Management (IAM)
AWS AWS Key Management

CloudTrail Service (KMS)
Storage Gateway – Files, volumes, and tapes
File gateway NFS (v3 and v4.1) interface

On-premises file storage backed by Amazon S3 objects
Volume gateway iSCSI block interface

On-premises block storage backed by S3 with EBS snapshots
Tape gateway iSCSI virtual tape library interface

Virtual tape storage in Amazon S3 and Glacier with VTL management
Storage Gateway – Common capabilities
Standard storage protocols integrate with on-premises applications
Local caching for low-latency access to frequently used data
Efficient data transfer with buffering and bandwidth management
Native data storage in AWS
Stateless virtual appliance for resiliency
Integrated with AWS management and security
File gateway
On-premises file storage maintained as objects in Amazon S3
Customer Premises
NFS
v3 / HTTPS
v4.1
Application File S3 S3 Glacier
Server Gateway Standard Standard -
Infrequen
t Access
• Data stored and retrieved from your S3 buckets
• One-to-one mapping from files-to-objects
• File metadata stored in object metadata
• Bucket access managed by IAM role you own and manage
• Use S3 Lifecycle Policies, versioning, or CRR to manage data
Volume gateway
On-premises volume storage backed by Amazon S3 with EBS snapshots
Customer Premises
iSCSI HTTPS
Application Volume Storage Gateway Amazon EBS

bucket in snapshots
Server Gateway
Amazon S3
Block storage in S3 accessed via the volume gateway

Data compressed in-transit and at-rest
Backup on-premises volumes to EBS snapshots
Create on-premises volumes from EBS snapshots
Up to 1PB of total volume storage per gateway
Tape gateway
Virtual tape storage in Amazon S3 and Glacier with VTL management
Customer Premises
CHANGER
MEDIA
iSCSI HTTPS
Tape Virtual Archived Tapes

Backup
DRIVE
TAPE
Gateway Tapes stored in
Server stored in Amazon Glacier
Amazon S3
Virtual tape storage in S3 and Glacier accessed via tape gateway
Data compressed in-transit and at-rest
Unlimited virtual tape storage, with up to 1PB of tapes active in library
Supports leading backup applications:
Hybrid storage use cases with Storage Gateway
Enabling cloud workloads

Move data to AWS storage for Big Data, cloud bursting, or
migration
Backup, archive, and disaster recovery

Cost effective storage in AWS with local or cloud restore
Tiered cloud storage

Easily add AWS storage to your on-premises environment
Storage Gateway – Key Benefits
Seamless integration across standard storage protocols
Low-latency access
Durability, cost, and elasticity of AWS Storage services
Efficient data transfer
Data encryption
Integrated with AWS monitoring, management, and security
W!
Amazon Snowball & Snowball Edge NE
• Petabyte scale data transport

• Uses secure appliances
• Economic and fast
• Faster than Internet for significant data sets
• Import into S3
• HIPAA Compliant New
What is Snowball?
Petabyte scale data transport
Ruggedized Rain & dust
case resistant
“8.5G Impact”
E-ink shipping
label Tamper-resistant
case & electronics
All data encrypted

end-to-end
80 TB
10G network
How fast is Snowball?
• Less than 1 day to transfer 250TB via 5x10G connections with 5 Snowballs,
less than 1 week including shipping
• Number of days to transfer 250TB via the Internet at typical utilizations
Internet Connection Speed

Utilization 1Gbps 500Mbps 300Mbps 150Mbps
25% 95 190 316 632
50% 47 95 158 316
75% 32 63 105 211
Amazon Snowmobile NEW!
• Exabyte-scale data transfer service
• Each Snowmobile can transfer up to
100PB
• Connects to your network via
removable high-speed network switch
• Appears as network-attached data
store
• Data encrypted with 256-bit encryption
keys, managed through KMS
• Snowmobile driven back to AWS and
data is loaded into S3, Redshift, Glacier
Using Multiple Storage Options Together
EBS + S3: snapshots
S3 + EC2 Instance Store: caching
S3 + CloudFront: edge caching
S3 + Glacier: data lifecycle archiving
It’s all about
choicePerformance-oriented
Cost-oriented
Any Questions?

Storage in AWS

Încărcat de

Informații document

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Storage in AWS

Încărcat de

Drepturi de autor:

Formate disponibile

Storage on AWS

Amazon Elastic File System (EFS)

Amazon EC2 Instance Store (Ephemeral Volumes)

Amazon S3 (Simple Storage Service)

• Persistent block level storage for EC2

Network attached block device

Available in multiple types

Perform Lowest Cost Burstable Predictable

Use Infrequent Data Boot volumes I/O Intensive

Media Magnetic (HDD) SSD SSD

•EBS Optimized Instances

Price $.05/GB/Month $.10/GB/Month $.125/GB/Month

• Dedicated storage throughput

Perform Lowest Cost Burstable Predictable

Use Infrequent Data Boot volumes I/O Intensive

Media Magnetic (HDD) SSD SSD

Max 100 on average Baseline 3 Consistently performed

Price $.05/GB/Month $.10/GB/Month $.125/GB/Month

• Maximum attainable throughput to each volume now

at 500 MB/s read or write traffic (on instance that

• Capable of dramatically reducing your storage costs

Perform Lowest Cost Burstable Predictable

Use Infrequent Data Boot volumes I/O Intensive

Media Magnetic (HDD) SSD SSD

Max 100 on average Baseline 3 Consistently performed

Provisioned IOPS (SSD)

Price $.05/GB/Month $.10/GB/Month $.125/GB/Month

General Purpose (SSD)

Clone From EBS Snapshot

Snapshot EBS Snapshot

EC2 EC2 EC2 EBS Snapshot

Free with your EC2 instance

Local, direct attached resource

Consistent sequential reads and writes

Use only for non-persistent data

• Fully managed file system for EC2 instances

• File systems grow and shrink

• File systems can grow to petabyte scale

• Designed to sustain AZ offline

Big Data Analytics

EC2 EC2 EC2 EC2 EC2 EC2

EFS DNS Name

• Web accessible object store

• Parallel I/O for max speed (Multipart Upload, Ranged GETs)

Data tiering using S3 Life Cycle Policies

Primary data File sync and share Deep/offline

File Sync & Active Enterprise Media Disaster Recovery /

“Hot” Data Starts at $0.023 / GB per month > 0K ≥ 0 Days

S3 Analytics, Storage Class Analysis

Worldwide network of 70+ edge locations and growing

Volumes Amazon Glacier

Amazon AWS Identity and Access

AWS AWS Key Management

File gateway NFS (v3 and v4.1) interface

Volume gateway iSCSI block interface

Tape gateway iSCSI virtual tape library interface

Standard storage protocols integrate with on-premises applications

Local caching for low-latency access to frequently used data

Efficient data transfer with buffering and bandwidth management

Native data storage in AWS

Stateless virtual appliance for resiliency

Integrated with AWS management and security

Application Volume Storage Gateway Amazon EBS

Block storage in S3 accessed via the volume gateway