Sunteți pe pagina 1din 42

Virtualization and Consolidation

Considerations for Microsoft


SQL Server 2008
Gopal Ashok
DAT305
Program Manager
Microsoft Corp

Agenda
SQL Server Consolidation
Virtualization & Microsoft Hyper-V
Architecture
Consolidation Performance
High Availability & Manageability offerings
Achieving scalability with Virtualization

Best Practices and Recommendations


SQL Relational Workload
A Case Study

Key Takeaways

Forces Driving Consolidation


Reduce Cap-Ex

rd
a
H

re
a
w

m
co

er o
b
Num

t in
u
p

p
ca

ity
c
a

pps
a
e
s
aba
t
a
fd

DBAs
Number of

1990

2000

2010

Underutiliz
Underutiliz
ed
ed
hardware
hardware

Overburden
Overburden
ed
ed
Administrat
Administrat
ors
ors

Upgrade & standardize


to fewer hardware
Reduce space, power
and thermal needs

Reduce Op-Ex
Improve Hardware
Utilization Efficiency
Improve Management
Efficiency
Lowered cost and
complexity for High
Availability

Infrastructure Agility
Load Balancing &
Dynamic Provisioning
Standardization of
services

Currently a variety of consolidation approaches exist


and are utilized.

Typically, as isolation goes up, density goes down and


operation cost goes up.

Virtual
IT Managed
Environment Machines

Instances

Sales_1

Databases

Consolidate_1

Marketing_1
Online_Sale
s
ERP_10
ERP_10

DB_1
DB_2
DB_3

Schemas

Costs
Lower Costs
Density, Lower
Higher
Higher Density,

Higher
Higher Isolation,
Isolation, Higher
Higher Costs
Costs

SQL Server Consolidation

Database level consolidation

Multiple databases are collapsed into a single instance

Common security, manageability and compatibility


models required

Might require changes to existing applications and scripts

Lower manageability costs

Better resource isolation and allocation capabilities


through Resource Governor
Virtual
Databases
Instances
Machines
Sales_1

Consolidate_1

Marketing_1
Online_Sale
s
ERP_10
ERP_10

DB_1
DB_2
DB_3

Lower
Density, Lower
Higher
Higher Density,
Costs
Costs

Higher
Higher Isolation,
Isolation, Higher
Higher Costs
Costs

SQL Server Consolidation Options

Instance level consolidation


Multiple Instances running on one system

Full schema and security isolation

Partial system resource & management isolation

Potential conflicts in namespace, resources and system


roles

IO, System Memory and CPU are typical density limiters


Virtual
Machines

Instances

Sales_1

Databases

Consolidate_1

Marketing_1
Online_Sale
s
ERP_10
ERP_10

DB_1
DB_2
DB_3

Lower
Density, Lower
Higher
Higher Density,
Costs
Costs

Higher
Higher Isolation,
Isolation, Higher
Higher Costs
Costs

SQL Server Consolidation Options

Virtual Machine consolidation

Strong isolation between applications

Ease of capturing and moving execution loads

Out of the box High Availability configuration

Flexible storage management

Fewer systems, but just as many OS images to manage

Increased resource usage

Virtual
Machines

Instances

Sales_1

Databases

Consolidate_1

Marketing_1
Online_Sale
s
ERP_10
ERP_10

DB_1
DB_2
DB_3

Costs
Lower Costs
Density, Lower
Higher
Higher Density,

Higher
Higher Isolation,
Isolation, Higher
Higher Costs
Costs

SQL Server Consolidation Options

Choosing the Right Consolidation


Evaluate key parameters for your environment
Approach
Reduce
Cap-Ex

Reduce
Op-Ex

Agile IT

Isolation between apps

Security isolation

Predictable Performance : Resource isolation

HA : Failure isolation

Density of apps

Performance : Resource utilization efficiency

Manageability Impact

HA : mitigating single point of failure

Time to Market

How long does it take to consolidate?

Can my solution Scale?

Agenda
SQL Server Consolidation
Virtualization & Microsoft Hyper-V
Architecture
Consolidation Performance
High Availability & Manageability offerings
Achieving scalability with Virtualization

Best Practices and Recommendations


SQL Relational Workload
A Case Study

Key Takeaways

Hyper-V Architectural Overview


Root
Partition

Windows Server 2008


R2
Virtual
Service
Providers

I/O
Stack
s

Child
Partition
Enlightened OS

NonEnlightened OS

Server

Server
Applications
Child
Partition

Windows Server
2008
I/O Stacks
TCP/IP , File
systems

Fast Path
Virtual
Service
Clients

Drivers

VMBus

Child
Partition

VMBus

I/O Stacks
TCP/IP , File
systems

Device
Emulation

Device
Emulation

Drivers

Drivers

Hypervisor
Devices

Processo
rs
Hardwa

re

User
Mode

Memory

Kernel
Mode

Hypervisor
Mode

Agenda
SQL Server Consolidation
Virtualization & Microsoft Hyper-V
Architecture
Consolidation Performance
High Availability & Manageability offerings
Achieving scalability with Virtualization

Best Practices and Recommendations


SQL Relational Workload
Analysis Services Workload
A Case Study

Key Takeaways

Monitoring Performance: CPU


Terminology
Logical Processor: One logical
computing engine in the OS,
application and driver view
Virtual Processor: Virtual logical
processor (upto 1:8 over commit of
LP)

% Processor Time
% Processor Time counters in
either guest or root may not be
accurate

Hyper-V Processor Counters


The best way to get true measure
of CPU utilization
Hyper-V Logical Processor: Total
CPU time across entire server
Hyper-V Virtual Processor: CPU
time for each guest virtual processor

Monitoring Performance: Storage


Terminology

Configura
tion

Considerations

Disk Offline at Root


Partition

Guest VM w/

Use Physical disk counters within root

Passthrough

partition to monitor I/O of passthrough

VHD (Virtual Hard


Disk)

Disks

Passthrough Disk:

Fixed Size VHD :


Space allocated
statically

disks

Guest VM w/

Use Logical or physical disk counters

VHD

within guest VM to monitor IO rates of a


VHD
Disk counters at the root partition

Dynamic VHD :
Expand on demand

provide aggregate IO of all VHDs


hosted on the underlying
partition/volume
Either

Very little difference in the values

configuration

reported by the counters from the root


partition with those within guest VM
Slightly higher latency values (Avg.
Disk/sec Read and Write) observed
within the guest VM

SQL Server Consolidation Performance: Native


vs.Configuration
Virtual
Results
Workload : Stock trading application

Relative Throughput = Batches/sec/


Hardware:
%Processor
PowerEdge R900 Intel Xeon 2.4GHz,
Same throughput attainable
4 cores x 4 CPU = Total 16 cores
There is more CPU overhead with hyper-v
Hitachi Data Systems AMS1000
enabled or when running within a VM
Virtual Machines:
Passthrough Performance ~5% Better
4 Virtual Processors, 14GB RAM, 12 GB for
than fixed size VHDs
SQL
Passthrough and Fixed-Size VHDs (2 Data, 1 Native Instances and Virtual Instances
achieves
same level of scalability
Log) and Scalability Comparison between Native and Virtual
Performance
Instancesthe
: 16core 64GB RAM 12GB Per SQL Instance

2500

60
55
50

2000

45

Batches/sec_Na
tive
Batches/sec_VM

40
1500

35
30
25

1000

20
15

500

10
5
0

0
1Instances

2Instances

3Instances

4Instances

Batches/sec/Inst
_Native
Batches/sec/Inst
_VM
%
Processor_Nativ
e
% Processor_VM

SQL Server Consolidation Performance: Native vs. Virtual


(overcommit)
Configuration

Results

Workload : Stock trading application


Hardware:

Relative Throughput = Batches/sec/

%Processor
Same throughput attainable with CPU

PowerEdge R900 Intel Xeon 2.4GHz,


overcommit
4 cores x 2 CPU = Total 8 cores
Additional overhead with over subscribe
Hitachi Data Systems AMS1000
logical processors
Virtual Machines:
Taking into consideration of the additional
4 Virtual Processors, 14GB RAM, 12 GB
overhead for capacity planning
Native Instances and Virtual Instances achieves
for SQL
the same level of scalability
Passthrough and Fixed-Size VHDs (2 Data,
1 Log)
Performance
and Scalability Comparison between Native and Virtual Instances: 8core 64GB RAM 12GB Per SQL Instance
110

2000

100

1800

90

1600

80

1400

70

1200

60

1000

50

800

40

Batches/sec_Na
tive
Batches/sec_VM
Batches/sec/Inst
_Native
Batches/sec/Inst
_VM

20

400

%
Processor_Nativ
e

10

200

% Processor_VM

600

30

0
1 Inst

2 Inst

3 Inst

4 Inst

Passthrough vs. Fixed Size VHD


VHDs

on Shared Storage vs. Dedicated Spindles using Passthrough Disks

Measuring
VHDs

average reads per second vs. latency

on shared disks has slight latency overhead and less throughput

Graph bars = Reads/sec

Lines = Avg. Disk/sec Read (.001 = 1 ms)


Dedicated Pass-through vs. VHDs on Shared Disks
Total IO/s per Second and Disk Latency

8,000
Total
Read IOs per Second
6,000
4,000

0.01
0.01
0.01
0

2,000

Total
Reads/sec
(Dedicated
LUNs)

Transaction Response Time


V
Hardware:
HP DL785 (32 core)
Hitachi Data Systems AMS2500 Storage
Virtual Machines: 4 virtual processors and 7 GB
RAM per virtual machine; Fixed size VHD

Results:

Transaction response time comparable


sub seconds application response time
When system under stress, VM appears to
have better scalability
Test your own workload, may scale
differently

% P ro c e s s o r t im e

OS: Microsoft Windows Server 2008 R2 Hyper-

R e s p o n s e T im e ( s e c )

Configuration:

Response Time_VM

Response Time_Native

%Processor_VM

%Processor_Native

Agenda
SQL Server Consolidation
Virtualization & Microsoft Hyper-V
Architecture
Consolidation Performance
High Availability & Manageability offerings
Achieving scalability with Virtualization

Best Practices and Recommendations


SQL Relational Workload
A Case Study

Key Takeaways

High Availability with Guest Clustering


Using
iSCSI
Scenario
Description:
Improve high availability with a
combination of guest clustering and host
clustering
Only iSCSI is supported for guest
clustering
iSCSI Initiator runs within the VM
allowing storage to be fully visible to the
VM
Storage is fully visible to the guest
cluster and enables high-availability of
services and applications in the virtual
layer
Support the use of multiple redundant
paths using Microsoft Multipath IO (MPIO)
or MCS (multiple connections per
from tolerance
within the VM
session)
Provide fault
both at
application and host level
All applications can run in the context of
the VM
Management efficiency based on SQL
Server and System Center
management tools
Backup applications have full visibility to
data within the application context
Storage providers such as MPIO, VDS,

Virtualization
Benefits:

VM

Guest
Cluste
r

Guest
Cluste
r

Redundant
Paths to storage

Shared
Storage
iSCSI

High Availability with Live Migration


Scenario Description:

Manage high availability with multipathing


and live migration for planned downtime
situations, such as hardware and software
maintenance
Migrating individual virtual machines
(VMs) to other hosts within a cluster by
using Cluster Shared Volume (in Windows
Server 2008 R2)
Use Microsoft System Center Virtual
Machine Manager for migrations. System
Center VMM can perform host
compatibility checks before migrations
and manage multiple Live Migrations with
queues.
Nodes in cluster can be active-active
Ensure there is enough CPU capacity for
the failover nodes in cluster
No loss of service during migration
with live migration.
Improve availability with less
complexity
Load balancing VMs across physical
machines as needed
Better server utilization due to
consolidation

Virtualization
Benefits:

Easier management through System

Consolidation increases the


importance of HA, since there is a high
cost to single system failure
VM

1
1

Live
Migration

Host cluster

Shared
Storage
iSCSI, SAS,
Fibre

Creating and managing Virtual Machines

Maximize Resources
Centralized virtual machine deployment and
management for Hyper-V, Virtual Server,
and VMware ESX servers
Intelligent placement of Virtual Machines
Fast and reliable P2V and V2V conversion
Comprehensive application and service-level
monitoring with Operations Manager
Integrated Performance and Resource
Optmization (PRO) of VMs

Increase Agility
Rapid provisioning of new virtual machines
with templates
Centralized library of infrastructure
components
Leverage and extend existing storage
infrastructure and clusters
Allow for delegated management and
access of VMs

Leverage Skills
Familiar interface, common foundation ,
powershell scripting

Agenda
SQL Server Consolidation

Virtualization & Microsoft Hyper-V


Architecture
Consolidation Performance
High Availability & Manageability offerings
Achieving scalability with Virtualization

Best Practices and Recommendations


SQL Relational Workload
A Case Study

Key Takeaways

Consolidation Hardware NUMA & 64bit


Memory
Memory

SMP

Front side bus contention increases w/ higher


#CPUs

CPU
CPU 0
0

CPU
CPU 1
1

CPU
CPU 2
2

CPU
CPU 3
3

CPU
CPU 4
4

NUMA
NUMA NODE
NODE 0
0

NUMA

CPU
CPU 6
6

CPU
CPU 7
7

NUMA
NUMA NODE
NODE 1
1

CPU
CPU 0
0

CPU
CPU 2
2

CPU
CPU 1
1

CPU
CPU 3
3

Local Memory Memory


Memory
Access

CPU
CPU 5
5

Foreign
Memory
Access
4x local

CPU
CPU 4
4

CPU
CPU 6
6

CPU
CPU 5
5

CPU
CPU 7
7
Memory
Memory

NUMA Node isolation presents inherent & significant advantages to software


that is designed to take advantage of locality
Microsoft Hyper-V localizes VM resource utilization to a NUMA boundary
64bit allows significantly more addressable space than 32bit

Consolidation Hardware SLAT

With Virtualization an additional level of mapping is required in address


translation
Second Level Address Translation (SLAT) - 2 nd generation virtualization
technology in Intel VT-x with EPT and AMD V with NPT chips accelerate
VM performance to be almost on par with bare metal for common workloads
Continuing innovations in hardware assists Device and IO virtualization

The Virtual Machine view


Virtual
Virtual Machine
Machine 1
1

Guest
Guest Physical
Physical

Guest
Memory
view
Guest Physical
Physical
Memory
view
Pages
Guest
Pages
Guest Physical
Physical
Guest
Memory
view
Guest Physical
Physical
Memory
view
Pages
Guest
Pages
Guest Physical
Physical

view
view Memory
Guest
Guest Physical
Physical
Memory
Pages
Pages

Virtual
Virtual Machine
Machine 3
3
Virtual
Virtual Machine
Machine 1
1

Host
Host Physical
Physical // real
real view
view
Physical
Physical Memory
Memory Pages
Pages

Hyper
Hyper Visor
Visor
Operating
Operating System
System

SQL Server Performance: SLAT Impact


Configuration:

Results:

OS: Microsoft Windows Server 2008 R2 Hyper-V


Hardware:

Increased throughput with consolidation


Near linear scale in throughput with no

HP DL585 (16 core) with SLAT


HP EVA 8000 storage
Virtual Machines: 4 virtual processors and 7 GB RAM
per virtual machine; Fixed size VHD

CPU over-commit
Improved performance with Windows
Server 2008 R2 and SLAT processor
architecture

Virtual Instances Scalability


% CPU
80

Almost
Almost Linear
Linear Scale
Scale
No
CPU
over-commit
No CPU over-commit

70

Throughput
(Batch
requests/sec)
3500

CPU over-commit

3000

60

2500

50

Heavy
Load

2000

40

30

1500
1000

20

500

10
0

Modera
te
Load
Low
Load

0
1VM

2VM
Batch req/sec

3VM

4VM
%CPU

5VM

6VM

7VM

Relative Throughput

8VM
Relative Throughput for
Windows Server 2008

Consolidation Hardware Snoop Filtering


NUMA
NUMA NODE
NODE 0
0

NUMA
NUMA NODE
NODE 1
1

CPU
CPU 0
0

CPU
CPU 2
2

CPU
CPU 4
4

CPU
CPU 6
6

CPU
CPU 1
1

CPU
CPU 3
3

CPU
CPU 5
5

CPU
CPU 7
7

Memory
Memory

C0
C1
C2
C7 MCH1

C3

MCH0

C4

C5

Memory

Fetch addr (first


C6
time)
1. Issue Fetch to
Memory
Controller Hub
(MCH)
2. Snoop local
caches
3. Caches
respond
4. Snoop foreign
caches
5. Caches
respond
6. Get data from
memory
controller

Memory
Memory

Consolidation Hardware Snoop Filtering


NUMA
NUMA NODE
NODE 0
0

NUMA
NUMA NODE
NODE 1
1

CPU
CPU 0
0

CPU
CPU 2
2

CPU
CPU 4
4

CPU
CPU 6
6

CPU
CPU 1
1

CPU
CPU 3
3

CPU
CPU 5
5

CPU
CPU 7
7

Memory
Memory

Snoop Filter
reduces
internode
traffic
Fetch addr (first

C3

MCH0

C4

C5

Memory

time)
1. Issue Fetch to
Memory
Controller Hub
(MCH)
2. Check local
snoop filter
3. Check foreign
snoop filter
4. Get data from
memory
controller

C0
C1
C2
C6
C7 MCH1

Memory
Memory

SQL Server Performance : Snoop Filter


Results:
Impact
Configuration:

OS: Microsoft Windows Server 2008 R2 Hyper-V


Hardware:

HP DL785 with and without snoop filter support


(Shanghai vs. Istanbul)
Hitachi Data Systems AMS2500 Storage

Drop-in compatibility of Istanbul processors with


existing infrastructure
AMD HyperTransport Assist (Intel snoop filter)
feature huge benefit to SQL workload

Virtual Machines: 4 virtual processors and 7 GB RAM

per virtual machine; Fixed size VHD

keep cache coherency traffic between the two


sockets from appearing on the external bus
~50% performance improvement

Comparing Istanbul vs. Shanghai


100
90

90.28

93.12

91.70

5000

94.05

4500

82.12

80
70
60

4000

55%
57.84
42%

60.15

57.84

59.79

60.67

3500
3000

50

2500

40

2000

30

1500

20

1000

10

500

0
VM1

VM2

VM4

VM6

VM8

Batches/sec_Ist
anbul
Batches/sec_Sh
anghai
%Processor
Time_Istanbul
%Processor
Time_Shanghai
Relative
Throughput_Ista
nbul

Agenda
SQL Server Consolidation

Virtualization & Microsoft Hyper-V


Architecture
Consolidation Performance
High Availability & Manageability offerings
Achieving scalability with Virtualization

Best Practices and Recommendations


SQL Relational Workload
A Case Study

Key Takeaways

Best Practices and Recommendations


Running SQL Server workloads within Hyper-V guest VMs is a
good option for production environment
When compared against native the same throughput can be
achieved within a guest VM at a cost of slightly increased CPU
utilization

Hyper-V benefits from newer generation of processor


architecture (SLAT, Snoop filter)
Assuming limitations of Guest VM meet requirements of the
workload.
Proper hardware sizing is critical to SQL Server performance
Test/Monitor your workloads

CPU Resources Over Commit on case by case basis for Higher


Density
May introduce noticeable performance overhead when all
workloads are busy

Best Practices and Recommendations


Integrated Component : Enlightenments
Better IO performance
Reduce memory access overhead

Passthrough and Fixed Size VHD for Better I/O


performance
IO Performance Impact is minimal
SQL IO performance and sizing recommendations apply
Dynamic VHD not recommended for SQL Server deployments

Proper sizing of memory capacity.


Memory is allocated for VMs in a static fashion and can only
be modified when a guest is offline

CPU Affinity Not Supported


Network intensive workload might experience more
overhead

Case Study MSIT SQL Consolidation


Microsoft IT Infrastructure Overview
Pre-Consolidation State

~2,700 Applications in MSIT Portfolio


~4797 SQL Server Instances
~100,000 databases
~20% end-of-life hosts/year
~10% CPU utilization across hosts

Consolidation Approach

Microsoft IT evaluated database, instance and host based


consolidation

Resource Management Approach

Resource Management effectiveness considered critical issue


Instance based would utilize WSRM
Hyper-V allows explicit allocation of CPU and IO channels

Case Study MSIT SQL Consolidation


Microsoft IT Consolidation Conclusions
Consolidation Strategy

Host Consolidation utilizing Hyper-V


Target of 6 to 1 consolidation ratio
Fixed Virtual Drives (VHDs) over Dynamic and Pass
Through

Consolidation Approach

Decision starting point was instance based consolidation


Evaluation led to decision change: Hyper-V based host
consolidation

WSRM vs. Hyper-V

Microsoft IT evaluated WSRM vs. Hyper-V & ultimately


chose Hyper-V

Case Study - Microsoft IT Consolidation


Annual Operating Cost at 6:1 Consolidation
$18,000,000
$16,000,000
$14,000,000
$12,000,000
$10,000,000
$8,000,000
$6,000,000
$4,000,000
$2,000,000
$0

Legacy

Consolidated

Annual Recycle Cost at 6:1 Consolidation

$50,000
$40,000
$30,000
$20,000
$10,000
$0

Legacy

Consolidated

Power and Cooling at 6:1 Consolidation


4,000,000
3,500,000
3,000,000
2,500,000
2,000,000
1,500,000
1,000,000
500,000
0

Legacy

Consolidated

Key Takeaways
No One-size Solution for Consolidation
Hyper-V is a solid platform for SQL Server, both for scalability and
performance
Understand performance and functional needs of workloads being
consolidated
If possible, avoid consolidation of heterogeneous workloads on the
same Windows Server

I/O design is very significant


Especially tempdb and log files
Monitor I/O latency and ensure healthy I/O performance
Avoid spindle sharing with other servers/applications when possible
Follow Best Practices for storage

Ensure No/minimal Contention for Memory Resources


Determine and set upper memory limits for SQL Server and SSAS
(both single and multiple instances)

Take into account memory needed for OS, other applications


Understand and test the impact of SSIS packages being run locally
Ensure enough memory for CLR procedures

Key Takeaways
Resource Management
Hyper-V CPU management tools to manage CPU resources at VM
Level
WSRM to manage multiple instances CPU resource
Resource Governor to manage within an instance

SQL Server Analysis Services


Determine and set upper memory limits for SQL Server and SSAS
(both single and multiple instances)
SSAS consolidation enables better utilization of hardware (new or
existing)
SSAS consolidation requires planning to resolve resource contention
and availability / single point of failure issues

Taking advantage of New Processor Architecture


SLAT
Snoop Filter

Share your Lessons Learned!

SQL Solution Accelerators


Infrastructure Planning
and Design Guide
Outline SQL Server 2008
infrastructure design decisions
and ensure that business and
technical requirements are met
Offer easy-to-follow steps to
design architecture including
decision flows
Download at
www.microsoft.com/IPD

Microsoft Assessment
and Planning Toolkit
Identify SQL instances
automatically
Assess Hyper-V virtualization
candidates for server
consolidation
Download at
www.microsoft.com/MAP

question &
answer

Resources
www.microsoft.com/teched

www.microsoft.com/learning

Sessions On-Demand &


Community

Microsoft Certification & Training


Resources

http://microsoft.com/techne
t

http://microsoft.com/ms
dn

Resources for IT Professionals

Resources for Developers

References
Running SQL 2008 in Hyper-V Environment
http://sqlcat.com/whitepapers/archive/2008/10/03/running-sql-server-2008-ina-hyper-v-environment-best-practices-and-performancerecommendations.aspx

Green IT in Practices: SQL Server Consolidation in Microsoft IT


http://www.msarchitecturejournal.com/pdf/Journal18.pdfhttp://msdn.microsoft.com/enus/architecture/dd393309.aspx

Support Policies of SQL Server in virtualized environments.


http://support.microsoft.com/?id=956893
http://blogs.msdn.com/psssql/archive/2008/10/08/sql-server-support-in-ahardware-virtualization-environment.aspx

Windows Virtualization Validation Program


http://windowsservercatalog.com/svvp.aspx?svvppage=svvp.htm

Windows Server Hyper-V site


http://www.microsoft.com/windowsserver2008/en/us/virtualization-consolidati
on.aspx

Hyper-V Technet center


http://technet2.microsoft.com/windowsserver2008/en/servermanager/virtuali
zation.mspx

SQL Server 2008 Business Value Calculator:

www.moresqlserver.com

Complete an
evaluation on
CommNet and enter
to win an Xbox 360
Elite!

2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S.
and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must
respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information
provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

S-ar putea să vă placă și