Christian Bolton MCA:Database, MCM:SQL Server Database Architect

christian@coeo.com http://coeo.com http://sqlblogcasts.com/blogs/christian

32-bit memory addressing Memtoleave 64-bit memory addressing Query/Workspace Memory Procedure Cache

232 = 4294967296 = 4GB Virtual address space

0x7FFFFFFF

Physical RAM Virtual Memory Memory Manager

0001 0010 0101 10 PTEs 0110 0000 1001 00 0101 0000 1101 00 0110 0111 1100 00 0001 0010 0111 11 0110 0000 1001 00 0101 0000 1101 00 0110 0111 1100 00

SQL Server Inst 1

0001 0010 0101 10 0110 0000 1001 00 0101 0000 1101 00 0110 0111 1100 00

0x00000000

2GB
0x7FFFFFFF

Page file

SQL Server Inst 2

0001 0010 0111 11 0110 0000 1001 00 0101 0000 1101 00 0110 0111 1100 00

0x00000000

2GB

0xFFFFFFFF

0xC0000000

0xBFFFFFFF

0x80000000 0x7FFFFFFF

> 2 GB virtual address space /3GB switch in boot.ini /LARGEADDRESSAWARE System PTEs Paged Pool Non_Paged Pool

0x00000000

Page Table Entries (PTEs)

4KB mapping between virtual and physical memory Memory\Free System Page Table Entries
Should be > 10,000

Symptoms
Server stops responding

/USERVA switch in Windows Server 2003

Gives less than 3GB to user mode Extra space kept in reserve for System PTEs Recommended values between 2900 and 3030

Paged Pool and Non-Paged Pool

Used to allocate system memory Non-Paged Pool used by drivers
Accessed at any Interrupt Request Level (H/W priority)

Maximum values determined at boot time Examples for Windows 2003 SP1:
System RAM 512MB 1024MB 1536MB 2048MB NonPaged Max 125MB 202MB 254MB 252MB Paged Max 184MB 168MB 352MB 352MB

Paged Pool and Non-Paged Pool Usage

Memory\Pool Paged Bytes Memory\Pool NonPaged Bytes Debugging tools or sysinternals Process Explorer linked to Microsofts symbols server to get actual max values

Windows Server 2008 and Vista have dynamic max values

> 2/3GB virtual address space PAE

Physical Address Extensions

0xFFFFFFFF

0xFFFFFFFF

0xC0000000 0xBFFFFFFF

0x80000000 0x7FFFFFFF

AWE
Address Windowing Extensions

Data pages only

0x00000000 0x00000000

0xFFFFFFFF 0001 0110 1001 0001 0110

1101 0001 1101 0101 0001

0x80000000 0x7FFFFFFF 0101 0000 1101 0001 0111

0101 1111 1001 0001 0000 0000 1100 1101 1101 1111

0x00000000

/PAE and /3GB together?

Not recommended /PAE increases PTE size from 4KB to 8KB Already reduced in size by /3GB Not supported 16GB+ RAM due to the kernel mode space required to support that much RAM

Area of SQL Server address space Outside the buffer pool Linked servers, XPs, COM objects Multi-Page Allocator (MPA)
<8KB handled by Single-Page Allocator

Error
"WARNING: Failed to reserve contiguous memory of Size= 65536."

MemToLeave = MemReserved + (NumThreads * StackSize)

MemReserved = 256 MB(unless a different value is specified with g at startup) NumThreads = the total number of worker threads configured StackSize = (0.5MB x86, 2MB x64, 4MB IA64)

Calculating Number of worker threads

Number of CPUs <= 4 processors 8 processors
16 processors 32 processors

32-bit computer 256 288

352 480

64-bit computer 512 576

704 960

select max_workers_count from sys.dm_os_sys_info

32-bit computer 256MB base value <= 4 cpu cores = 256 workers 256*0.5MB = 128MB Memtoleave = 384MB

Buffer Pool will never be more than physical RAM 1GB RAM VAS/memtoleave: 834MB!

264
=18,446,744,073,709,551,616 =17,179,869,184 TB = 16 EB

Virtual address space

8TB on x64 7TB on IA64

2 flavours
x64 created by AMD for server and desktop IA64 strictly patented by Intel and HP for servers

Windows XP x64 announced Intel responds with x64 compatible extensions called EM64T AMD renamed their implementation AMD64 Intel renamed theirs to Intel64 We now refer to x64 as meaning either platform

64-bit computer 256MB base value <= 4 cpu cores = 512 workers 512*2MB = 1024MB Memtoleave = 1280MB 7.99TB of VAS left!

By default SQL engine will not acquire all requested memory upon startup Configure Max Server Memory
Leave some headroom for the OS and for memory requests outside the buffer pool

Calculating Max Server Memory

Calculate worse case 2GB for the OS xGB for worker threads based on table below (2MB each on x64) 1GB for Multi-page allocations, linked servers etc 1-3GB for other applications
Number of CPUs <= 4 processors 8 processors 16 processors 32 processors 32-bit computer 256 288 352 480 64-bit computer 512 576 704 960

Scenario 8 cores, 32GB RAM 2GB for the OS 1GB for worker threads 1GB for Multi-page allocations, linked servers etc 1GB for backup program = 5GB Max Server Memory to 27GB

SQL Server responds to memory pressure If its not quick enough Windows can aggressively trim SQL Servers working set New message in SP2
A significant part of SQL Server process memory has been paged out. This may result in a performance degradation.

3 options to avoid this

Set max server memory appropriately so that windows never needs to trim SQL Server Upgrade to Windows Server 2008 Use AWE to allocate memory

Setting Max Server Memory is the safest

Why does AWE help?

Memory allocated using the AWE APIs cant be trimmed If service account for SQL Server has the privilege and its Enterprise Edition, SQL will use AWE

A user must have the Lock Pages in Memory privilege to use the AWE APIs

On 64-bit the AWE APIs are used to lock pages in the working set
AWE enabled option in sp_configure ignored on 64-bit

Only data cache thats locked, others can be trimmed

http://support.microsoft.com/kb/918483

Query Memory / Workspace Memory

Use by Hash and Sort operators to process data Memory requirements of individual operators added up to get total Query Memory requirement Non-overlapping operators can share Showplan with runtime information contains information about the actual grant

Allocating Query Memory

Total Query Memory dynamically managed between 25% and 75% on non-AWE buffer pool Higher if buffer pool not under pressure Perfmon: Memory Manager:Maximum Workspace Memory DBCC MEMORYSTATUS: Query Memory Objects

Allocating Query Memory

5% of Total Workspace Memory is reserved for small queries Max cost 3 Memory required < 5MB DBCC MEMORYSTATUS: Small Query Memory Objects No individual query gets a Grant or more than 20% of the Total Query Memory (after the 5% is reserved)

Query Memory Grant Queue

Organised as 5 queues based on query cost < 101, < 102, < 103, <104, >= 104 Each query put into appropriate queue based on cost FCFS policy within each queue Lets smaller queries be processed even as larger queries wait Query may timeout in Memory Grant Queue Default timeout 25 * cost of the query in seconds Controlled by the Query Wait configuration option

120 20

1GB Query Memory, 1 CPU

Memory Grants Pending < 10 < 102 < 103 < 104 < 104

Memory Grants Outstanding

140 60

120

150

180 60 80

Free Used

120

80

120

100

790 910 890

Query Memory Diagnostics

Perfmon: Memory Manager
Granted Workspace Memory Maximum Workspace Memory Memory Grants Pending Memory Grants Outstanding

DBCC MEMORYSTATUS
Query Memory Objects Small Query Memory Objects

sys.dm_os_wait_stats RESOURCE_SEMAPHORE

Procedure Cache
Allocated from the buffer pool Query Plans Stored in Procedure Cache
Compiled plan Executable plan Parse tree Extended stored procedure sys.dm_exec_cached_plans to see the entries in the

cache
DBCC MEMORYSTATUS to see the size of the cache

Where does cost from?