Peter Heumel

Automatic Parallel Execution

in Practice

Nürnberg, DOAG 2016

 This presentation will be about

• A feature which automates Parallel Query

• Why the configuration is difficult
• Two other features for Parallel Query
• A parameter which enables all three features
• Why it is important that your application is suitable for
this feature combination

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 1 of 37
 History of Parallel Query

Parallel Query Automatic Parallel Query

• Introduced in Oracle 7.1
• To speedup queries
• Needs static parallel degree
• Introduced in Oracle 11.2
• Activated by
PARALLEL_DEGREE_POLICY=AUTO
• Dynamic parallel degree

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 2 of 37
 Automatic degree of parallelism (auto DOP)

PARALLEL_DEGREE_POLICY=AUTO

1. Find best serial plan (dop=1)

2. If estimated time < PARALLEL_MIN_TIME_THRESHOLD
then execute serial plan
3. Else calculate a parallel degree n based on query
properties and limit it by PARALLEL_DEGREE_LIMIT
4. Re-run plan calculation but with dop=n
5. Execute the best plan from 4.

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 3 of 37
 Cost calculation and access path selection

We will look at three examples

1. Serial plan (without DOP)
2. Parallel plan with manual DOP
3. Parallel plan with auto DOP

We will see
• Why the cost-based optimizer (CBO) needs a
concrete parallel degree
• How automatic parallel query works

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 4 of 37
 Cost calculation for parallel queries

Following Amdahl's law

cost = cs + cp / dop

Costs of the access path,

• Which cannot be parallelized: cs
• Which can be parallelized: cp
• Degree of parallelism: dop

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 5 of 37
 1. Serial plan (without DOP)

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 6 of 37
 2. Parallel plan with manual DOP

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 7 of 37
 3. Parallel plan with auto DOP

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 8 of 37
 Questions

• How is the automatic parallel degree calculated?

• What is the optimization goal?
• Can the values be adjusted if they do not fit to my
application?

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 9 of 37
 Disappointing answers

• The Oracle documentation contains virtually no

information about how the parallel degree is
calculated or what the actual optimization goal is
• A manual configuration is
– not implemented for Oracle 11.2
– implemented in Oracle 12.1 – but it does not work
• At least PARALLEL_DEGREE_LEVEL (new in 12.1)
allows some adjustment

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 10 of 37
 IO calibration – on the way to a workaround

For Oracle versions 11.2.0.2 until 11.2.0.4: IO

calibration is needed, without:
---------------------------------------------------------
| Id | Operation | Name | Rows |
---------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 |
| 1 | TABLE ACCESS BY INDEX ROWID| EMP | 1 |
|* 2 | INDEX RANGE SCAN | X_EMP_01 | 1 |
---------------------------------------------------------
Note
-----
- automatic DOP: skipped because of IO calibrate
statistics are missing

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 11 of 37
 IO calibration – on the way to a workaround

IO calibration run for the database

SQL> var lat number

SQL> var iops number
SQL> var mbps number
SQL> exec DBMS_RESOURCE_MANAGER.CALIBRATE_IO
(2,10,:iops,:mbps,:lat);

PL/SQL procedure successfully completed.

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 12 of 37
 IO calibration – on the way to a workaround

IO calibration result is stored in the dictionary

SQL> select * from SYS.RESOURCE_IO_CALIBRATE$;

MAX_IOPS MAX_MBPS MAX_PMBPS LATENCY NUM_DISKS

---------- ---------- ---------- ---------- ----------
34963 813 454 4 2

• For auto DOP crucial: MAX_PMBPS (scan rate)

• No restart of the database is needed
• Oracle 11.2.0.1 (and 12.1.0.2) uses a default of
200 MB/s for the scan rate

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 13 of 37
 Workaround in Oracle 11.2 (manual config)

1. Adjust the value of MAX_PMBPS:

delete from SYS.RESOURCE_IO_CALIBRATE$;

insert into SYS.RESOURCE_IO_CALIBRATE$
values
(systimestamp,systimestamp,0,0,200,0,0);
commit;

2. Restart the database

MOS Note: “Automatic Degree of Parallelism in 11.2.0.2” (Doc ID 1269321.1)

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 14 of 37
 Which value for MAX_PMBPS?

Iterate until the value fits

1. Choose a value (for example 200 MB/s)

2. Restart the database
3. Test with your application
– If auto DOP is too small then decrease the value
– If auto DOP is too big then increase the value

Risks: an „unintended“ IO calibration run

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 15 of 37
 Manual configuration in 12.1

• IO calibration and workaround for 11.2 works

• But you should switch to the new interface for manual
auto DOP configuration:

DBMS_STATS.SET_PROCESSING_RATE
– No restart is necessary
– The values set can be viewed in
V$OPTIMIZER_PROCESSING_RATE

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 16 of 37
 Manual configuration in 12.1 - example

Set scan rate of 30 MB/s

exec DBMS_STATS.SET_PROCESSING_RATE('IO_ACCESS',30);

Query the setting

SELECT OPERATION_NAME,
MANUAL_VALUE,
CALIBRATION_VALUE,
DEFAULT_VALUE
FROM V$OPTIMIZER_PROCESSING_RATE
WHERE OPERATION_NAME = 'IO_ACCESS‘;

OPERATION_NAME MANUAL_VAL CALIBRATIO DEFAULT_VA

-------------------- ---------- ---------- ----------
IO_ACCESS 30.00000 200.00000

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 17 of 37
 Manual configuration in 12.1 – bug

Bug 21876851: manual values will be ignored, workaround:

alter system set “_optimizer_proc_rate_source”='MANUAL‘;

Value Scan rate (IO_ACCESS)

DEFAULT 200 MB/s or (if available)
(default) SYS.RESOURCE_IO_CALIBRATE$.MAX_PMBPS
MANUAL Value set by
DBMS_STATS.SET_PROCESSING_RATE('IO_ACCESS',<n>)

OPERATION_NAME MANUAL_VAL CALIBRATIO DEFAULT_VA

-------------------- ---------- ---------- ----------
IO_ACCESS 30.00000 30.00000

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 18 of 37
 Parallel degree calculation (Oracle 11.2)

dop = floor( object_size / (scanrate*ptu) )

Object size
• object_size = blocks * block_size
• Optimizer statistics (DBA_TABLES.BLOCKS) or dynamic sampling
Scanrate
• 200 MB/s (default)
• SYS.RESOURCE_IO_CALIBRATE$.MAX_PMBPS
Ptu (= 10 sec)

Example: table 6000 MB, scan rate 200 MB/s, then auto DOP = 3.

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 19 of 37
 Optimization goal

dop = floor( object_size / (scanrate*ptu) )

• If the scan rate is set to the real value, which a PX

slave is able to process, then the goal is the ptu (= a
running time of 10 seconds)
• This fits with the default for
PARALLEL_MIN_TIME_THRESOLD which is also 10
seconds (AUTO)

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 20 of 37
 Visualization of the optimization goal
Data generation
• Increase the object size by 100 blocks per step
DBMS_STATS.SET_TABLE_STATS
('SCOTT','T_EMP_20',numblks=>&1);
Measure
• The calculated dop
• The estimated running time
select avg(sal) from t_emp_20
------------------------------------------------------
| Id | Operation | Name | Time |
------------------------------------------------------
| 0 | SELECT STATEMENT | | 00:00:13 |
Note
-----
- automatic DOP: Computed Degree of Parallelism is 2

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 21 of 37
 Visualization of the optimization goal

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 22 of 37
 In-memory parallel query (IMPQ)
Direct read
• Introduced in Oracle 7.1.5
• Used by Parallel Query
• Does not use buffer cache
Parallel Query is now suitable for queries on objects which fit
into the buffer cache
Last Update November 11, 2016
In-memory parallel query
• Introduced in Oracle 11.2
• Activated by
PARALLEL_DEGREE_POLICY=AUTO
• Caching is only considered for
objects smaller than 80% of
the buffer cache
• Adjustment (80%) by
_parallel_cluster_cache_pct
Automatic Parallel Execution in Practice
Page 23 of 37
 In-memory parallel query and RAC

On Oracle RAC caching has some optimizations to

avoid caching blocks twice, thrice, ...
• Blocks of an object are mapped to exactly one instance in the
RAC
• PX slaves are distributed on the RAC in such a way that they
always have to read specific blocks on the same instance
(from the local cache)
• The mapping depends on the parallel degree
• If caching is planned the execution plan (12c) notes
- parallel scans affinitized for buffer cache

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 24 of 37
 In-memory parallel query - considerations

Careful consideration is needed

• In-memory parallel query and smart scans (Exadata)
• In-memory parallel query and in-memory column store (In-
Memory Option)

To disable in-memory parallel query set

_parallel_cluster_cache_policy=adaptive

For bugs see

MOS Note: Init.ora Parameter "PARALLEL_DEGREE_POLICY" Reference
Note (Doc ID 1216277.1)
Last Update November 11, 2016 Automatic Parallel Execution in Practice
Page 25 of 37
 Parallel Statement Queuing

Parallel Statement Queuing

• Queuing for Parallel Query
• Introduced in Oracle 11.2
• Activated by PARALLEL_DEGREE_POLICY=AUTO

If not enough PX servers are available statements will be

queued.

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 26 of 37
 Parallel Statement Queuing

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 27 of 37
 Parallel Statement Queuing

PARALLEL_DEGREE_POLICY=AUTO

For queries with a parallel plan (dop=n)

1. Check how many PX servers are active
(GV$RSRC_CONSUMER_GROUP.CURRENT_PQ_SERVERS_ACTIVE)
2. If the number is < PARALLEL_SERVERS_TARGET then run
the statement with the requested dop
3. Else add the statement to a queue (FIFO)
4. As soon the number of active PX servers falls below
PARALLEL_SERVERS_TARGET, run the next statement
from the queue with its requested dop

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 28 of 37
 Parallel Statement Queuing

Advantages (according to Oracle)

• Statements will be run with their optimal parallel degree
• Parallel queries cannot consume too much resources
(operational data warehouse)

Observations
• Better avarage response times possible for single queries
• Actually it is session not statement queuing
• Spreadsheet applications are problematic
• Application deadlocks are possible

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 29 of 37
 Better average response times possible
Example: start of four queries (each consumes 100% of IO)
PARALLEL_DEGREE_POLICY=AUTO
With Parallel Statement Queuing

Without Parallel Statement Queuing

_parallel_statement_queuing=false
Last Update November 11, 2016 Automatic Parallel Execution in Practice
Page 30 of 37
 Spreadsheet applications
Data browsers (like SQL*Developer)
The first n rows are displayed

The allocated PX servers for the user query

• Will be held
• Are actually idle until the user scrolls down
• Are counted as active for PARALLEL_SERVERS_TARGET

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 31 of 37
 Queuing on an idle system
Idle spreadsheet query (PARALLEL_SERVERS_TARGET=2)
SQL> select avg(sal) from t_emp_20; -- parallel query will be queued

SQL> SELECT NAME,

CURRENT_PQ_SERVERS_ACTIVE,
CURRENT_PQS_QUEUED
FROM GV$RSRC_CONSUMER_GROUP;

NAME CURRENT_PQ_SERVERS_ACTIVE CURRENT_PQS_QUEUED

------------ ------------------------- ------------------
OTHER_GROUPS 2 1

SQL> SELECT SID, USERNAME, EVENT, SECONDS_IN_WAIT, STATE

FROM GV$SESSION
WHERE USERNAME = 'SCOTT';

SID USERNAME EVENT SECONDS_IN_WAIT STATE

------ ---------- ---------------------------- --------------- -------
265 SCOTT SQL*Net message from client 2139 WAITING
32 SCOTT PX Deq Credit: send blkd 2177 WAITING
262 SCOTT PX Deq Credit: send blkd 2177 WAITING

25 SCOTT resmgr:pq queued 2099 WAITING

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 32 of 37
 Session or statement queuing?
If we continue the example we observe that an additional parallel
query from within SQL*Developer is not queued
• The user pins the first query and starts the next („Query Result 1“)
• The additional parallel query using the active QC connection is not queued

• PARALLEL_SERVERS_TARGET can be arbitrarily exceeded

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 33 of 37
 Session or statement queuing?
SQL*Developer QC session (sid=265) is now holding 6 PX slaves
The parallel query from SQL*Plus (sid=25) is still queued
NAME CURRENT_PQ_SERVERS_ACTIVE CURRENT_PQS_QUEUED
------------ ------------------------- ------------------
OTHER_GROUPS 6 1

SID USERNAME EVENT SECONDS_IN_WAIT STATE

------ ---------- ---------------------------- --------------- -------
265 SCOTT SQL*Net message from client 502 WAITING

32 SCOTT PX Deq Credit: send blkd 6718 WAITING

262 SCOTT PX Deq Credit: send blkd 6718 WAITING

24 SCOTT PX Deq Credit: send blkd 718 WAITING

274 SCOTT PX Deq Credit: send blkd 718 WAITING
19 SCOTT PX Deq Credit: send blkd 502 WAITING
272 SCOTT PX Deq Credit: send blkd 502 WAITING

25 SCOTT resmgr:pq queued 6640 WAITING

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 34 of 37
 Summary

PARALLEL_DEGREE_POLICY=AUTO enables three features

for parallel query

• Automatic degree of parallelism (automatic parallel execution)

• In-Memory parallel query
• Parallel statement queuing

This feature combination is well suited to interactive queries in an

(operational) data warehouse

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 35 of 37
 References

• J. Lewis, Cost-Based Oracle Fundamentals, Berkeley: Apress, 2006

• Y. Baskan, Optimizer Processing Rates for Auto DOP (The Data
Warehouse Insider)
• Y. Baskan, In-Memory Parallel Query (The Data Warehouse Insider), 2016
• MOS, Init.ora Parameter "PARALLEL_DEGREE_POLICY" Reference Note
(Doc ID 1216277.1), Oracle
• MOS, Automatic Degree of Parallelism in 11.2.0.2 (Doc ID 1269321.1),
Oracle
• K. Osborne, 12c - parallel_degree_level (control for auto DOP), Kerry
Osborne's Oracle Blog
• Kerry Osborne, Randy Johnson und Tanel Poder, „Expert Oracle Exadata,“
Berkeley, Apress, 2011, pp. 143-173

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 36 of 37
 Automatic Parallel Execution in Practice

Thank you for your attention.

Peter Heumel
Value Transformation Services
Am Tucherpark 12
D-80538 München

Telefon: +49 (0) 89-378 26920

E-Mail Peter.Heumel_V-TServices@de.ibm.com
Internet: www.v-tservices.com

Last Update November 11, 2016 Automatic Parallel Execution in Practice

Page 37 of 37