Idug Emea 2012 PDF

Click to edit Master title style
To Online or Not to Online?

Jacek Rafalak
ASSECO Poland S.A.
Session Code: A13
7.11.2012 | Platform: z/OS
Micha Biaecki
IBM Silicon Valley Lab
INVESTMENTS
Global IT player
Federation model
BUSINESS
FINANCIALS
The largest IT company in CEE
Double digit EBIT and

EBITDA margins
High quality proprietary

products and services
10-40% dividend
policy
Business diversification
More than 14.000 employees
SHRLEVEL NONE
SHRLEVEL
REFERENCE / CHANGE
Reasons for using ONLINE DB2 Utilities
Lack of maintenance window

No CPU during business hours
Better data availability
- your reasons?
Our purpose:
to share experiences
and best practices concerning
DB2 online utilities
Implementation
environment
description
Best practices
& recommendations
DB2 utilities &

their specific
features
IMPLEMENTATION
ENVIRONMENT
WHERE THE EXPERIENCE COMES FROM
Implementation environment
1/3
> 80 TB production data (5 members in DataSharing)

DML 48K/s, GETPAGES 392K/s, IO RW 13K/s IO R 12K/s
> 900 GB data growth per month
Largest objects
NPI INDEX 170 GB, Tablespace 200 GB
> 300 000 objects to maintain

> 250 000 separate utilities objects run monthly
Almost 80% CPU GP consumption
Weekly cost of maintenance for:
RUNSTATS - 240 MIPS
COPY - 90 MIPS
REORG/REBUILD - 80 MIPS

Production partitions
LPAR1
CEC1
DBP1 DBP7
DASD
LPAR3
2/3
DBP3
CF1
Parallel Sysplex
ZWAPPLX
DB2 Data Sharing
CF2
Production partitions
LPAR2
DBP2
LPAR4
DBP4
CEC2
Windows
Server
UNIX
Application
Server
WAN
Implementation environment
3/3
Components used in the implementation of DB2

ONLINE Utilities:
Tivoli Workload Scheduler (TWS)

in-house REXX scripts
Catalog DB2 (statistic tables)
DB2 9 Utilities for z/OS
Database utilities and maintenance cycle

Decision reports at least once a week critical thresholds daily!
RUNSTATS every object after 10% changes,
or at least every 2 months
REORG TS/IX & REBUILD IX
SHRLEVEL REFERENCE - during maintenance
window, once a week
SHRLEVEL CHANGE - on demand
COPY
Incremental SHRLEVEL CHANGE
twice a day for objects with > 10% changes OR
after 3 days, if any changes occur (in order to shorten DB2 log
scaning during recovery process)
Full - once a week with SHRLEVEL REFERENCE
10
Challenge #1
Long UOW or/and LRR (Long Running Readers)
blocks SHRLEVEL CHANGE utilities in switch phase
Potential solutions:
Run SQLs/jobs with smaller scope of input data
(thus higher ratio of commits per changes)

Adjust UT parameters: DRAIN_WAIT, DRAIN, DELAY
Adjust ZPARMs: URCHKTH, URLGWTH, LRDRTHLD,
IRLMRWT, UTIMOUT
Verify if there is enough CPU to keep your applications
working (maybe higher WLM service class is needed)
Monitor access path degradation
11
Challenge #2
Resource constraints:
CPU
shadow datasets (storage)
GBP
Consider using DPSI versus NPI (application performance can
suffer if not careful, DML changes needed)
Carefully check if unused indexes can be dropped
(SYSIBM.SYSINDEXSPACESTATS.LASTUSED
versus UNIQUE index and COLCARD used by optimizer)
Make objects non-GBP dependent
- ACCESS DB(dbname) SPACE(spacename) MODE(NGBPDEP)
Check if any of your zIIPs are available
Find CPU beyond prime time
Review objects allocation parameters to minimize needs for DASD
12
Challenge #3
Incompatible operations/tasks for DB2 ONLINE Utilities
Merge ranges (in order to rebuild NPI once per
tablespace)
Do REORG TABLESPACE instead of REORG +
REBUILD for segmented tablespace (detect objects
that can be redundant, e.g. REORG TS also REBUILD IX)
13
General recommendations for DB2 ONLINE

Utilities
Keep up to date with maintenance
Use current DB2 releases
See also DB2 for z/OS Utilities Update - presentation by
Haakon Roberts for most current APARs
(see bit.ly/TsN6Hv)
See presentation by Christian Michel on v10 utilities
(IDUG DB2 Tech Conference, Berlin 2012, session: A8)
V9 STATCLUS=ENHANCED is an IBM recommended
default (affects the way RUNSTATS gather statistics).
14
DB2 ONLINE UTILITIES
15
How To Prepare Your System

Find available CPU
Dedicate optimal service class to run utilities within your
production workload
Verify Storage Groups space (for FLASHCOPY and
shadow datasets)
Find suitable time to run utilities and avoid conflicts with

LRTs (Long Running Transaction) otherwise switch
phase will conflict
Initialise Real Time Statistics, e.g. TOTALROWS
16
How We select objects DECISION REPORTS

Make decision reports based on:
DB2 catalog
Admin actions, e.g. moving data
between volumes
Insert objects and your own parameters into control

table in order to:
prepare specific syntax for every object for REORG, REBUILD,
RUNSTATS and COPY incremental
run tasks simultaneously
easily exclude objects
17
How to select objects DECISION REPORTS

using DSNACCOX
1/5
DSNACCOX:
Is a DB2-supplied stored procedure basing on real-time statistics IFI calls
(DISPLAY status on objects)
Gives recommendations that help you maintain DB2 database objects
Lets you make decision reports to perform:
REORG
RUNSTATS
COPY
Can be customized to the users threshold (no one size fits all)
DSNACCOX is part of a DB2 product, which means that any

changes in the maintenancestream (PTF) or across versions
are transparent.
18

using DSNACCOX
2/5
Exception table:
user-created DB2 table
CREATE TABLE DSNACC.EXCEPT_TBL
DBNAME CHAR(8) NOT NULL,
NAME CHAR(8) NOT NULL,
QUERYTYPE CHAR(40)
...add more cols, if needed.. ) CCSID EBCDIC;
INEXCEPTTABLE column carries important information for your

decision logic returned from DSNACCOX, e.g.:
INSERT INTO DSNACC.EXCEPT_TBL VALUES(DSNDB04, MYHOTTABLE, do not
reorg on Fridays peak time);
19

using DSNACCOX set parameters
3/5
String QueryType = ALL";
String ObjectType = ALL";
..
int ChkLvl = 1 + 2 + 4 + 8;
String Criteria = "DBNAME
LIKE 'DSN8%'";
String Unused = "";
...
Int CRU*, ICR* parms for COPY
ommitted
Int RR* parms for reorg
ommited
...
int SRTInsDelUpdPct = 15;
int SRTInsDelUpdAbs = 5;
int SRTMassDelLimit = 0;
int SRIInsDelUpdPct = 15;
int SRIInsDelUpdAbs = 5;
int SRIMassDelLimit = 0;
int ExtentLimit = 2;
String lastStatement = null;
int IFCARetCode = 0;
int IFCAResCode = 0;
int XSBytes = 0;
int rc = 0;
String resultMessage = null;
String message = null;
boolean hasResultSet = false;
20
How to select object DECISION REPORTS

using DSNACCOX call example
4/5
CALL SYSPROC.DSNACCOX(:QUERYTYPE, :OBJECTTYPE, :ICTYPE, :CATLGSCHEMA,
:LOCALSCHEMA, :CHKLVL, :CRITERIA, :UNUSED, :CRUPDATEDPAGESPCT,
:CRUPDATEDPAGESABS, :CRCHANGESPCT, :CRDAYSNCLASTCOPY,
:ICRUPDATEDPAGESPCT, :ICRUPDATEDPAGESABS, :ICRCHANGESPCT,
:CRINDEXSIZE, :RRTINSERTSPCT, :RRTINSERTSABS, :RRTDELETESPCT,
:RRTDELETESABS, :RRTUNCLUSTINSPCT, :RRTDISORGLOBPCT, :RRTDATASPACERAT,
:RRTMASSDELLIMIT, :RRTINDREFLIMIT, :RRIINSERTSPCT, :RRIINSERTSABS,
:RRIDELETESPCT, :RRIDELETESABS, :RRIAPPENDINSERTPCT,
:RRIPSEUDODELETEPCT, :RRIMASSDELLIMIT, :RRILEAFLIMIT,
:RRINUMLEVELSLIMIT, :SRTINSDELUPDPCT, :SRTINSDELUPDABS,
:SRTMASSDELLIMIT, :SRIINSDELUPDPCT, :SRIINSDELUPDABS,
:SRIMASSDELLIMIT, :EXTENTLIMIT, :LASTSTATEMENT, :RETURNCODE,
:ERRORMSG, :IFCARETCODE, :IFCARESCODE, :XSBYTES);
As a result of this call, you receive a report determining

if to run RUNSTATS/COPY/REORG on those objects, or not.
21
DSNACCOX formula for REORG
22

using DSNACCOX result table
5/5
23
RUNSTATS Utility How We Use It
24
1/2
As much of synchronization with production workload as

possible
ZPARM STATCLUS=ENHANCED (an IBM recommended

default)
A post check-up of calculated date columns after running
RUNSTATS - overestimates/underestimates or wrong
values may happen, and need manual corrections UPDATE/INSTERT into CATALOG stats
(open a ticket with IBM, if value is wrong)
RUNSTATS Utility How We Use It
25
2/2
Remember that COLGROUPs need entire objects

for calculation (SAMPLE is ignored. However in DB2 10
TABLESAMPLE SYSTEM can be specified)
Remember that inline statistics in REORG

do not operate on colgroup level
Supporting tools - Statistics Advisor
Recommendation levels:
HIGH important statistics are missing or are obsolete

MAINTENANCE low priority, for periodical RUNSTATS to
maintain statistics accuracy
26
27
REORG INDEX SHRLEVEL CHANGE versus

REBUILD INDEX SHRLEVEL CHANGE
Use REBUILD INDEX SHRLEVEL
CHANGE for DEFER YES
Use REORG for physical moving
of datasets between SMS Storage Groups
CHARACTERISTICS
DRAIN
REORG INDEX
REBUILD INDEX
SHRLEVEL CHANGE
SWITCH
START
SHADOW DATASETS
YES
NO
SQL ACCESS USING IX
YES
NO
MULTITHREADING
NO
YES
REBUILD INDEX SHRLEVEL CHANGE SQL

Access Considerations
Be aware that your access path can degrade for dynamic SQL
while REBUILD INDEX is being run (RFE # 26259 opened)
Pay attention to unique indexes restriction:

SHRLEVEL CHANGE is not well suited for unique indexes and
concurrent DML because the index is placed in RBDP while being
built. Inserts and updates of the index will fail with a resource
unavailable (-904) because uniqueness checking cannot be done
while the index is in RBDP
Utility Guide and Reference
Remember that REBUILD was designed rather for disaster

recovery or rebuilding destroyed IX, and not for REORGalike approach.
28
REORG TABLESPACE - How We Use It

Make decision reports, prioritise objects, update your control table
Select candidate object for control table
Release GBP dependency
REORG TABLESPACE
RUNSTATS TABLESPACE and RUNSTATS INDEX
Verifiy and update/insert manually DB2 catalog stats (only when applies)
Update your control table for a chosen object with finish flag, when tasks end
29
Control table
CREATE TABLE
PG.MAINTENANCE_CONTROL_TABLE (
ST_OBJECT
ST_DATABASE
ST_PARTITION
ST_OBJECT_TYPE
ST_PRIORITY
ST_PLANNING_DATE
ST_UPDATEPRIO_DATE
ST_JOBID
ST_SAMPLE
ST_SQLID
ST_RULE_NAME
ST_NACTIVE
ST_ONDEMAND
RUNSTATS part
30
REPORTING part
AD_COPY
AD_REORG
AD_REBUILD
AD_CHECK
AD_ALTER
AD_RUNSTATS
AD_REPORT_ID
RE_IMPORTANCE
RE_UPDATEDATE
RE_JOBID
RE_IDPOD
RE_UPDATEPRIO_DATE
REORG part
REORG TABLESPACE Points To Consider 1/2

Prepare your environment for huge shadow datasets
A real life case - for 3 partitions were required:
100 GB for shadow data sets / indexes NPI
171-minute elapsed time
CPU: 0 HR 45 MIN
Keep in mind that NPIs are always rebuilt entirely even

when a partition is empty
DSNU050I 186 10:15:51.42 DSNUGUTC - REORG TABLESPACE BR03PC.SPCCEKKD PART(1, 2, 5:6) LOG NO
SORTDATA SORTKEYS
SORTDEVT 3390 SORTNUM 10 SHRLEVEL REFERENCE
DSNU701I -DBP2 186 10:15:53.06 DSNURULN - 1560 INDEX ENTRIES UNLOADED FROM 'R03.X1_CMKKD'
DSNU251I -DBP2 186 10:15:53.34 DSNURPUT - UNLOAD PHASE STATISTICS - NUMBER OF RECORDS
UNLOADED=0 FOR TABLESPACE
BR03PC.SPCCEKKD PART 1:1
31
REORG TABLESPACE Points To Consider 2/2

Be aware that UTS (PBG) after using ALTER ADD
COLUMN and REORG PART (n) data may not be able to
fit into its original place - (RFE # 27089 opened)
DSNU1170I -GB5 097 10:50:43.19 DSNURWAP - UNABLE TO EXTEND
PARTITION-BY-GROWTH TABLESPACE ZDB.SW2 BECAUSE OF PARTITION RANGE.
DSNU398I -GB5 097 10:50:43.19 DSNURWBF - UNEXPECTED PROCESSING ERROR,
REASON=X'00E40318' ON TABLE - ...
Keep 1 table per 1 tablespace

Try to use partitioned tablespace wherever possible
32
Integration of the DB2 Utilites with TWS

automation
1/2
Characteristics of implementation:
Universal JCL pattern for every job

Application limit - 255 programs per application
Automatic restart in case of an error
Dependencies between TWS applications, in this case:
utility and batch production workload
One steering wheel for all maintenance tasks
33
Integration of the DB2 Utilites with TWS

automation
2/2
34
IN A NUTSHELL
35
Best Practices for Implementing DB2 Utilities

in a Complex Environment
What works the best for us:
Complex reporting based on IBM manuals
(e.g. DSNACCOX) and your databases changes (RTS)
Utilities compatible with each other and with production
workload (LRT to be observed!)
Weighting pros and cons of REBUILD INDEX and
REORG INDEX
Objects grouped into ranges
As much of automation of maintenance
tasks as possible
36
Automation of DB2 Utilities in a Complex

Environment - Benefits
Automation of DB2 utilities:

Allows utilities to be run 24/7
Saves MIPS, especially when you use RTS statistics
Involves less work for DBAs:
Potential human errors less probable
Less monitoring required
In a complex environment with a rapid data growth

automation results in immediate cost cuts.
37
Q&A
38
Jacek Rafalak
Micha Biaecki
ASSECO Poland S.A.
IBM Silicon Valley Lab
Jacek.Rafalak@asseco.pl
Michal.Bialecki@pl.ibm.com
Session: A13
To Online or NOT to Online?
APPENDIX
40
APARS LIST
41
1/3
DSNACCOX
PM19064 DSNACCOX DOES NOT RETURN A VALUE IN OBJECTSTATUS FOR LOBS
PM25652 DSNACCOX ENHANCEMENT TO RECOMMEND REORG ON HASH ACCESS OBJECTS
PM27269 DSNACCOX PERFORMANCE ENHANCEMENT
PM31040 DSNACCOX GIVES INCORRECT OUTPUT AND LOOPS IF THE SEARCH CRITERIAIS
ON DATABASE DSNDB01
PM48689 DSNACCOX servicability APAR
PM51730 DSNACCOX LISTS OBJECTS WITH REORG NO WHEN TYPE REORG REQUESTED
PM61574 DSNACCOX INCLUDES NPI INDEXES WHEN CHKLVL=32
APARS LIST
42
2/3
RTS:
PM22081 REAL-TIME STATISTICS COLUMN TOTALENTRIES MAY BE INCORRECT AFTER LOAD
RESUME YES FOR PARTITIONED OBJECT
PM31916 RTS TOTALROWS NULL FOR NEWLY CREATED PBG PARTITION
PM33331 REALTIME STATISTICS ( RTS ) TOTALROWS OUT-OF-SYNC WITH ACTUAL TABLE COUNT
PM38593 COLUMN COPYUPDATEDPAGES OF RTS TABLE TABLESPACESTATS IS NOT UPDATED.
PM45872 RTS TOTALROWS INCORRECT AFTER MULTIPLE REORG SHRLEVEL NONE PART
PM55797 CUSTOMER GOT INCORRECT NLEVLES IN RTS TABLE SYSINDEXSPACESTATS
PM58575 AUTOBIND DEADLOCKS WITH RTS UPDATE ENGINE
PM60552 RUNSTATS INSERTS ROW INTO RTS (REAL TIME STATISTICS) FOR DEFINE NO OBJECT
PM60866 A MASS DELETE OPERATION ON A MULTI-TABLE SEGMENTED TABLESPACE RESETS ALL
RTS COUNTERS TO NULLS, INCLUDING REORGMASSDELETE
PM66386 THIS APAR PROVIDES AN PERFORMANCE ENHANCMENT IN MASS DELETE/ /TRUNCATE
ON DGTT BY AVOIDING SENDING NOTIFY ME
PM67558 High DBM1 pre-emptible SRB CPU for RTS system thread in DSNGERTS
APARS LIST
RUNSTATS/REORG/COPY see Haakons presentation:
DB2 for z/OS Utilities Update -> bit.ly/TsN6Hv
43
3/3
Opening Request for Enhancement
44
1/2
Instruction:
The DB2 for z/OS Request for Enhancements (RFE) Community
enables customers to directly submit, manage and track their
requirements online. Additionally, customers can access the
requirements submitted to by other customers to vote on, comment on
and watch in a social media paradigm.
DB2 for z/OS RFE Community:
http://www.ibm.com/developerworks/rfe/?BRAND_ID=184
Opening Request for Enhancement
45
2/2
How to submit a new DB2 for z/OS RFE requirement

Once you've logged onto the DB2 for z/OS RFE Community with your
IBM ID, click on the 'Submit RFE' link and complete the form. You will
notice that many of the fields are auto-filled already for you. To open
a new DB2 for z/OS RFE requirement, set:
* Brand = Information Management
* Product Family = DB2 for z/OS
* Product = DB2 for z/OS
It is important that you provide detailed information in the Description,
Use Case, and Business Justification fields to help the DB2 for z/OS
Subject Matter Expert understand and disposition your requirement
(and decide if set to PUBLIC). However, do not enter any sensitive
or confidential information including names.
Request for Enhancement - RFE Group

How can I view, vote, and watch private RFE requirements opened
by others in my company?
The RFE Community provides a construct called RFE Group,
consisting of a collection of individuals interested in the same RFEs, who
can share a common watchlist of RFE requirements, regardless of public
or private. RFE Groups can be public or private as well. Public RFE
Groups are visible in the RFE Community's Group Directory and allow
interested individuals to petition the Public RFE Group owner to join.
46

Idug Emea 2012 PDF

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Idug Emea 2012 PDF

Încărcat de

Drepturi de autor:

Formate disponibile

Click to edit Master title style

To Online or Not to Online?

Click to edit Master title style

The largest IT company in CEE

Double digit EBIT and

High quality proprietary

More than 14.000 employees

Click to edit Master title style

Click to edit Master title style

Reasons for using ONLINE DB2 Utilities

Lack of maintenance window

Click to edit Master title style

DB2 utilities &

Click to edit Master title style

Click to edit Master title style

> 80 TB production data (5 members in DataSharing)

> 300 000 objects to maintain

Click to edit Master title style

DB2 Data Sharing

Click to edit Master title style

Components used in the implementation of DB2

Tivoli Workload Scheduler (TWS)

Click to edit Master title style

Database utilities and maintenance cycle

Click to edit Master title style

(thus higher ratio of commits per changes)

Click to edit Master title style

Click to edit Master title style

Click to edit Master title style

General recommendations for DB2 ONLINE

Click to edit Master title style

DB2 ONLINE UTILITIES

Click to edit Master title style

How To Prepare Your System

Find suitable time to run utilities and avoid conflicts with

Click to edit Master title style

How We select objects DECISION REPORTS

Insert objects and your own parameters into control

Click to edit Master title style

How to select objects DECISION REPORTS

DSNACCOX is part of a DB2 product, which means that any

Click to edit Master title style

How to select objects DECISION REPORTS

INEXCEPTTABLE column carries important information for your

Click to edit Master title style

How to select objects DECISION REPORTS

Click to edit Master title style

How to select object DECISION REPORTS

As a result of this call, you receive a report determining

Click to edit Master title style

DSNACCOX formula for REORG

Click to edit Master title style

How to select objects DECISION REPORTS

Click to edit Master title style

RUNSTATS Utility How We Use It

As much of synchronization with production workload as

ZPARM STATCLUS=ENHANCED (an IBM recommended

Click to edit Master title style

RUNSTATS Utility How We Use It

Remember that COLGROUPs need entire objects

Remember that inline statistics in REORG

Click to edit Master title style

Supporting tools - Statistics Advisor