Oracle Trace File Analyzer (TFA)

Oracle Trace File Analyzer
(TFA)
Diagnostics for the Cloud
12.2.1.3.0
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 2

Why TFA?
Easy to use real-time

Secure consolidation of
health monitoring,
distributed diagnostic
fault detection & diagnosis
collections
via a single interface
Machine learning driven,

Continuous availability so
autonomous degradation
you always get what’s
detection, reduces your
needed for resolution
overheads

Autonomous Usage
Distributed diagnostics
3 are consolidated and
Diagnostics 2 packaged
are collected
TFA
Oracle Grid Infrastructure Oracle Support

& Databases
5 Diagnostic collection is
uploaded to Oracle
1 4 Support for root cause
TFA detects a fault Notification of fault is sent analysis & resolution

On-Demand Usage
Real-time Diagnose Perform
2 status 3 4 diagnostic
with DB
summary tools collection
Upload diagnostic
collection to Oracle
5 Support
TFA

& Databases
1 Request desired
action on-demand

Command Interfaces
Command line Shell Menu
• Specify all command options at 1. Set and change context 1. Select menu navigation
the command line options then choose the
2. Run commands from within command you want to run
tfactl <command> the shell
tfactl menu
tfactl
tfaclt > database MyDB
MyDB tfactl > oratop

Data Redaction
mask_strings.xml
• If you want to mask sensitive data
<mask_strings>
<mask_string> – Hostname
<original>WidgetNode1</original>
<replacement>Node1</replacement> – IP address
</mask_string>
<mask_string> – Etc
<original>192.168.5.1</original>
<replacement>Node1-IP</replacement> • Create mask_strings.xml
</mask_string>
<mask_string> – In the directory tfa_home/resources
<original>WidgetNode2</original>
<replacement>Node2</replacement> – Define all data replacements you want
</mask_string>
<mask_string>
<original>192.168.5.2</original>
<replacement>Node2-IP</replacement>
</mask_string>
</mask_strings>

Configure User Access
Users & groups authorized to access tfactl by default • Enable or disable user access
tfactl access enable
tfactl access disable
• List user access All commands apply

TFA tfactl access lsusers
cluster wide unless
-local used to restrict to
local node
• Add users
tfactl access add –user <user> [-local]
• Remove users or everyone

tfactl access remove –user <user> [-local]
tfactl access removeall [-local]
• Reset to default access

tfactl access reset
Grid Home Owner DB Home Owners Everyone Else

Set Email Notification Addresses
• TFA can send email notification
when faults are detected
• To set notification email for any
problem detected:
tfactl set notificationAddress=john.doe@oracle.com
• To set notification email for specific

ORACLE_HOMEs include the OS
owner:
tfactl set notificationAddress=oracle:another.person@oracle.com

On-Demand Usage
with DB
Upload diagnostic
5 Support
TFA

& Databases
1 Request desired
action on-demand

Real-time Status Summary
High-level summary of all

Database components
tfactl summary
Choose an
option to drill
down

Real-time Status Summary – Drill Down
Drill downs show real-time

analytics & details of any
problems found

On-Demand Usage
with DB
Upload diagnostic
5 Support
TFA

& Databases
1 Request desired
action on-demand

Investigate Logs & Look for Errors
• Analyze all important recent log entries: • Search recent log entries:
tfactl analyze –last 1d tfactl analyze -search “ora-006" -last 8h
Searching for
“ora-00600”

Perform Analysis Using the Included Tools
Tool Description Tool Description
orachk or Provides health checks for the Oracle stack. grep Search alert or trace files with a given database and file name pattern, for
exachk Oracle Trace File Analyzer will install either a search string.
• Oracle EXAchk for Engineered Systems, see document 1070954.1 for
more details summary Provides high level summary of the configuration
or vi Opens alert or trace files for viewing a given database and file name
• Oracle ORAchk for all non-Engineered Systems, see document pattern in the vi editor
1268927.2 for more details
tail Runs a tail on an alert or trace files for a given database and file name
oswatcher Collects and archives OS metrics. These are useful for instance or node pattern
evictions & performance Issues. See document 301137.1 for more details
param Shows all database and OS parameters that match a specified pattern
procwatcher Automates & captures database performance diagnostics and session level
dbglevel Sets and unsets multiple CRS trace levels with one command
hang information. See document 459694.1 for more details
history Shows the shell history for the tfactl shell
oratop Provides near real-time database monitoring. See document 1500864.1
for more details. changes Reports changes in the system setup over a given time period. This
alertsummary Provides summary of events for one or more database or ASM alert files includes database parameters, OS parameters and patches applied
from all nodes calog Reports major events from the Cluster Event log
ls Lists all files TFA knows about for a given file name pattern across all nodes events Reports warnings and errors seen in the logs
pstack Generate process stack for specified processes across all nodes
managelogs Shows disk space usage and purges ADR log and trace files
ps Finds processes
triage Summarize oswatcher/exawatcher data
Not all tools are included in Grid or Database install.
Download from 1513912.1 to get full collection of tools Verify which tools you have installed: tfactl toolstatus

ORAchk & EXAchk (Support Tools Bundle)
Oracle Stack Health Checks

• Oracle EXAchk for Engineered Systems
• Oracle ORAchk for everything else
• Proactively scans for the most impactful problems across the layers of your stack
• Simplifies and streamlines how to investigate and analyze which known issues
present a risk to you
• Lightweight tool that runs within your environment without requiring config data to
be sent to Oracle
• High level reports show your system health risks with the ability to drill down into
specific problems and understand their resolutions
• Can be configured to send email notifications when it detects problems
• Collection Manager, a companion Application Express web app, provides a single
dashboard view of collections across your entire enterprise

OS Watcher (Support Tools Bundle)
Collect & Archive OS Metrics

• Executes standard UNIX utilities (e.g. vmstat, iostat, ps,
etc) on regular intervals
• Built in Analyzer functionality to summarize, graph and
report upon collected metrics
• Output is Required for node reboot and performance
issues
• Simple to install, extremely lightweight
• Runs on ALL platforms (Except Windows)
• MOS Note: 301137.1 – OS Watcher Users Guide

Procwatcher (Support Tools Bundle)
Monitor & Examine Database Processes

• Single instance & RAC
• Generates session wait, lock and latch reports as well as call stacks
from any problem process(s)
• Ability to collect stack traces of specific processes using Oracle Tools
and OS Debuggers
• Typically reduces SR resolution for performance related issues
• Runs on ALL major UNIX Platforms
• MOS Note: 459694.1 – Procwatcher Install Guide

oratop (Support Tools Bundle)
Near Real-Time Database Monitoring

• Single instance & RAC
• Monitoring current database activities
• Database performance
• Identifying contentions and bottleneck

Analyze
• Each tool can be run using tfactl in shell mode
• Start tfactl shell with tfactl
• Run a tool with the tool name tfactl > orachk
1. Where necessary set context with database <dbname> tfactl > database MyDB
2. Then run tool MyDB tfactl > oratop
3. Clear context with database MyDB tfactl > database

Automatic Database Log Purge
• TFA can automatically purge database logs
– OFF by default
– Except on a Domain Service Cluster (DSC),
which it is ON by default
• Turn auto purging on or off: tfactl set manageLogsAutoPurge=<ON|OFF>
• Will remove logs older than 30 days

– configurable with: tfactl set manageLogsAutoPurgePolicyAge=<n><d|h>
• Purging runs every 60 minutes

– configurable with: tfactl set manageLogsAutoPurgeInterval=<minutes>

Manual Database Log Purge
• TFA can manage ADR log and trace files
– Show disk space usage of individual diagnostic destinations
– Purge these file types based on diagnostic location and or age:
• "ALERT“, "INCIDENT“, "TRACE“, "CDUMP“, "HM“, "UTSCDMP“, "LOG“
tfactl managelogs <options>
Option Description
Runs as the ADR home
–show usage Shows disk space usage per diagnostic directory for both GI and database logs owner. So will only be able
-show variation –older <n><m|h|d> Use to determine per directory disk space growth. to purge files this owner
Shows the disk usage variation for the specified period per directory. has permission to delete
-purge –older <n><m|h|d> Remove all ADR files under the GI_BASE directory, which are older than the time specified
–gi Restrict command to only diagnostic files under the GI_BASE
–database [all | dbname] Restrict command to only diagnostic files under the database directory. Defaults to all,
alternatively specify a database name
-dryrun Use with –purge to estimate how many files will be affected and how much disk space will be May take a while for a
freed by a potential purge command. large number of files

tfactl managelogs –show usage tfactl managelogs –show variation –older <n><m|h|d>
Use -gi to only

show grid
infrastructure
Use –database to only

show database

tfactl managelogs –purge –older n<m|h|d> -dryrun tfactl managelogs –purge –older n<m|h|d>
Use –dryrun
for a “what if”

Disk Usage Snapshots
• TFA will track disk usage and record snapshots to:
– tfa/repository/suptools/<node>/managelogs/usage_snapshot/
• Snapshot happens every 60 minutes, configurable with:
tfactl set diskUsageMonInterval=<minutes>
• Disk usage monitoring is ON by default, configurable with:

tfactl set diskUsageMon=<ON|OFF>

Change Trace Levels
• Sometimes Oracle Support may require certain specific trace levels to be
enabled when reproducing a problem
• TFA makes it easy to enable and disable the correct trace levels with a
single command
• All the required trace level settings are organised into problem specific
profiles
• The relevant profile trace logging can be enabled with:
tfactl dbglevel –set <profile>
• Profiles only available for Clusterware today, Database profiles coming soon..

On-Demand Usage
with DB
Upload diagnostic
5 Support
TFA

& Databases
1 Request desired
action on-demand

One Command SRDCs
• For certain types of problems
Oracle Support will ask you to
run a Service Request Data
Collection (SRDC)
• Previously this would have
involved:
• Reading many different
support documents
• Collecting output from
many different tasks
• Gathering lots of different
diagnostics
• Packaging & uploading
• Now just run:
tfactl diagcollect -srdc <srdc_type> -sr <SR#>

Collection Uploaded to My Oracle Support
At the end of the collection the file is automatically

uploaded to the SR

Upload any File
With Wallet: Without wallet:
tfactl upload -wallet -sr <sr_number> <file1 file2> tfactl upload -user <user_id> -sr <sr_number> <file1 file2>

Faster & Easier SR Data Collection
tfactl diagcollect –srdc <srdc_type> -sr <SR#>
Type of Problem SRDC
Type of Problem SRDC Database storage • asm
• ORA-00020 • ORA-04031
Excessive SYSAUX Space used by the Automatic • dbawrspace
• ORA-00060 • ORA-07445
Workload Repository (AWR)
• ORA-00600 • ORA-27300
ORA Errors • ORA-00700 • ORA-27301 Database startup / shutdown • dbshutdown
• ORA-01555 • ORA-27302 • dbstartup
• ORA-01628 • ORA-30036 Data Guard • dbdataguard
• ORA-04030
Enterprise Manager tablespace usage metric • emtbsmetrics
Other internal database errors • internalerror
Database performance • dbperf • emdebugon
Enterprise Manager general metrics page or
• emdebugoff
• dbpatchinstall threshold problems - Run all three SRDCs
Database patching • emmetricalert
• dbpatchconflict
Database resource • dbunixresources • emcliadd
• emclusdisc
XDB installation or invalid object • dbxdb
Enterprise Manager target discovery / add • emdbsys
• dbinstall • emgendisc
Database install / upgrade • dbupgrade • emprocdisc
• dbpreupgrade
Enterprise Manager OMS restart • emrestartoms
One Command SRDCs – Examples of What’s Collected
ORA-04031: Database Performance
tfactl diagcollect –srdc ORA-04031 tfactl diagcollect –srdc dbperf
1. IPS Package 1. ADDM report

2. Patch Listing 2. AWR for good and problem period
3. AWR report 3. AWR Compare Period report
4. Memory information 4. ASH report for good and problem period
5. RDA 5. OS Watcher
6. IPS Package (if errors during problem
period)
7. ORAchk (performance related checks)

Manual Data Gathering vs One Command SRDC
Manual Data Gathering TFA SRDC
1. Generate ADDM reviewing Document 1680075.1 1. Run tfactl diagcollect –srdc dbperf
2. Identify “good” and “problem” periods and gather AWR 2. Upload resulting zip file to SR
reviewing Document 1903158.1
3. Generate AWR compare report (awrddrpt.sql) using “good”
and “problem” periods
4. Generate ASH report for “good” and “problem” periods
reviewing Document 1903145.1
5. Collect OSWatcher data reviewing Document 301137.1
6. Check alert.log if there are any errors during the “problem”
period
7. Find any trace files generated during the “problem” period
8. Collate and upload all the above files/outputs to SR

One Command SRDC
Interactive Mode
tfactl diagcollect –srdc <srdc_type>
4. All required files are

identified
5. Trimmed where
applicable
6. Package in a zip ready

to provide to support
1. Enter default for event date/time and database name
2. Scans system to identify recent 10 events in the system (ORA600

example shown)
3. Once the relevant event is chosen, proceeds with diagnostic

collection

One Command SRDC
Silent Mode tfactl diagcollect –srdc <srdc_type> -database <db> -for <time>
1. Parameters(date/time, DB name) are provided

in the command
2. Does not prompt for any more information
3. All required files are identified
4. Trimmed where applicable
5. Package in a zip ready to provide to support

Default Collection
• Run a default diagnostic
collection if there is not
yet an SRDC about your
problem:
tfactl diagcollect
• Will trim & collect all

important log files
updated in the past 12
hours:
• Collections stored in the
repository directory
• Change diagcollect
timeframe with
–last <n>h|d

Custom Collection Options
Time
tfactl diagcollect <option>
Option Description Option Description

-last <n><h|d> Collect since the previous n hours or –from "yyyy-mm-dd" Collect to the date and optionally time
days -to "yyyy-mm-dd" specified.
-from “<yyyy-mm-dd>” Collect from the date and optionally Valid date / time formats:
time specified. "Mon/dd/yyyy hh:mm:ss"
Valid date / time formats: or "yyyy-mm-dd hh:mm:ss"
"Mon/dd/yyyy hh:mm:ss" or "yyyy-mm-ddThh:mm:ss"
or "yyyy-mm-dd hh:mm:ss" or "yyyy-mm-dd"
or "yyyy-mm-ddThh:mm:ss" -for “yyyy-mm-dd” Collect for the specified date.
or "yyyy-mm-dd " Valid date formats:
"Mon/dd/yyyy"
or "yyyy-mm-dd"

Nodes, Databases & Components
Option Description Option Description
-node Limit collection to comma separated list of nodes -emagent Collect EMAGENT logs
-oms Collect OMS logs
-database Limit collection to comma separated list of databases
-ocm Collect OCM logs
-asm Collect ASM logs
-emplugins Collect EMPLUGINS logs
-crsclient Collect Client Logs that are under GIBASE/diag/clients
-em Collect EM logs
-dbclient Collect Client Logs that are under DB ORABASE/diag/clients
-acfs Collect ACFS logs and Data
-dbwlm Collect DBWLM logs
-install Collect Oracle Installation related files
-tns Collect TNS logs
-cfgtools Collect CFGTOOLS logs
-rhp Collect RHP logs
-os Collect OS files such as /var/log/messages
-procinfo Collect Gathers stack and fd from /proc for all processes
-ashhtml Collect Generate ASH HTML Report
-afd Collect AFD logs
-ashtext Collect Generate ASH TEXT Report
-crs Collect CRS logs
-awrhtml Collect AWRHTML logs (requires –database option as well)
-wls Collect WLS logs

Misc
Option Description
-collectdir Collects other files in specified comma separated directory list. Only files updated in the relevant time
range (12 hours by default) get collected.
-tag Store the collection in a use a custom directory name within the repository, using the tag value specified
-z Append the value specified to the collection zip name
-nocopy Prevent the copying of zip files back to the initiating node
-notrim Prevent the trimming of files
-silent Start collection as a background process so output is not echoed to the console
-nocores Do not collect core files

Simple IPS (Incident Packaging Service) interface
• Database incidents are stored in the ADR (Automatic Diagnostic Repository)
• IPS packages details about these incidents for later diagnosis
Command Description
tfactl ips Invoke IPS
• TFA can invoke IPS to query and

tfactl ips show incidents Show all IPS incidents
tfactl ips show problems Show all IPS problems
collect these packages tfactl ips show package Show all IPS packages
tfactl diagcollect -ips –help Show all available diagcollect IPS options
tfactl ips <options> tfactl diagcollect –ips Perform an IPS collection following prompts
tfactl diagcollect -ips -adrbasepath Perform an IPS collection in silent mode.
or <adr_base> -adrhomepath <adr_home>
tfactl diagcollect -ips -incident <incident id> Collect ADR details about a specific incident id
tfactl diagcollect –ips <options>
tfactl diagcollect -ips -problem <problem id> collect ADR details about a specific problem id

Show Incidents
• Display details of incidents
recorded in ADR
tfactl ips show incidents
• Incident ID
• Problem Key
• Create Time

Show Problems
• Display details of
problems recorded in ADR
tfactl ips show problems
• Problem ID
• Problem Key
• Last Incident
• Last Incident Time

Show Package
• Display details of
packages recorded in ADR
tfactl ips show package
List of packages &

details of each package

Collect Packages
Select ADR home Rest of collection
Generate package Resulting collection zips
tfactl diagcollect -ips

Maintenance
• Option 1
– Applying standard PSUs will automatically
update TFA
– PSUs do not contain Support Tools Bundle
updates
• Option 2
– To update with latest TFA & Support Tools
Bundle 1513912.1

Safe Harbor Statement
The preceding is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.

Oracle Trace File Analyzer (TFA)

Încărcat de

Informații document

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Oracle Trace File Analyzer (TFA)

Încărcat de

Drepturi de autor:

Formate disponibile

Oracle Trace File Analyzer

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 2

Easy to use real-time

Machine learning driven,

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 3

Oracle Grid Infrastructure Oracle Support

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 4

Oracle Grid Infrastructure Oracle Support

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 5

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 6

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 7

• List user access All commands apply

• Remove users or everyone

• Reset to default access

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 8

• To set notification email for specific

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 9

Oracle Grid Infrastructure Oracle Support

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 10

High-level summary of all

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 11

Drill downs show real-time

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 12

Oracle Grid Infrastructure Oracle Support

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 13

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 14

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 15

Oracle Stack Health Checks

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 16

Collect & Archive OS Metrics

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 17

Monitor & Examine Database Processes

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 18

Near Real-Time Database Monitoring

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 19

• Run a tool with the tool name tfactl > orachk

2. Then run tool MyDB tfactl > oratop

3. Clear context with database MyDB tfactl > database

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 20

• Turn auto purging on or off: tfactl set manageLogsAutoPurge=<ON|OFF>

• Will remove logs older than 30 days

• Purging runs every 60 minutes

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 21

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 22

Use -gi to only

Use –database to only

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 23

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 24

• Disk usage monitoring is ON by default, configurable with:

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 25

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 26

Oracle Grid Infrastructure Oracle Support

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 27

tfactl diagcollect -srdc <srdc_type> -sr <SR#>

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 28

At the end of the collection the file is automatically

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 29

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 30

1. IPS Package 1. ADDM report

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 32

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 33

4. All required files are

6. Package in a zip ready