Documente Academic
Documente Profesional
Documente Cultură
Monitoring Exadata
Enterprise Manager Cloud Control 12c
Farouk Abushaban
Senior Principal Technical Analyst
Oracle USA, Engineered Systems Support
September, 2014
Copyright 2014, Oracle and/or its affiliates. All rights reserved. | Oracle Confidential Internal/Restricted/Highly Restricted
Copyright 2014, Oracle and/or its affiliates. All rights reserved. | Oracle Confidential Internal/Restricted/Highly Restricted
Objectives
Understand EM 12c Topology
Agent Deployment Best Practices
Component Level Monitoring
Discovery Deep-Dive
Copyright 2014, Oracle and/or its affiliates. All rights reserved. | Oracle Confidential Internal/Restricted/Highly Restricted
Program Agenda
Q&A Session
Program Agenda
Q&A Session
Section 1:
EM OVERVIEW
System Management
Software
Lights-Out Monitoring
and Notification
Management and
Administration
Single GUI
Centralized
Management
Target Administration
Life-Cycle
Management
Automation
Complete IT Monitoring
Oracle Products
Non-Oracle
technologies
Out-of-the-box
Metrics and Alerting
Real-time + Historical
Perf. Trending
Reports Publishing
EMCLI
EM Console
Agent
Repository
Database
Management
Server
Agent
Agent
10
11
Exadata Plug-In
12
Supported HW and SW
13
Program Agenda
14
Q&A Session
Section 2:
15
Exadata Monitoring
Install
16
Deploy Agents
Introduces Host Targets
Discover
Monitor
Customize Monitoring
Automate Tasks
Database Servers
Storage Servers
InfiniBand Switches
Cisco Switch
17
PDUs
KVM Switch
Agent Deployment
Install on each compute node
Database Servers
18
Exadata Monitoring
Deploy Agents
Agents run on compute nodes only
Compute nodes are RAC host targets
Monitor Exadata targets remotely
No additional software on Cells, IBs, KVM, PDUs, Cisco, and ILOM
19
20
21
22
23
Program Agenda
24
Q&A Session
Section 3:
Exadata Discovery
Deep-Dive
25
Specify Schematic
Active pre-requisite check
Sets up SSH user equivalence
Subscribes to SNMP
Supports re-discovery of newly added hardware
components
Assigns Primary and Backup agents to each component
26
27
28
Runs $/usr/sbin/ibnetdiscover
Reads the cell hostnames and IP addresses from the output
29
30
31
https://exa01db01.acme.com:3872/emd/main
ca 2 H-00212800.. # exa01db01 S 192.168.HCA-3
Prerequisite Checks
You can manually run this pre-requisite check ahead of time from the
compute node:
$ORACLE_HOME/perl/bin/perl exadataDiscoveryPreCheck.pl
32
Voltage
Temperature
35
36
Temperature
Voltage
Fan speeds
Configuration Data: Firmware version and Serial number, etc
37
38
Fan status
Power state
Factory settings
39
Program Agenda
40
Q&A Session
Section 4:
Challenges and
Troubleshooting
41
Challenges
Redeployment of a rack:
DEV to UAT to PROD etc
Partitioning full rack to smaller independent racks:
Full rack >> One rack + two racks
Combining partitioned racks to a larger rack:
Two racks >> Full rack
Two racks >> One rack
etc.
42
Update existing
OneCommand
configurations
Generate new
schematic files for
each partitioned rack
Generate new
OneCommand
configuration to
consolidate racks
43
44
45
Challenges
Discovery
Adding new hardware
Expanding or
Adding storage cells
Attaching additional rack
Attaching Storage Expansion rack
Adding spine switch
. etc
46
Challenges
Networking
Network configuration changes
Re-IP some or all components
Domain name changes
Hostname changes
Subnet changes
Additional backup network / NICs
Additional listeners (IB listeners or TNS)
Firewall rules
Etc
47
Troubleshooting
Discovery Issues
48
Fix
# ibnetdiscover
Match up to agent hostname
# /usr/sbin/set_nodedesc.sh
Troubleshooting
Discovery Issues
Extra or missing components new DBM
49
Check
Fix
Missing components
Troubleshooting
Discovery Issues
Discovery just hangs
50
Check
Review / Fix
Examine network
Hostname resolution
Accessibility from OMS to Agent(s)
Execute a simple job from the console
MW_HOME/gc_inst/sysman/log/emoms.log
Repository issues
Agent logs
$AGENT/agent_inst/sysman/log/gcagent.log
Troubleshooting
Schematic Issues
Schematic page blank
Check for Browser support and EM 12c
Run through discovery again and watch for messages
Check emoms.log for exceptions at the same time
Components missing
Add manually to the schematic page - Edit button
Check for component presence in EM (is it monitored?)
51
Troubleshooting
Target Status Issues
Target status shows DOWN inaccurately
Cell: Check ssh equivalence (cellmonitor user)
http://<pdu name>
Is it connected to the lan?
Cisco: Check for proper SNMP subscriptions
Troubleshooting
Metric Collections
Target status shows Metric Collection Error
Hover over the Icon or navigate to Incident Manager
Read the full text of the error
Visit the Target Setup >> Monitoring Configuration page and examine
Trigger a new collection: Target menu > Configuration > Last Collected >
https://<agent URL>/emd/browser/main
Click the target >> click Response and evaluate the results / log an SR
53
Troubleshooting
Pending Status
Cellsys target in Pending status forever
Must have Cluster ASM, Database and Storage Cell association
Check / fix the status of the associated target database
Check / fix the status of the associated target ASM cluster
Ensure UP status for all cell server targets
Delete unassociated cellsys targets
Check for problematic DBMS_JOBS in the repository database
54
Troubleshooting
Pending Status
Database Machine target or any associated components in Pending
status
Check for duplicate or pending delete targets:
Setup >> Manage Cloud Control >> Health Overview
Check target configuration:
Target Setup >> Monitoring Configurations
Search for the target name in the agent or OMS logs
$ grep <target name> gcagent.log or emoms.log
55
Troubleshooting
Maintenance
EMDiag
Download and install the latest version
Always check for the latest repvfy drop. Note 1426773.1
Run: repvfy verify exadata level 9 details
Run: repvfy verify
This will summarize all critical / fatal issues in the repository
Share the output with Support and explain the symptoms
56
Summary
What we covered today
57
References
Documentation Libraries, Notes, etc..
58
Learn More
Available References and Resources to Get Proactive
59
Program Agenda
60
Q&A Session
61
Join us for a relaxing Happy Hour after a busy day at Oracle OpenWorld!
Take a break and unwind with your peers
Get to know the Oracle support engineers you depend on
Meet My Oracle Support executives and developers
Enjoy drinks and hors doevres
Admission is free with your Oracle OpenWorld badge
Copyright 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential Internal/Restricted/Highly Restricted
62
THANK YOU
63
64