Best Practices at AT&T

Oracle Real Application Clusters (Oracle RAC) & Oracle E-Business Suite
Bryon Rickey Ravi Kayarthodi September 20th, 2010

2010 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks contained herein are the property of their respective owners.

Agenda
Welcome and Introductions Supply Chain Management (SCM) environment Overview/highlights Architecture Partnership with Oracle to transform our Business

Key project highlights

Future initiatives Solution Support Center (SSC) and on-site support Q&A

About AT&T
AT&T is one of the worlds largest Telecommunications companies AT&T Global Headquarters is based in Dallas, TX

AT&T has approximately 290,000 employees

AT&T has more than 90 million wireless customers

AT&T has more than 16 million high speed internet subscribers

AT&T has more than 2.5 million uverse subscribers Exclusive U.S. provider of iPhone 3G / 3GS / 4

AT&T SCM: Architecture highlights

Oracle E-Business Suite Centric Scale-up with cluster database on very large infrastructure

High availability with Oracle Data Guard, Oracle GoldenGate, and site level redundancy
Central inventory/financial management; utilizes product hub, planning/forecast with ASCP/Demantra

Order orchestration and brokering with in-house developed Oracle based application system
Internet deployment for interaction with external suppliers

Management reporting/profitability analysis; procurement and spend analytics

Industry leading warehouse/distribution management systems

AT&T SCM Platform: Financial Integration

Both Network and Retail SCM - financial counterparts are integrated in the same platform: Shared AP, AR, PA, FA, and GL

Supply Chain transactions leverage same referential data and accounting rules
Financial integration of supply chain transactions are seamless and are recorded real-time in the general ledger No need for reconciliation between sub ledgers Ability to forecast financial implications based on SCM forecasts Online financial budgeting validation on purchasing and requisitions Analytical GL Cube and Inventory Cube

Results
AT&T has saved more than $500 million and has improved its purchasing power by combining accounts with its vendors. Estimated to save $1B in inventory costs, and $300M per year in reduced operating costs. Warehouses have been able to keep inventories low without missing SLA, resulting reduced overall inventory costs. Ability to identify sold merchandise, resulting in fewer fraudulent returns

Industry Recognition
2010 IDGs CIO Award
Recognizes

organizations around the world that exemplify the highest level of operational and strategic excellence in information technology

Systems Architecture

AT&T SCM: E-Business Suite Centric

AT&T Corporate Systems
Provisioning Legacy Procurement Technician Ordering

Direct Fulfillment

COR Stores Other Billing Systems Profitability Analysis

Other Channels

Order Broker/Orchestrator

Asset Tracking

TMS

SOA Middle ware

Construction
General Ledger Receivables

3PL portal

EDI
Accounts Payable Purchasing

Procurement

Procurement partners Tax Lease Management

Oracle E-Business Suite 11i

External Facing; Suppliers and Auctions
Inventory Projects

Order Management

Assets

Planning/ Forecasting Network

Procurement & Spend Analytics

Spreadsheet Demand Planning

HR Employee

Warehouse/Distribution System

Archive

Returns

Retail Planning

AT&T SCM: Key Systems Architecture

ODP

eAuction

PO

AP

GL

OM
Inventory Planning

ASCP

RNF

PA
Financials

FA

Purchasing

Oracle E-Business Suite 11i

Receipts Shipments Receipts Moves Audits

Repairs

Warehouse Mgmt
Distribution Center

Inventory Tracking Corporate Asset Tracking

Oracle 11i E-Business Environment

Forward Orders
Store Replenishment Dealers/ iStore National Retailers DSL and uVerse HSIA Direct Fulfilment Upfront EPL

Planning

Purchasing

Order Management

Inventory

Receivables

Financials

Oracle E-Business Suite 11i Apps

Store XBM/Insurance Dealer/NR Service Depots RU30

Sales

Opus

Returns

Reverse

Company Owned Retail

416 CPU Cores & 1.7TB of Memory in DB Cluster

11i DB

14TB of storage for transaction DB

3TB of storage for Archived DB

AT&T SCM Platform: Oracle10g RAC Infrastructure: VCS+ CRS

AT&T SCM Platform: Transaction Volume

Approximately 3.35 million logons to the production areas per day 2.06 billion SQL executions per day Approximately 136.5 Billion logical reads and 41TB of physical reads performed per day Approximately 7 TB of writes performed per day Approximately 1.05 billion user calls and 52 million transactions completed per day 200,000 concurrent request per day

AT&T SCM Platform: Operational architecture

Decoupled month-end book closing preparation Statistics gathered on a clone Separate reporting environment refreshed nightly; Inventory and GL Cubes for reporting

Revenue share and allocation on a separate system

Direct supplier interaction with Internet-deployed application modules Utilizes market making mechanisms (sourcing)

AT&T SCM: Oracle 11i Dev/Test Configurations

Bothell, WA
HDS USP-V HDS USP-V

Alpharetta, GA

NetApp GOLD

Data Guard/ Custom Shipping

L-TRA

Data Guard

PROD

Data Guard

R-TRA
Data Guard/ Custom Shipping NetApp GOLD

Shadow Image

Shadow Image
NetApp

MET2
FTP

Master
Flex Clone

SnapMirror

Flex Clone

DEV DEV QA DEV DEV

FTP

Shadow Image

RPT

QA DEV QA

Flex Clone

NetApp Master

MET1

Flex Clone

QA

DEV DEV DEV

Storage Efficiency with NetApp de-duplication Tools

Performance test environment on tier-1 storage

AT&T SCM: Oracle Automatic Storage Management Local and Remote DR databases and also Reporting databases have been migrated to ASM storage to meet storage and capacity needs Avoided SFRAC costs Performance

SCMP DR/BC database converted to RAC database

Saved SFRAC license cost on E25K frames Comply with AT&T standard for cluster database (ASM is required) Expected to save cost in the future when we upgrade other Corporate Systems databases

AT&T SCM: Archive and Purge

Archive Stats 16 custom and core modules including (GL, INV, OM, AR, WSH, PO, FA) Executed weekly, monthly, quarterly, and yearly History instance Informatica Data Archive application used Schedule is monitored real-time to ensure adherence Purge Stats 70 jobs via Oracle concurrent programs Executed daily and weekly Schedule is monitored real-time to ensure adherence

AT&T Supply Chain

Geographic Distribution DB Flow to Test and Dev

Primary Site
XX XX

XX

XX

Test Site 1
XX XX

Test Site 2
XX XX

Development Site 2
XX XX

DR Site
XX Prod Dev/Test

XX
XX XX XX

XX
XX XX

XX

Development Site 1
XX XX XX XX XX XX

XX

XX

AT&T SCM: Spend Analytics

Oracle Business Intelligence Enterprise Edition (Oracle OBIEE) Powered Sourcing from Oracle E-Business Suite and Legacy Applications
Spend
$0 $5

Top 5 Categories

$M

Top 5 Suppliers

Jan

Feb

Mar

Apr

May

Jun $0 $5 $10 $M

Line Of Business

Jan

Feb

Mar

Apr

May

Jun

Contract Leakage

Off Contract Spend Payables Leakage

AT&T SCM: Database Maintenance/Performance Management Monthly month-end close of books stats validation database statistics preparation on production clone

Table Re-organization routine and ad-hoc

Periodic index rebuild Daily performance call

Daily/weekly/monthly/yearly purge/archive schedules vigilance in adhering to them

Extensive performance monitoring on all levels of the system

Partnering with Oracle Advanced Customer Service

Oracle Solution Service Center (SSC)

SSC: Project Spotlight: Patch Recommendations

Project Activities o Identification of available latest required patches for AT&T environment Participation in meetings to review recommendations Description

An initiative to incorporate E-Business Suite recommended patches, such as, Rollup (RUP) in AT&Ts release schedules for 2010

Value Addition
SSC and AT&T worked together to create a new process to efficiently identify required patches with the goal to minimize the impact of rolling them out in our production environment and to current/future projects: Quarterly document for AT&T environments with the latest recommended patches. Participation on AT&T IT meetings to clarify recommendations and questions Provide detailed patch analysis on a ad-hoc basis

Successful roll out of AP.O patchset along with other Financials patches in February 2010: stabilized production environment and brought environment to the latest code base. Successful completion of the ASCP Rup 37, Collections Rollup (RUP) 32 and Application Technology Group (ATG) RUP-7 for the April 2010 production rollout.

23

SSC: Project Spotlight: Performance Improvement

Project Activities o Understand AT&T batch window requirements and transaction volumes Identification and root cause analysis of problem areas Implementation of resolutions Description An initiative to address various performance issues related to MonthEnd-Close processes and other batch processes after Oracle 10g database upgrade.

o
o

Value Addition
Active participation on Conference bridge calls and OWC sessions. Engage Oracle development groups (Financials and Apps Performance) to provide SQL tuning improvements. Improve month end close activities like AR close, depreciation and asset retirement, optimizer, Invoice Validation processes. Maximize performance during month end, preventing delay; AT&T was able to successfully meet the SLAs for the Month End Close Processes month after month. Provide recommendations for major upgrades (DB, Applications and Tools)

SSC: Critical SR Spotlight: Other Key areas of focus

Project Activities Description
WF_DEFERRED queue growing exponentially; AT&T engages SSC to evaluate and address the issue. upgrade.

Workflow background process

o o

RAC CRS Configuration Device Life Cycle Database Upgrade

Collect diagnostic and configuration data Troubleshoot frequent RAC node reboot/performance issue post upgrade Discover unforeseen problems Validate AT&T configuration with RAC Best Practices
Provided upgrade expertise and support (pre and post deployment for 11g R2 upgrade.

Value Addition
Active participation on conference bridge calls and OWC sessions Review current setups and schedules of workflow Background Process Engage on Severity-1 SR and work collaboratively with Oracle development to provide workaround (eg. skip script) Work with at&t and Oracle development organization to provide scripts to clear workflow DEFERRED queue Monitor the implementation of recommended suggestions Engage Oracle Development and RAC COE team Engage on Architectural design discussions Stand-by support for DB/RAC upgrade deployment activities Ensure successful implementation of RAC upgrade Rollout

SSC: AT&T Targeted Services: Operations

The Services Yield to the Customer targeted for the next 6-month planning period is comprised of key Support activities, aligned with key customer objectives/activities towards achievement of AT&Ts Account Goals Supply Chain Transformation (Streamline Operations for newly deployed modules)
Work with ACS sales to determine appropriate support for upcoming SC rollout. Determine/secure funding for new modules (Demantra and Transportation Management Suite)

Develop, Enhance and Secure development sponsors

Participate in scheduled project Deep Dive calls with PMO and project team Site visit to Memphis to understand warehouse and logistic operations understand business functions. Supply Chain Business develop key relationships with Retail and Network business users Next generation iPhone launch and 2010 Holiday Retail Season readiness Top Ten Prioritization

11i Financials and Supply Chain Production Supportability (System)

SSC: AT&T Targeted Services: Transportation Manager

OneATT- New Horizon to transform Network and Asset Management: Logistics TMS

Key Secondary Activities

Participate in scheduled project Deep Dive calls with PMO and project team Work with PMO/SSC to determine appropriate support model for New Horizons Work closely with OCS /PMO on project milestones develop relationships Continue to maintain Dev OTM Network Supply Chain Business develop key relationships Site visit to Warehouses SSC walk-thrus Enhance Supply Chain SSC presence and understand usage and architecture for OTM Site visit to understand implementation, document in configuration guide. Performance testing as requested. Amend SSC contract to include OTM products

Key Deliverables

SSC: Oracle E-Business Suite Release 12/Database 11g R2 Upgrade Upgrade existing Oracle 11i.10 SCM and Planning environments to E Three stage upgrade (2 3 year journey)

Business Suite R12. This will enhances supportability for our complex environment and provide additional needed functionalities
Supply Chain planning to release 12 RDBMS for 10g to 11G Supply chain platform to release 12
System Availability, Supportability, and Software Currency SSC Top Ten Prioritization: Service Request review meetings with Supply Chain, Financials, and DBA support teams Patching/Release Management strategy for Release 12 interoperability; Leverage Tools and Templates Daily SSC Performance Calls Proactive Diagnostics (Configuration/Performance) Contracted deliverables Semi-Annual Account Review

Objective: Key Secondary Activities

Key Deliverables

Solution Support Center

Service Highlights
Focused Engineering Team of Oracle Subject Matter Experts Fast track solutions based on intimate Knowledge of customer Environment and business needs Proactive Software Advice and Knowledge Transfers

Performance and Patch Assessments

Single point of contact for all technical issues - Technical Lead Single point of contact for all escalations SDM 24/7 coverage

Value
Mitigate Upgrade Risks substantially
All project milestones are met Ensured optimal usage of Oracle features and functionality Reduced overall Maintenance/upgrade/ training costs

Solution Support Center

Key Delivered Features
Focused Engineering Team Single Point Accountability SDM Exclusive 1 800 number 24x7x365 Stand-by Support for Critical Go-lives Proactive Alerts / Patch Notifications

Review of Critical Issues as Needed

Patch Analysis

Project Planning / Migration Assistance

Periodic Review Sessions Applied Research

Questions

For More Information

On this presentation, e-mail Bryon Rickey (BR3681@att.com) or Ravi Kayarthodi (RK8409@att.com).

For general information on Advanced Customer Services, go to

oracle.com/acs
or e-mail

acsdirect_us@oracle.com