Data Warehousing

Basics of Data Warehousing

 Session Objectives

At the end of this session, you will be able to answer

 What, Why and Where -- Data Warehouse

 Terminologies / Technical jargons used

 OLTP and OLAP

 DW Architecture

 ETL process in a Data Warehousing environment

 Understand the Evolution of ETL tools

 DW – 3 W’s

Data Warehouse --- What? Why? Where?

Decision Support System

In order to make correct decisions, accurate, meaningful information about

business environments, external issues, and internal workings must be available
in a timely fashion.
 What is a Data Warehouse ?

A data warehouse is a subject-oriented, integrated,

nonvolatile, time-variant collection of data in
support of management's decisions.
- WH Inmon

WH Inmon - Regarded As Father Of Data Warehousing

 Integrated - Characteristics of a Data
Warehouse
Appl A - m,f
Appl B - 1,0 m,f
Appl C - male,female

Appl A - balance dec fixed (13,2) balance dec

Appl B - balance pic 9(9)V99 fixed (13,2)
Appl C - balance pic S9(7)V99 comp-3

Appl A - bal-on-hand
Appl B - current-balance Current balance
Appl C - cash-on-hand

Appl A - date (julian)

Appl B - date (yymmdd) date (julian)
Appl C - date (absolute)

Integrated View Is The Essence Of A Data Warehouse

 Non-volatile - Characteristics of a Data
Warehouse
insert change

Operational Data
Warehouse
insert
delete
load
read only
access
replace
change

Data Warehouse Is Relatively Static In Nature

 Historical Look at
Informational Processing
The goal of Informational Processing is to turn data into
information!

Why?

Because business questions are answered using information and

the knowledge of how to apply that information to a given problem.

Data Information Knowledge

 A Need For New Technology
 Government and industrial entities have been collecting data in electronic format
since the 1960s.
 Today, organizations collect millions of pieces of information about every
aspect of their operation on a daily basis.
 Data is obtained from multiple disparate sources.
 Often information is replicated, leading to confusion.
 Related data is often retained in seemingly heterogeneous and incompatible
platforms.
 Common data attributes are represented in nonstandard formats and naming
constructs across systems.
 Most systems are built for data collection (transaction based).
 Designed to support On-Line Transaction Processing (OLTP).
 Designed to support day-to-day business operations.
 Very specific applications built to support interaction with the data.
 Perform best when handling small specific volumes of data.
 Does not accept information from dissimilar sources readily.
 A Need for New Technology
contd..
 Capable of answering questions of a specific nature and time frame.
 How many items do I have in stock today?
 How many tickets were sold on a specific date?
 What is the current price of an item?

 Transaction based systems experience great difficulty in answering analytical

and decision support questions.
 Analysis takes a long time, interfering with:
 transaction performance
 daily operations
 The nature of the data is dynamic and dispersed.
 A Need for New Technology
contd..
Most organizations have created a “spider web” of systems and data sources.

Databases Applications
 A Need for New Technology

All of this has created “data overload” and “data confusion”.

 What do I do with all of this data?
 What does it mean?
 Do I really need this data?
 I am overwhelmed with the amount of data I am confronted with.
 I cannot make a timely decision (too much data from too many sources).
Data, Data everywhere
• I can’t get the data I need
– need an expert to get the data

 I can’t find the data I need

 data is scattered over the network
 many versions, subtle differences
• I can’t understand the data I found
– available data poorly documented

• I can’t use the data I found

– results are unexpected
– data needs to be transformed from one
form to other
 Data Warehousing

Data warehousing is:

A large historical database designed to accept key analytical data from
multiple and disparate sources that manage the day-to-day management of
enterprise data. Furthermore, the role of the warehouse is to transform
transaction data into corporate information.
information The warehouse is provided in
a read-only fashion to a user.
 Data Warehousing

A data warehouse will provide:

The ability to ask business analysis questions in a real-time, iterative

fashion, obtaining decision support information readily and quickly.
 Data Warehousing
 A data warehouse is not:
not
A repository for all corporate data.
 A data warehouse will not:
not
Single handedly solve all of the problems associated to an enterprise.
 DW Jargons
 OLTP – Online Transaction Processing
 OLAP – Online Analytical Processing
 ETL – Extraction Transformation and Loading
 DSS – Decision Support System
 Metadata – Data About Data
 Fact – Numeric values
 Dimensions – Subject Areas of Business
 ODS – Operational Data Store
 STG – Data Staging Layer
 Cube
 Star Schema
 Snowflake Schema
What is an Operational System /
OLTP?
 Operational systems are just what their name implies; they are the systems that help
us run the day-to-day enterprise operations.

 These are the backbone systems of any enterprise, such as order entry inventory etc.

 The classic examples are airline reservations, credit-card

authorizations, and ATM withdrawals etc.,

 Characteristics of Operational
Systems

 Continuous availability

 Predefined access paths

 Transaction integrity

 Volume of transaction - High

 Data volume per query - Low

 Used by operational staff

 Supports day to day control operations

 What is OLAP?

 OLAP (On Line Analytical Processing) applications - designed for online

ad-hoc data access and analysis.

 Data organized into multiple dimensions.

 Access to analytical content such as time series and trend analysis views

and summary level information.

 A set of functionality that attempts to facilitate multidimensional analysis.

 Offers drill-down, drill-across and slice and dice capabilities.

 OLAP - Fast Analysis

• On Line No piles of paper, please!

• Analytical Establish patterns

• Processing Data-based

• Fast Analysis of Shared Multidimensional Information

 What is ETL?
 ETL (Extraction, Transformation and Loading) is a process by which data is
integrated and transformed from the operational systems into the Data
Warehouse environment
Filters and
Extractors
Cleanser
Error
Operational systems Cleaning View
Rules Check
• Rule 1 Correct
• Rule 2
• Rule 3
Transformation
Rules

• Rule 1
• Rule 2
• Rule 3

Transformation
Engine
Integrator

Error
View
Check
Correct Loader Warehouse
 Data Warehousing Architecture
Overview
Data Warehouse Architecture

Information Directory Repository

Data
Legacy Data
Transformation

Data Warehouse

Legacy Data Data Warehouse Management Layer

External Data Source

 Data Warehousing

Model concepts:
 Fact table(s)
 A table containing multiple measurable descriptors relating to a

specific area of business

 Each fact can be viewed, calculated, and aggregated against various
defining areas of the business (time, geography, customer)
 Data Warehousing

Model concepts:
 Dimension Table(s)
 Retains information (product description, geography description,
customer description) that is descriptive and remains moderately
constant over time
 Data Warehousing

Data Warehouse Modeling

 Special modeling techniques must be applied to provide rapid response of
queries on large volumes of data.
 OLTP systems are built with update operations in mind, resulting in
normalization and greatly reduced browse performance.
 Data Warehousing

Common data model techniques are as follows:

 star schema
 snowflake
 fact constellation
 relational
 Data Warehousing
Sample Star Schema Model

TIME GEOGRAPHY

Dimensions Dimensions
SALES

STORE CUSTOMER
Sales Facts
 Data Warehousing
Year
North
Sample Snowflake Model
Qtr South

Month TIME East

GEOGRAPHY
West

Dimensions Dimensions
SALES

East Region
STORE CUSTOMER
West Region Sales Facts
 Data Warehousing
Sample Fact Constellation Model

TIME Regional GEOGRAPHY

Sales

District
Dimensions Dimensions
Sales

Store
Sales
STORE CUSTOMER
 Data Mart

Data mart is:

A functional segment of an enterprise restricted for purposes of security,
locality, performance, or business necessity using modeling and information
delivery techniques identical to data warehousing.
 Data Mart

Why build a data mart?

 Allows an organization to visualize the large but focus on the small and
attainable.
 Provides a platform for rapid delivery of an operational system.
 Minimizes risk.
 A corporate warehouse can be constructed from the union of the enterprise
data marts.
 Data Mart

Data From
Transaction Sources Data
Warehouse
Update From the
Warehouse

The data warehouse

populates
the data marts. Financial Logistics Contract
Data Mart Data Mart Data Mart
 Data Mart

The data marts populate

the data warehouse.
Data
Warehouse
Update From the
Data Marts

Financial Logistics Contract

Data Mart Data Mart Data Mart

Data From
Transaction Sources
 Data Mart

Virtual Data Warehouse

Abstract Data Warehouse
Data is moved through the Access Layer
abstract layer on demand.

The data warehouse layer

manages the data marts Financial Logistics Contract
as a warehouse. Data Mart Data Mart Data Mart
Data From
Transaction Sources
 OLAP
 OLAP is a powerful graphics-oriented tool used to access the data warehouse
 OLAP supports
 Business analysis queries
 Data visualization
 Trend analysis
 Scenario analysis
 User defined queries
 OLAP
 Drill Down
 Move from summary to detail
 Roll Up
 Move from detail to summary
 Slice and Dice
 Look at a specific interest of the business
 Pivot and Rotate
 Looking at data from varying perspectives
 Drill Through
 Move to a near transaction level of detail
 OLAP
 The flavors of OLAP
 Multidimensional On-Line Analytical Processing (MOLAP)
 Relational On-Line Analytical Processing (ROLAP)
 Hybrid On-Line Analytical Processing (HOLAP)
 OLAP
 MOLAP
 Produces a hypercube
 Pre-aggregated and pre-calculated
 Rapid response times
 Limited in the amount of data that can be managed
 ROLAP
 Data remains in a relational format
 Some degree of aggregation
 Slower response times
 Scales to large amounts of data
 HOLAP
 Can manage data both as ROLAP and MOLAP
 Currently evolving
 MOLAP vendors are finding it easier to move into the HOLAP market space
 Data Mining

As defined by the Gartner Group in 1995, data mining is:

“…the process of discovering meaningful new correlations, patterns, and

trends by sifting through large amounts of data stored in a repository, using
pattern recognition technologies and statistical and mathematical
techniques.”
 Data Mining
 Data mining requires an analyst who is familiar with the domain to appropriately
model scenarios.
 Data mining assists analysts in uncovering nontrivial data relationships.
 Analysis must be conducted to determine the meanings of these newly identified
relationships.
 Why Use a Data Warehouse ?
 Data warehousing is a must for anyone who uses multiple data sources to make
decisions and understand business (trends, forecasting).
 Those who do not move to warehousing will not be capable of responding to
problems and business conditions, thus falling behind the competition.
 For organizations wanting to minimize costs and maximize productivity,
warehousing is a must.
 Individuals who spend time gathering data instead of analyzing data require the
assistance of a warehouse.
 Organizations that collect data but have difficulty determining meanings and
impacts need a data warehouse.
 Making the Warehouse a
Reality
 Think big but work small.
 Match technology to requirements.
 Build for the future (scalability).
 Work closely with the users.
 Requirements
 Rapid Application Development (RAD)
 Periodic releases to the user community
 Real World Success Stories
 Radio Shack
 Sales and stocking analysis
 Marketing (regionalized mailings)
 Wal-Mart
 Sales and stock analysis
 Trend analysis
 Vendor analysis
 Naval Surface Warfare Center (NSWC)
 Procurement
 Supply
 Workload
 Harris Semiconductor
 Yield
 Product
 Personnel productivity
 A Few Observations About
Data Warehouses
 Industry and our experience indicate that:
 Warehouses that succeed average an ROI of 400% with the top end being as
much as 600% in the first year.
 The incremental approach is most successful (build the warehouse a
functional area at a time).
 The average time to gather requirements, perform a design, and deploy a
warehouse increment is six months.
 New tools may be required that differ from the transaction environment.
 Software oriented toward intelligent analysis and query of the data

warehouse
 Hardware oriented to support the massive storage requirements and

analytical queries
 Keys to Success
 Do you understand why you are building the warehouse?
 Have you identified both technical and business professionals that you will need
to build the warehouse?
 Do you have a strong management sponsor?
 Are you managing the expectations of the users?
 Careers in Data Warehousing
 System Administration  DBA
 DW Architect  Application Developer
 Data Architect  Data Cleansing/ Transformation Analyst
 DW Manager  Business Analyst
 DW Administrator  Management
 Decision Support Analysts