Lecture 01 Tue, Jan 20, 2009 1800 : 2100 FAST NU, Karachi

Course Outline

Introduction to Data Warehousing and Background Dimension Modeling Architecture and Infrastructure Extract Transform Load Data Quality Management OLAP Implementation Methods of Data Warehouse Data Mining Overview

Course Material
Data Warehousing Fundamentals by Paulraj Ponniah John Wiley and Sons Articles
Class Notes

Marks Distribution

Objective of the course

Why exactly the world needs a Data Warehouse?
How Data Warehouse differs from traditional databases

and RDBMS? Where does OLAP stands in the Data Warehouse picture? What are different Data Warehouse and OLAP models/schemas? How to perform ETL? What is data cleansing? How to perform it? What are the famous algorithms? Which different Data Warehouse architectures are there? What are their strengths and weaknesses?

What is a Data Warehouse?

The Data Warehouse is an integrated, subject-

oriented, time-variant, non-volatile database that provides support for decision making
Subject Oriented
Organized along the lines of the subjects of the corporation. Typical subjects are customer, product, vendor and transaction.

Integrated
Single, Enterprise-Wide view.

Time Variant
Every record in the data warehouse has some form of time dimension attached to it.

Non Volatile
Refers to the inability of data to be updated. Every record in the data warehouse is time stamped in one form or the other.

Decision Support is a methodology (or a series of

methodologies) designed to extract information from data and to use such information as a basis for decision making
6

What is a Data Warehouse?

Legacy Data
Large Scale Data Collection Generation or Digitization Exercise Online Online Operational Online Operational Source Online Operational Source Operational Source Source

Corporate Decision Support Infrastructure Reporting End DW Servers User

Needs for Strategic Information

Retain the present customer base
Increase the customer base by 15% over the next 5

years Gain market share by 10% in the next 3 years Improve product quality levels in the top five product groups Enhance customer service level in shipments Bring three new products to market in 2 years Increase sales by 15% in the Northern Division
8

Need of a Data Warehouse

The amount of data the average business collects and

stores is doubling each year Total hardware and software cost to store and manage 1 Mbyte of data
1990: ~ $15 2002: ~ 15 (Down 100 times) 2005: ~ 1 (Down 1500 times)

A Few Examples

Cern: Up to 20 PB by 2006 Stanford Linear Accelerator Center (SLAC): 500TB France Telecom: ~ 100 TB WalMart: 24 TB
9

Operational Systems
User needs information
User requests reports from IT IT places request on backlog IT creates ad queries IT sends requested reports User hopes to find the right answer User needs information

10

Operational vs. Informational

Operational Data Content Data Structure Access Frequency Access Type Usage
Current values Optimized for transactions High

Informational
Archived, derived, summarized Optimized for complex queries Medium to low

Read, update, delete Predictable, repetitive

Read Ad hoc, random, heuristic

Response Time
Users

Sub seconds
Large number

Several seconds to minutes

Relatively small number
11

Data Warehouse
Information Sources Data Warehouse Server (Tier 1) OLAP Servers (Tier 2) e.g., MOLAP Semistructured Sources extract transform load refresh etc. Operational DBs Analysis Data Warehouse serve Query/Reporting serve e.g., ROLAP serve Data Mining Clients (Tier 3)

Data Marts

12

Online Transaction Processing (OLTP)

Also known as operational sources Day-to-day handling of transactions that result from

enterprise operation Airline reservation systems, Electronic point of sale systems, Automatic teller machines etc Typically several systems within same enterprise Read and Update mostly Standard, Predefined, less complex queries Queries based on individual or a relatively less number of records (Single-Hit Queries) Typically used in Tactical Management
13

Decision Support Systems

Decision Support is a methodology (or a series of

methodologies) designed to extract information from data and to use such information as a basis for decision making
Communication Driven DSS
Data Driven DSS Document Driven DSS Knowledge Driven DSS Model Driven DSS

14

Data Driven DSS

15

Online Analytical Processing (OLAP)

Goal of OLAP is to support ad-hoc querying for the

business analyst Multidimensional view of data is the foundation of OLAP Extend spreadsheet analysis model to work with warehouse data
Read Only Access Semantically enriched to understand business terms

(e.g., time, geography) Combined with reporting features

16

OLTP vs. Data Driven DSS

Trait User Function DB Design Data View Usage Unit of work Access Operations Records accessed #Users Db size Metric

OLTP
Sales Staff, IT Professionals Day to day operations Application-oriented (E-R based) Current, Isolated Detailed, Flat relational Structured, Repetitive Short, Simple transaction Read/write Index/hash on primary key Tens to Hundreds Thousands 100 MB-GB Trans. throughput

Data Driven DSS Knowledge worker Decision support Subject-oriented (Star, snowflake) Historical, Consolidated Summarized, Multidimensional Ad hoc Complex query Read Mostly Lots of Scans Thousands to Millions Hundreds 100GB-TB Query throughput, response
17

Data Mining
Knowledge Extraction
Verification: OLAP type analyses, hypothesis testing Discovery: Extracting rules or patterns

Data Mining is finding hidden patterns in data

Predict which customers will buy new policies Identify behavior patterns of risky customers Identify fraudulent behavior Characterize patient behavior to predict office visits Identify successful medical therapies for different illnesses

18

Knowledge Discovery in Databases (KDD)

Non-trivial extraction of implicit, previously unknown

and potentially useful knowledge from data KDD stages

Problem definition Data selection Cleaning Enrichment Coding and organization Data mining Reporting
19

DW and DB Clarifying Confusions

Is DW different from DB No The difference is historical not technical DW is a DB inside and out DW is to Data Driven DSS what DB is to OLTP

20

Brief History of DB Design

Master file design
Integrated, subject-oriented design Relational design Star join design

21