How to Implement Virtual Master Data

This is new type of Master Data InfoProvider possible in SAP BW 7.4.This characteristic can be used with
virtual master data in Virtual Provider.It is possible to use HANA Attribute, Analysis and Calculation

Views to create Virtual Master Data in BW.

Steps to Implement

We are trying to create a Virtual Master Data for Employee Master Data from an Attribute view with following

structure. Note : Field EMP_ID from Attribute view is of type and size VARCHAR and 4.

Start of creation is exactly same as creating any normal/standard Characteristic InfoObject.Virtual Key Figures

cannot be created.
As EMP_ID is of Type and Size VARCHAR and 4 respectively Data Type and Length has been entered

exactly of same type and size.

In Master data / text tab, down below select SAP HANA View as Master Data Access. Select SAP HANA

Package and SAP HANA View

With Master Data is checked Medium Text is selected.

Now click on Propose HANA-Assignments

Select only those fields under Propose Mapping which needs to be added as attributes.EMP_ID and Name will

not be mapped with Attributes so left unchecked.EMP_ID field of HANA View will be mapped with main master

data object and Name will be mapped with Medium Text here.
Now a second pop up will request for selecting corresponding BW InfoObjects.System provides options of

similar InfoObjects. Select One relevant for each.

Attributes will show now in Attributes tab. These are Assigned BW InfoObjects.To see all HANA field – BW

Mapping Assignments and to Map Main Master data Object and Text follow steps as shown next.

Click now on Maintain HANA – Assignments.

Assignment of HANA View Field is not there for Main Master Data Object ZVIRTCH2. We need to add HANA

View field EMP_ID for it.Similarly HANA View field NAME will be mapped with Medium Text.

Fields Added.Fields Transferred. Virtual Master Data ZVIRTCH2 Activated .

Right click and Display data shows Data from Attribute view of HANA.

Creating an ADSO from Open ODS view

The Open ODS View, one of SAP BW 7.4's main enhancements and one of the strategic modeling objects for

SAP BW on HANA in SAP's simplification approach (see SAP BW 7.4 SP8 powered by SAP HANA and

Roadmap | SCN and What’s new in SAP NW BW 7.4 SP8 ).

One of the key features of the Open ODS view is developing BW objects, without having actual data

persistency in SAP BW. The Open ODS View reads data from a remote source, while leveraging BW

functionality such as Master data access when working with InfoObjects. The great thing about this, is that the

SAP BW developer can actually start developing, testing and prototyping on data, before having to worry about

the underlying data model, changes in requirements etc. it really supports agile development and provides

more flexibility in projects.

In a lot of situations though, there will come a certain point in your project where data persistence is required,

and a traditional ETL process will need to come in place. That's where the Advanced DSO (ADSO) comes into

play. From SAP BW 7.4 SP8, it is now possible to actually generate an Advanced DSO from your existing Open

ODS view, inheriting all InfoObject assignments and field information. When used on an SAP ERP Datasource,

it even creates a transformation to that Datasource and a Data Transfer Process.

An example:

I created a basic generic extractor on table AUFK in SAP ECC, and create a simple Open ODS View of type

FACT, without associations to other Open ODS Views or InfoObjects.

After checking and activating, Data preview shows us about 6417 rows (demo system )

Them, in the RSA1 modeling workbench, we go into change mode for this Open ODS View, and select the

'Generate Dataflow' option. Clicking this button opens a dialog with settings for the dataflow generation. We

choose the ADSO name here, and we can choose between source object data types for the fields, or for BW

types. Since both are ABAP systems in this case, we just go for source system types.

After successful completion of the process, we now have an Advanced DSO and corresponding data flow

leading up to this ADSO! (Infopackage has to be created manually)

Loading and activating data shows us the same 6417 data records we had using the Open ODS View, but now

persistent in BW. The data is exactly identical to the data we previewed with the Open ODS View earlier:

Tools for migrating BW to HANA (BoH)

SAP NetWeaver Business Warehouse powered by SAP HANA or BW on HANA (BoH) is

around for some time already. It was introduced in 2nd quarter of 2012 and meanwhile
it got a lot of attention. Basically BW is usual choice when companies are trying to
evaluate HANA. It is natural that first they just migrate to HANA as database and after
some time they start to implement new project using new BW objects leveraging HANA
functionalities. Or starting of re-implementing current BW data models to use HANA can
happen. Anyway BW is getting a lot of attention as first system to be migrated to HANA.
It may be very tricky for people who are new to HANA even start with basic evaluation
of what does it mean to migrate to HANA. To support tasks like this SAP is providing a
few tools. In further text I try to list the tools and provide basics information about them.

Tools are embraced by central tool called – BW Migration Cockpit for SAP HANA.
From the cockpit other tools for migrating BW systems to HANA are accessible.

Task Type ABAP report / Tcode Docu Description

N/A ZBW_HANA_MIGRATION_COCKPIT 1909597 BW Migration

Cockpit

Check ZBW_HANA_CHECKLIST 1729988 Automation of BW

checks for

ZBW_HANA_CHECKLIST_3x - prerequisites for

version for BW 3.x HANA migration

and guidelines for

operation

Tcode: RSRV link BW Objects

Checks -

consistency checks

on the data and

metadata stored in
a BI system
 RSPLS_PLANNING_ON_HDB_ANALYSIS 1824516 Determines

whether a

planning function

is executed in

ABAP or in HANA

Size /SDF/HANA_BW_SIZING 1736976 Sizing Report for

BW on HANA

Downsizing Tcode: RSDANLCON link Customizing: Set

Configure:NLS Up Near-Line

Storage

Connections

Downsizing Tcode: RSDAP link Setup of Data

Configure: SAPLRSDA_DAP_MAINTAIN Archive Process in

Data Archive order to archive

Process BW data in cubes

and DSOs

Downsizing Tcode: SARA link Execute archive

Execution SAPMAADM processes to

reduce system size

- Data Archiving

(ADK)

House keep - Tcode: STC01 1614266 Executes routine

Automation task list: 1707321 housekeeping

SAP_BW_HOUSEKEEPING activities; to be

run on a regular

basis

House keep - Tcode: SLG2 N/A Application log

Tasks ABAP: SBAL_DELETE deletion of expired

logs

RSDDSTAT_DATA_DELETE 934848 BW statistics

deletion

Tcode: SARA N/A IDoc archiving

 RSBTCDEL N/A Job log deletion

Tcode: BPS_STAT0 540398 Planning Statistics

UPC_STATISTIC3 990000 Deletion

Tcode: RSREQARCH 2092315 Management of

SAPLRSREQARCH archive requests

N/A 706478 List of large

system tables

Migrate - Pre Tcode: /ASU/START 1000009 Tasks to be

Migration /ASU/ASUSTART performed before

Tasks starting upgrade

or migration to

SAP HANA

SMIGR_CREATE_DDL 1908075 Generate DDL

statement for

migration

Migrate - Tcode: STC01 Task Manager –

Automation task list: SAP_BW* execute task list

Tcode: STC02 Task Monitor – to

task list: SAP_BW* solve errors

occurred running

Task List

Tcode: SNOTE link Notes Analyzer -

Z_SAP_NOTE_ANALYZER application to
check the pre-

note requisites for the

assistant usage of BW PCA

(Post Copy

Automation) and

Housekeeping.

Allows to list Notes

to be

implemented.
 1614266 Basis XML – XML

file contains Basis

Notes list

1707321 BW XML - XML file

contains Basis

Notes list

Warehouse Tcode: RSMIGRATE link Migrating Update

Mngt SAPLRSMIGRATE_FRONTEND Rules, 3.x

InfoSources and

Transfer Rules

ZBW_TRANSFORM_FINDER 1908367 BW

ZBW_TRANSFORM_FINDER_3X - Transformation

version for BW 3.x Finder - check for

certain types of

transformation

rules

LogSys: RS_LOGSYS_ACTIVATE Object Activation

IO: RSDG_IOBJ_ACTIVATE tool – collection of

DSO: RSDG_ODSO_ACTIVATE several tools for

IC: RSDG_CUBE_ACTIVATE activation of

HybridProv: RSDG_HYBR_ACTIVATE different BW

InfoSet: objects

RSQ_ISET_MASS_OPERATIONS

MProv: RSDG_MPRO_ACTIVATE
DS:

RSDS_DATASOURCE_ACTIVATE_ALL

IS: RS_COMSTRU_ACTIVATE_ALL

Transfer Rules:

RS_TRANSTRU_ACTIVATE_ALL

Update Rules:

RSAU_UPDR_REACTIVATE_ALL

TRFN: RSDG_TRFN_ACTIVATE

DTP: RSBKDTPREPAIR

BEx BEx Query Designer Link Starts BEx Query

Designer
 Tcode: RSRT Link Starts BEx Query

Monitor

Tcode: RSRTQ N/A Starts BEx Query

RSRQ_QUERYDEFINITION Definition

BEx Web Application Link Starts BEx Web

Application

Web Item Conversion 832712 Starts Web Item

Conversion

BEx Analyzer 1901268 Starts BEx

Analyzer

RSZW_ITEM_MIGRATION_3X_TO_70 Link Starts Workbook

Conversion

Security RSECADMIN link Maintenance of

- analysis

Authorization authorizations

objects

SAPLRSEC_MIGRATION link Conversion of 3.x

reporting

authorizations to

7.x analysis

authorizations

Security Tcode: PFCG link Maintenance of BW

- Roles Roles

Optimize ZBW_ABAP_ANALYZER 1847431 BW ABAP Routine

ZBW_ABAP_ANALYZER_3X - version for Analyzer - Scan

BW 3.x custom ABAP

routines used in:

process chain,

transformation,

transfer rule

(IS/DS/SS),

update rule (IC,

IO, IS), planning
 area/level/function

and ADP.

Tcode: SCI link ABAP Code

Inspector

Looking forward to hear from you what is your experience using these tools.

Upgrade was never been easier ...

With the Release of the current SL toolset 1.0 SP12 - SL Toolset 1.0 SPS 12: improved Software Logistics

Tools there is now a new option for ABAP and JAVA based Upgrades available.

Already introduced with the database migration option (DMO) the new SAPUI5 based Frontend allows even

more possibilities when it comes to the monitoring of any Upgrade process despite the underlying database
- SUM: New SL Common UI available for AS ABAP scenarios

This Graphic shows the old and new UI access:

All common Browser types are supported, i.e. IE 11, Google, Firefox, Safari in their latest Versions.

A possible usage on mobile devices with the same Browsers should work as well, as it is HTML5
based access to the Backend.
one of the mayor advantages is the fact, that the SAPHostAgent which is part of every SAP based
Installation takes care of the complete communication between the SAPup and the Frontend without
having an online Window to the OS open.

Various Options allow a flexible usage of the new SUM UI in the usage of the upgrade
process.

Different colors show individual process steps with different importance or errors
Additional possibilities like access to important log files can be accessed via "tail
mode" independent to the used Backend OS.
See also the SAP Note 2107392 - SL Common UI for AS ABAP scenarios in SUM for additional
technical background and the current participation on the Pilot usage.

The “advanced” DataStoreObject – renovating BWs persistency layer

With BW7.4 SP08 another “feature package” was delivered with a lot of new functionality and optimizations based

on the HANA platform (seehttp://scn.sap.com/docs/DOC-35096). In accordance with the BW strategy to deliver

the HANA-based innovations and renovations non-disruptively to our customers the SP08 delivered a first step

in renovating our Persistence Layer. With the possibilities of HANA we are able to consolidate all BW

InfoProviders with persistent data into a single object with a single and simple data base layout. This is similar to

the consolidation and simplification in the Virtual Layer based on the CompositeProvider.

Since its properties, table layout and services are closest to today’s DataStoreObject we simply kept the name.

Only if we need to explicitly distinguish between the current object and the new object we add “advanced”

DataStoreObject or “classic” DataStoreObject. This also emphasizes the role of BWs DataStoreObject as THE

central building block for a Data Warehouse architecture.

Major features

New intuitive Eclipse-based modeling UI

The UI renovation continues. The ultimate goal: bring the heavy-weight modeling UIs to the Eclipse-world and

the administration and monitoring UIs to the Web. With the Eclipse-UI for the advanced DSO it is now possible

to model all InfoProviders of a complete end2end scenario in the BW Modeling Tools (from an OpenODS View,

to the advanced DSO, CompositeProvider and BW Query). All this is deeply integrated and aligned with the

HANA Studio and Modeler for all “mixed-case scenarios” and a unique modelling experience.

Combine field- and InfoObject-based modeling

The “higher” up in your architecture an InfoProvider is located the more important are the rich features of BWs

masterdata modeled in an InfoObject (like re-usability, data consistency, visibility, and quality). But for new

scenarios the InfoObject approach is a high hurdle, since it requests a top-down approach right from the start.

With the Open ODS View we introduced in BW7.4 SP05 already a virtual field-based approach to integrate data

without the need of having InfoObjects. With the advanced DSO this is now also possible for the Persistence

Layer. I.e. you can load data into BW without assigning InfoObjects to each and every field without losing

functionality.

In combination with the Open ODS View this approach allows you to integrate new data easily and fast, switching

seamlessly from an e.g. even virtual data access to a managed persistence with an increasing level of data

integrity. All this especially tabbing into “non-classic BW data” with the support additional data types and adapters.

High frequent data loads – based on optimized request

management

The request management is, if you like, the cornerstone of BWs managed approach to Data Warehousing since

it organizes availability and status of the data from the time it is requested of the source system up to the caching

by the Analytic Manager (aka OLAP Engine) or the delta buffers of the integrated planning. With the size of the

BW systems growing and growing, the complexity increasing and the demand for more and more (close-to-) real-

time scenarios we decided to re-write our request- and process-management for the advanced DSO.

The current request management still works for the “classic” objects (DSOs, InfoCubes, …) and a single dataflow
can work with both types as well. The “new request ID” (internally called “TSN” – Transaction Sequence Number)

is no longer an INT4 value, but a timestamp plus a generated, increasing postfix. This not only removes the 2
billion limitation (for the system-wide number of requests), but also allows for new logic and new semantic derived

directly out of the request. The new request management comes with a new “manage UI” directly accessible from

the Data Warehousing Workbench that allows you to quickly navigate through very large sets of requests and

protocols and perform manual tasks or monitoring activities.

Reporting performance optimizations

With the goal to replace the InfoCube we also had to make sure that some of the more complex (or “exotic”)

reporting scenarios perform well. Two features are important to achieve this goal. Only in case of an INNER JOIN

between the table with the facts and the masterdata table, can HANA perform optimal and BW can push OLAP

operations to HANAs Calculation Engine. The check whether or not an INNER JOIN can be used (instead of a

LEFT OUTER JOIN) is the BW referential integrity check during loading or activating the data (aka SID-creation,

but is much more than just determining an INTEGER value). This check exists in the classic DSO as well (the

so-called “BEx-flag” – the “create SIDs” option). But for the advanced DSO it is possible to set this flag not only

on InfoProvider level, but individually per characteristic of the DSO. The data integrity check then is only executed

on the selected characteristics.

The second feature is also related to the JOIN with the masterdata tables. Also HANA benefits from the fact that

for an InfoCube JOINs to the masterdata tables are via INT4 values, compared to STRING JOINs for a DSO. In

rare cases the performance difference can be crucial, e.g. if you have a compound, high-cardinality InfoObject

like 0MAT_PLANT and the reporting mostly includes navigational attributes of this InfoObject and therefore forces

this JOIN to be executed very often. In such cases you can, for individual InfoObjects(!), turn on the persistency

of the InfoObject-SID in the DSO tables. An additional column will then be created in the active data table to hold

the SID and will be filled during data activation.

And More
Just to mention a few additional highlights: Up to 120 key fields are supported. Template-based design using

references to classic BW InfoProviders types or best-practice LSA++ templates. Integrated Remodeling for

structural- or type-changes. DB-partitioning on all key fields. … and so on …

Positioning

The advanced DSO will be the central persistency object in BW-on-HANA replacing especially InfoCube, classic

DSO, HybridProvider and PSA. While there are still some gaps to cover the complete functionality, we

recommend considering the advanced DSO for all new projects as the central (and only) persistency object. A

widespread conversion of existing scenarios into advanced DSO is currently not recommended, but should be

done only case-by-case and demand-driven. We plan to provide a conversion tool to support the conversion of

objects in future SPs. The current persistency objects are of course still supported and do co-exist with the

advanced DSO. There are no plans to change this.

Roadmap

New minor features will be added and some gaps will be closed in subsequent SPs (SP10, SP11, SP12) – a list

can be found attached to note2070577. Another “feature package” of the BW7.4 release is planned with SP13

(scheduled for Q4 2015). This “feature package” shall then close all major gaps between today’s functionality for

InfoCubes, DSOs, PSAs and the advanced DSO. Additionally the advanced DSO is planned to support several
new features like Dynamic Tiering (fka ExtendedStorage) support, direct- and Bulk Load enablement, … . And a

conversion tool will then also allow you to convert existing InfoProviders into an advanced DSO.
Availability

Please check note 2070577 for the details according to your support package level. Additional and detailed

information can be found in the SAP online help.

SAP BW 7.4 - Feature Overview and Platform Availibilty

Please find a list of all major features delivered with SAP BW 7.4 below. This list will be enhanced
and updated with each new SAP BW Support Package accordingly. It is intended in addition to the
overview SAP Notes for OLAP functions, planning features: and BW Modelling Tools:

 SAP Note 2063449 - Push down of BW OLAP functionalities to SAP HANA

 SAP Note 1637199 - Details about availability of pushed down Planning

functionality

 SAP Note 1905207 - BW Modeling Tools for SAP BW on HANA

To see all features of BW 7.4 please refer to the release notes.

Feature
Type HANA only Available since

HANA-optimized BW Business Content Enhancement Yes BW 7.3 SP5

XXL-Attributes (characteristics values <= 255, long
texts with 1333 char) Enhancement No BW 7.4 SP5
High-Cardinality InfoObject (SID-less InfoObject) Enhancement Yes BW 7.4 SP5
BW Modeling Tools in Eclipse (Composite Provider,
Open ODS View… ) New Feature Yes BW 7.4 SP5
Availibity of the new CompositeProvider 7.4 New Feature Yes BW 7.4 SP5
HANA Model Generation for BW InfoProvider New Feature Yes BW 7.4 SP5
Virtual Master Data (InfoObjects based on
Calculation View) New Feature Yes BW 7.4 SP5
Inventory Keyfigures for DSO, VirtualProvider,
CompositeProvider Enhancement Yes BW 7.4 SP5
OLAP: Calculation push-down Optimization Yes BW 7.4 SP5

OLAP: Stock coverage keyfigure New Feature Yes BW 7.4 SP5

OLAP: FIX operator New Feature No BW 7.4 SP5
 Feature
Type HANA only Available since

OLAP: Multi-dimensional FAGGR New Feature No BW 7.4 SP5

OLAP: Current Member New Feature No BW 7.4 SP5

PAK enhancements & optimizations Optimization Yes BW 7.4 SP5

Planning on local provider in BW Workspace New Feature Yes BW 7.4 SP5
Planning function push-down Optimization Yes BW 7.4 SP5
Planning: ODATA & Easy Query extensions New Feature No BW 7.4 SP5
Planning: Support of HANA views for facts and
master data New Feature Yes BW 7.4 SP5
Availibilty of Open ODS Views New Feature Yes BW 7.4 SP5
Availibilty Support of Smart Data Access New Feature Yes BW 7.4 SP5
Availibilty HANA Analysis Process New Feature Yes BW 7.4 SP5

HANA optimized Transformations New Feature Yes BW 7.4 SP5

Open Hub: Push data into a connected database New Feature No BW 7.4 SP5
Operational Dataprovisioning - PSA becomes
optional New Feature No BW 7.4 SP5
Operational Dataprovisioning - ODQ for SLT New Feature Yes BW 7.4 SP5
Operational Dataprovisioning - DTP for HANA Views New Feature Yes BW 7.4 SP5
Operational Dataprovisioning – Data Services
Integration New Feature No BW 7.4 SP5
Data request housekeeping and monitoring Enhancement No BW 7.4 SP5
Monitoring integrated in DBA cockpit for Sybase IQ Enhancement No BW 7.4 SP5
Optimized Query-access to NLS data in Sybase IQ
leveraging SDA Optimization No BW 7.4 SP5

BW Workspace enhancements: Data Cleansing New Feature Yes BW 7.4 SP5

Re-Modeling Toolbox Enhancements Enhancement Yes BW 7.4 SP5
New WebDynpro-based Masterdata Value
Maintenance Enhancement No BW 7.4 SP5
HANA-optimized BW Business Content (Data flow
optimizations) Enhancement Yes BW 7.4 SP5
 Feature
Type HANA only Available since

New source object types for OpenODS Views

(Transformations, ADSO) Enhancement Yes BW 7.4 SP8
Automated Data Flow generation for OpenODS
Views Enhancement Yes BW 7.4 SP8
Support for UNION CompositeProviders in BW-IP
scenarios Enhancement Yes BW 7.4 SP8
Stacked UNION CompositeProvider Enhancement Yes BW 7.4 SP8
CompositeProvider Joins with OpenODS Views Enhancement Yes BW 7.4 SP8
Availiblity of Advanced DataStore Objects
(see also SAP Note 2070577 for more details about
functions and gaps) New Feature Yes BW 7.4 SP8
HANA model generation for BW Queries Enhancement Yes BW 7.4 SP8
Generated SAP HANA model able to read the data
from Nearline Storage Enhancement Yes BW 7.4 SP8
ABAP managed database procedure (AMDP) in
Transformations New Feature Yes BW 7.4 SP8
Availiblity of SAP HANA dynamic tiering for SAP BW New Feature Yes BW 7.4 SP8
Enhanced NLS operations (Archiving Proposals,
Automated DAP generation, NLS DBA Cockpit
integration) Enhancement No BW 7.4 SP8
Query pruning for NLS InfoProvider Enhancement No BW 7.4 SP8

NLS support for non cumulative Key Figures Enhancement No BW 7.4 SP8
XXL – Attributes (Mime Types, long strings) Enhancement No BW 7.4 SP8
BW Search (new eclipse UI, optimized for HANA) Enhancement Yes BW 7.4 SP8
ODP Support of hierachy loads Enhancement No BW 7.4 SP8
Re-modelling Tool Box support of SPOs Enhancement No BW 7.4 SP8
Process Chain Monitor (SAP UI5 mobile app) New Feature No BW 7.4 SP8
BW 7.4 SP9
Query in BWMT in eclipse New Feature Yes
Efficiently managing your data in SAP BW on HANA (Dynamic Tiering, NLS)

There is some confusion around the different options available for managing data in BW. Hence I
am writing this to ease that confusion and hopefully achieve a clear understanding of what options
are available and which are best suited for what purpose.

In a typical non-HANA environment most customers will retain all of their data in SAP BW and some will retire

using tapes or disk or using a near-line storage solution with a secondary DB.

When it comes to running SAP BW on HANA, the cost to putting all data in RAM in HANA can be high if the

volumes are large. Moreover, not all the data needs to be in-memory because typically in an organization only

30-50% of the entire BW data is really used very actively for reporting and other operations and hence they are

the ideal candidates to fully utilize the in-memory capabilities of HANA. The other 50-70 % portion of the data is

infrequently accessed and hence can be managed in a low cost plan.

SAP BW on HANA offers a number of ways to manage these different data temperatures so you can achieve

an overall lower TCO for your investment. For customers this becomes an interesting area because it

encourages the adoption of an archiving policy which when managed and maintained efficiently can limit the

need to buy more HANA and thus saving heavy Opex costs.

Broadly there are 3 types of data temperatures -

HOT DATA

This is the area where 100 % of your PRIMARY IMAGE DATA is in the HANA in-memory space (RAM) and is

instantly available for all operations.

In the BW world, this is typically the InfoCubes and Standard DSOs as they constitute the reporting and

harmonization (EDW) areas respectively as show below. They are very frequently accessed for reporting and

harmonization purposes and hence is the ideal candidates for being fully in-memory and to fully benefit from

the HANA capabilities.

Although the frequency is fast, the data that is typically accessed very frequently for reporting purposes is

between 2-3 years old. Hence this portion of the most recent accessed information is the real hot data that

needs to be in-memory all the time to deliver top level performance.

The older data (typically beyond 3 yrs.) are rarely accessed for reporting but are still required for to be retained
for regulatory and compliance purposes. Hence these older data can be safely archived to a very low

cost plan using the COLD DATA management option using SAP IQ as explained in the next section.

The data in the PSAs and w/o (write optimized) DSOs constitute the staging area and corporate memory.

Although they require frequent access, they tend to be used primarily for non-reporting purposes, i.e. for data

propagation and harmonization. Hence they can be moved to a WARM store area, which is explained in the

next section.

The below diagram shows the areas where the HOT, WARM and COLD concepts will apply in a typical SAP
BW EDW architecture.
 Access VERY FREQUENT OPERATIONS, that run every few seconds, to every minute to

every hour

Response REALLY FAST, Fully in-memory

Time

Use case To provide fast access - To queries, data loading, data activation, data

transformation and data look-ups

Likely RECENT DATA from InfoCubes, Standard DSOs, Open DSOs and All Master Data

candidates and Transformations and related look-up DSOs.

COLD DATA

In the context of this document I am only discussing SAP IQ as the cold storage, whereas with BW there are

other certified partners who are providing Near-Line Storage solutions such as PBS Software and DataVard.

You can look up for “NLS” from the partner site at - http://go.sap.com/partner.html

This is the area where 100 % of your PRIMARY IMAGE DATA is in a SECONDARY DATABASE (ON DISK)

and the response is slightly slower than HANA but still offers reasonably fast READ ONLY access to data for

reporting purposes, as if they were in one database.

In the BW world, the standard DSOs & InfoCubes constitute the harmonization and reporting layers. But

typically only the last 2-3 years of data is the most frequently requested. The older data (typically beyond 3 yrs.)

are very in-frequently accessed for reporting but are still required for to be retained for in-frequent reporting or

regulatory and compliance purposes. Hence these older data can be safely archived to a very low cost plan.

This is where the NLS comes into play. Keeping the existing models and architecture the same, you can

remove the older sets of data from these Infoproviders (typically slicing the data according to time dimensions

or completely moving the entire data) out from the primary HANA database and move it to a secondary low

cost/low maintenance IQ database. The READ access to IQ NLS is in most cases is much faster than READ

access to traditional databases. For customers running BW on xDB and using IQ as NLS, the NLS DB actually

turns into an ‘accelerator’ and provides much faster response times than the primary database.

The NLS4IQ Adaptor in SAP BW offers tight integration between SAP BW and SAP IQ, such that all data

management, retiring and control processes can be done through SAP BW using the Data Archiving Process

(DAP). A lot of new enhancements have been recently added with the BW 7.4 SPx releases that help to

manage the entire end-to-end archiving life cycle process in a much more simpler and efficient way.

Talking about SAP IQ, it offers columnar tables for faster read access, upto 90% compression and runs on a

conventional hardware, thus offering overall lower TCO benefits plus it is a highly mature database with a large

install base for the past 15+ years. Hence it is a trusted environment to retire old data as a low cost/low

maintenance DB option but still have all the benefits of accessing it in near real-time whenever needed or

requested.
Also for historical data the SLAs are usually not the same as the high availability data and hence the NLS

process helps by moving the bulk of the inactive data out of the primary database to a slightly relaxed SLA

environment. Secondly what NLS is providing is an on-line archiving solution, so as the volume grows and data

gets older, they can be seamlessly moved out of the primary HANA database. This way you can reduce the

OPEX costs by significantly reducing the need to buy more HANA, thus reducing the TCO of the landscape

dramatically.

Access SPORADIC, typically data that is older than 2-3 years but is still required for

reporting purposes either regulatory or statistical or compliance.

Response TYPICALLY 5-10 % less than HOT store.

Time

Use case This is used for data retiring purposes where you REMOVE part of your DATA

(HISTORIC DATA) from your PRIMARY STORAGE and MOVE to a low cost

database, typically generating an archiving scenario, but still making the data

available anytime and anywhere with near real-time access as on request.

Likely HISTORIC DATA from InfoCubes, Standard DSOs, and SPOs.

candidates
WARM DATA

This is the area where the PRIMARY IMAGE DATA is in the DISK storage of HANA Database instance, but is

always available on request. Using this you can manage your LESS RECENT DATA and LESS FREQUENT
DATA more efficiently within the HANA database such that data is instantly available for READ, WRITE,

UPDATE etc (all operations), but still offers the lower TCO benefits.
In the BW world, PSAs and W/O Optimized DSOs constitute the staging area and corporate memory area. The

value of the data in the PSAs is good as long as it is the newest data. Once it is loaded to the upper level

targets then the value of that data diminishes and is only required if there are discrepancies in the report results

and a trace back/reload is required. Although some customers do maintain a regular housekeeping, the PSAs

persists the data for a few days to few weeks to few months, depending on the SLAs and policies. Hence their

size can grow very quickly thus blocking a lot of space and memory which otherwise could have been used for

other important processes. Similarly with corporate memory, they are primarily used to support the

transformations, harmonisations, reconstructions etc.; hence their usage is only required when such activities

are taking place.

There are 2 options to do the WARM concept –

1. Non-Active Concept

The Non-active concept is available since SAP BW 7.3 SP8 and is primarily used to efficiently manage the

available HANA memory space.

This concept primarily applies to PSAs and W/o DSOs. The PSAs and W/O DSO are partitioned by data

request which means that the complete datarequest is written to a partition. Once a threshold value is

exceeded for the number of rows of a partition then a new partition is created. The default threshold value for

PSAs is 5 Million lines and for write-optimized DSOs it is 20 Million lines.

Using the non-active concept the PSAs and W/o DSOs can be classified as low priority objects, so whenever

there is a shortage of memory, only the older partitions containing the inactive data are quickly displaced from

memory, thus making room for other high priority objects/processes to use the freed memory. The new/recent

partition of the PSAs and the w/o DSOs are never displaced from memory and they always remain in memory

for operations that are required as part of the data loading process.
Although the concept can be applied to InfoCubes and Standard DSO, but it is a HIGHLY UNRECOMMENED

OPTION. Please check SAP Note1767880. Since cubes and standard DSOs are not partitioned by request, the

concept of making them low priority objects and displacing and reloading them does not work efficiently in

these cases. As they can hold large volumes of data, whenever a load or activation is requested the entire table

has to be brought back to memory and this will result in drop in performance. For these Infoproviders, it is ideal

to keep either ALL of their data in HOT OR to keep the newer data in HOT and move the older data sets to a

COLD STORE like IQ using NLS concept.

Access MEDIUM FREQUENT DATA

Response REALLY FAST, if all partitions are in-memory.

Time

If the data is displaced from the partitions and require a reload back to memory then

there is considerable lag depending on the volume of data and the infrastructure

strength. This is one of the key reasons why non-active concept is not a highly

recommended in a very active data warehousing solution, as pulling the data back

into memory from disk has negative implications in performance.

Use case To efficiently manage the low value data in the HANA in-memory space for PSAs &

w/o DSOs, and retain the available HANA memory footprint.

Likely PSAs and W/O DSOs only.

candidates

Some considerations -

* Non-active concept is not a way to retire or store your older data into a low cost plan, but rather it is a way to

more efficiently manage the limited available memory so that when the higher priority objects/processes require

memory the lower priority objects are displaced and memory is made available to do higher priority tasks.

*The non-active concept works only when there is a memory shortage. This means that the entire PSA & w/o

DSO will always be in-memory unless there is a shortage during which ONLY the older partitions are flushed

out of memory to disk, but still always retains the recent/new partition in memory.

* If the data is displaced from the older partitions and later if some BW processes requires the data then these

older partitions are reloaded back to memory. This causes considerable lag depending on the volume of data
and the infrastructure strength. This is one of the key reasons why non-active concept is not a highly
recommended option in a very active data warehousing environment, as pulling the data back into memory

from disk has negative implications in performance.

*The non-active concept does not reduce the in-memory space as the objects still occupy the required space.

If there are large numbers of such objects then it can result in a significant blockage of the HANA memory. This

is one of the main reasons why we have the Dynamic Tiering solution.

2. Dynamic Tiering

The Dynamic Tiering is ONLY available for SAP BW 7.4 SP8 onwards and HANA 1.0 SPS 9 onwards and

currently only applicable to PSAs, W/O DSOs and in the future support for Advanced DSOs.

If we recall the non-active concept, it only works when there is a memory shortage which means that the entire

PSA & w/o DSO will always be in-memory unless there is a shortage during which ONLY the older partitions of

these objects are flushed out of memory to disk. This means that the recent/new partition will always be in

memory and thus will occupy some space. Also whenever the older partitions need to be accessed by any BW

operations they are brought back to memory thus occupying more space. So effectively, this concept occupies

space in the HANA memory at all times and there is a risk that if this concept is over utilized then it could result

in slower performance and impact other processes.

Dynamic Tiering is very different to what the non-active concept offers. In the DT concept, all data of a PSA

and w/o optimized DSO is 100% on disk; which means that the entire image of the object is in the PRIMARY

disk. There is no concept of high priority objects and displacement mechanism. This is effectively keeping the

entire data of these objects in a separate low cost area but at the same time offering an integrated mechanism

to access them whenever required with optimal performance.

The tables in the DT concept are called extended tables (ET) and they sit in a separate warm store “host” on

the same storage system as shown in the below diagram. Logically the ET tables are located in the SAP HANA

database catalog and can be used as if they were persistent SAP HANA tables. These tables are physically

located in disk-based data storage however, which has been integrated into the SAP HANA system. The user

sees the entire system as one single database and the persistence of data written to the extended table is

hard-disk-based and not main-memory-based. Any data written to an extended table is written directly to the

disk-based data storage.

DT offers a single consolidated way of managing the less frequent and less critical data in a very low cost

manner and still giving the same level of performance as the hot store. This is possible because the DT uses

the main memory for caching and processing thus offering in-memory performance benefits and also the data

in the warm memory is accessed using algorithms, which are optimized for disk-based storage; thus allowing

the data to be stored in disk. All the data load processes and queries are processed within the warm store and

it is transparent for all operations and hence no change for BW processes are required.

Unlike the concept of Non-active, the main memory in SAP HANA is not required for data persistence (in

extended tables). The concept of Dynamic Tiering can optimize the main memory resource management even

further than the concept of Non-active data by completely moving the staging area data from the hot store to a
separate low cost warm storage. This has a positive effect on hardware sizing, especially when dealing with a

large quantity of warm data in the PSAs and write-optimized Data Store objects.
 Access MEDIUM FREQUENT DATA

Response
Medium Fast. Slightly lower performance than HOT store
Time

Use case To efficiently manage the low value and low frequent data in the HANA in-memory

space and overall offer significantly lower HANA memory footprint

Likely PSAs, W/O DSOs and Advanced DSOs only.

candidates

*Currently there are certain limitations of using Dynamic Tiering in a true Data Centre operation because of the

limited scope of Disaster Recovery and limited automation for High Availability, but this is intended to be made
available with HANA SP10.

Summary

When you look at the 2 warm options; Non-active concept and Dynamic Tiering concept, the non-active

concept has overheads in terms of HANA memory sizing and could result in performance drawbacks if over

utilized; whereas the Dynamic Tiering concept mostly replaces the non-active concept by allocating a dedicated

disk based storage to endlessly manage the big volumes at a very low cost plan but still delivering optimal

performance as in-memory.

As with Dynamic Tiering, it is an area that has the current data and demands frequent access and does all

normal HANA operations (READ, WRITE, CREATE, UPDATE, DELETE etc). The DT concept works on

differentiating between the less critical layers and the most critical layers of the EDW; effectively giving a

dedicated storage for the less critical layers but still managing it as one integral part of the solution.
As for the COLD storage, it is quite clear that it is an area which demands very sporadic READ only access and

is ideally an on-line archiving solution that retains and maintains historic information at a very low cost plan.

The NLS concept works on differentiating the new data and the old data; effectively moving the old data to a

low cost COLD storage solution but still maintaining the tighter integration with the the primary database and is

always on-line for reporting.

So where are the savings? Let’s quickly look at an example below;

Let’s assume customer ABC need a 1TB BW on HANA system to migrate their current BW on DB system. If

ABC retains all that data in HOT then they will need to license 1 TB of HOT store licenses and 1 TB of HANA

hardware. As the volumes and requirements grow there will be a further need to invest in additional HOT

licenses and additional HOT memory hardware.

SAP BW on HANA Solution = SAP BW on HANA

Instead if we apply the WARM/COLD concepts and enforce a proper data management policy, then we can

split the data according to usage/value/frequency and maintain them in a low cost storage solution. If we

assume a 20:40 split for WARM/COLD, then the requirement for HOT store reduces to merely 40%. So as

volumes and requirements grows, the low value/low usage/low frequency data can be pushed directly to the

low cost storage systems without even impacting the HOT storage; thus avoiding the need to invest in any

further HOT storage licenses or hardware.

SAP BW on HANA solution = SAP HANA (HOT) + Dynamic Tiering (WARM) + SAP NLS IQ (COLD).

So effectively SAP is offering a fair proposition with different components that complements each other and fits

well into the EDW architecture of SAP BW running on SAP HANA; thus providing an efficient way of managing

different data temperatures depending on their usage, value and frequency.

S/4HANA and Data Warehousing

One of the promisses of S/4HANA is that analytics is integrated into the [S/4HANA] applications to bring

analyses (insights) and the potentially resulting actions closely together. The HANA technology provides the

prerequisites as it allows to easily handle "OLTP and OLAP workloads". The latter is sometimes translated into

a statement that data warehouses would become obsolete in the light of S/4HANA. However, the actual

translation should read "I don't have to offload data anymore from my application into a data warehouse in

order to analyse that data in an operational (isolated) context.". The fundamental thing here is that analytics is

not restricted to pure operational analytics. This blog elaborates that difference.

To put it simple: a business application manages a business process. Just take the Amazon website: it's an

application that handles Amazon's order process. It allows to create, change, read orders. Those orders are

stored in a database. A complex business (i.e. an enterprise) has many such business processes, thus many
apps that support those processes. Even though some apps share a database - like in SAP's Business Suite or

S/4HANA - there is usually multiple databases involved to run a modern enterprise:

 Simply take a company's email server which is part of a

communications process. The emails, the address book, the traffic
logs etc sit in a database and consitute valuable data for analysis.
 Take a company's webserver: it's a simple app that manages access
to information of products, services and other company assets. The
clickstream tracked in log files constitutes a form of (non-
transactional) database.
 Cash points (till, check-outs) in a retail or grocery store form part of
the billing process and write to the billing database.
  Some business processes incorporate data from 3rd parties like
partners, suppliers or market research companies meaning that their
databases get incorporated too.

The list can be easily extended when considering traditional processes (order, shipping, billing, logistics, ...)

and all the big data scenarios that arise on a daily base; see here for a sample. The latter add to the list of new,

additional databases and, thus, potential data sources to be analysed. From all of that, it becomes obvious that

not all of those applications will be hosted within S/4HANA. It is even unlikely that all the underlying data is

physically stored within one single database. It is quite probable that it needs to be brought either physically or,

at least, logically to one single place in order to be analysed. That single place hosts the analytic processing

environment, i.e. some engines that apply semantics to the data.

Now, whatever the processing environment is (HANA, Hadoop, Exadata, BLU, Watson, ...) and whatever

technical power it provides, there is one fundamental fact: if the data to be processed is not consistent,

meaning harmonised and clean, then the results of the analyses will be poor. "Garbage in - garbage out"

applies here. Even if all originating data sources are consistent and clean, then the union of their data is

unlikely to be consistent. It starts with non-matching material codes, country IDs or customer numbers,

stretches to noisy sensor data and goes up to DB clocks (whose values are materialised in timestamps) that

are not in sync - simply look at Google's efforts to tackle that problem.

In summary: while analytics in S/4HANA is operational, there is 2

facts that make non-operational (i.e. beyond a single, isolated
business process) and strategical analyses challenging:
1. It is likely that enterprise data sits in more than 1
system.
2. Data that originates from various systems is
probably not clean and consistent when being combined.

A popular choice to tackle that challenge is a data warehouse. It has the fundamental task to expose the

enterprise data in a harmonised and consistent way ("single version of the truth"). This can be done by

physically copying data into a single DB to then transform, cleanse, harmonise the data there. It can also be

done by exposing data in a logical way via views that comprise code to transform, cleanse, harmonise the data

(federation). Both approaches do the same thing, simply at different moments in time: before or during query

execution. But, both approaches do cleanse and harmonise. There is no way around. So, either physical or

logical data warehousing is a task that does not go away. Operational analytics in S/4HANA cannot and does

not intend to replace the strategical, multi-systems analytics of a physical or logical data warehouse. This
should not be confused by the fact that they can leverage the same technical assets, e.g. HANA.
On purpose, this blog has been neutral to the underlying product or approach used for data warehousing. This

avoids that technical product features are mixed up with general tasks. In a subsequent blog, I will tackle the

relationship between S/4HANA and BW-on-HANA.

This blog has been cross-published here. You can follow me on Twitter via @tfxz.