Managing the Data Lifecycle

A key requirement of SAP BW/4HANA is to store data - a lot of data!

Data is collected from all source systems in the organization. Many of those systems generate huge amounts of data on a daily basis. SAP BW/4HANA can even collect data in real-time using streaming mechanisms, and so the flow of data never stops.

From a technical perspective, SAP HANA - the database that powers SAP BW/4HANA - could cope with huge data growth. We would simply scale-up the hardware by adding more memory. Once the server reaches its memory capacity, we would then add more servers. This is called scale-out. But this is a very expensive solution to the data growth problem.

Apart from these increasing costs of storing more and more data in SAP BW/4HANA memory, we are in danger of clogging up the system with data that is rarely used. So let's take a step back and think about the best solution.

Data has a life-cycle. When data is new, it is usually very important to the business and used in decision-making. But as data ages, its value usually decreases as it is used less and less. We should move the less valuable data, from the highest performance / highest cost storage - which is memory - to the lower performing / cheaper storage options - such as disk.

What is needed are tools to manage the data life-cycle so that we can move data from memory to disk when the time is right. Such tools are included in SAP BW/4HANA.

SAP has defined a multi-temperature data storage concept. Data is classified as HOT, WARM, or COLD that aligns to the frequency of data access and the requirement for performance. These criteria include the type of data involved, how useful it is for business purposes, the importance of these processes, frequency of access, performance and security requirements. More facts that influence the setup of the SAP Multi-Temperature strategy for storing data include:

• Budget restrictions that limit spend on hardware

• Technical restrictions regarding the capacity of the SAP HANA database

• Storage of historical data (due to data growth)

• Rules for storing data, such as the requirement to save all data for at least some years for legal reasons

A large portion of data managed in a large enterprise data warehouse such as SAP BW/4HANA, is processed frequently and needs fast access. This kind of data can be considered as HOT.

In addition, there is often also a large volume of data that is not accessed frequently and perhaps might not require fast access. For this reason, this type of data can be defined as WARM. Data that is no longer needed but must still be kept, perhaps for legal reasons, can be classified as COLD.

SAP BW/4HANA data is classified by access frequency and use case into HOT, WARM, and COLD.

In the context of the SAP BW/4HANA reference architecture for data warehousing (Layered Scalable Architecture for BW/4HANA), the various areas in a data warehouse and the different architectural layers of the EDW architecture can be assigned to these multi-temperature data categories.

Let's learn more about these multi-temperature data categories:

HOT

• All layers related to mission-critical, day-to-day business analysis and planning

• Hot data is accessed frequently for reporting and planning purposes, or by regular SAP BW/4HANA processes such as lookups during data loading. Examples of SAP BW/4HANA objects that typically handle hot data, include the following:

  • Data Mart DataStore Objects (advanced)

  • Standard DataStore Objects (advanced)

• No functional restrictions: read and write actions are allowed on this data.

WARM

• All layers related to data acquisition

• Warm data is accessed less frequently. Performance is not the top priority for this data. . and does not have to be permanently stored in the main memory. Examples of SAP BW/4HANA objects that handle warm data include the following:

  • Objects in the Corporate Memory, typically Staging DataStore Objects (advanced).

  • Objects in the Open Operational DataStore layer, typically Staging DataStore Objects (advanced).

• No functional restrictions: read and write actions are allowed on this data.

COLD

• All layers related to the retention of historical data

• Cold data is accessed very rarely or not at all.

• Functional restrictions: this data is used mainly for read only. It can only be made available for reporting by enabling a setting in the query. By default this data is not accessed by queries. Writing to the data is possible in exceptional cases only such as corrections. If reading this type of data is required, expectations regarding performance must be set accordingly.

For implementing multi-temperature data management in SAP BW/4HANA, SAP provides a tool called Data Tiering Optimization (DTO).

The physical storage locations for the various data temperatures are:

• HOT: Data is managed completely in-memory of SAP HANA.

• WARM: Data is managed on SAP HANA disk.

• COLD: Data is managed outside SAP HANA in an SAP IQ database.

In SAP BW/4HANA, the DataStore object (advanced) is the object used to store transactional data. Transactional data is where significant data growth typically occurs. For this reason a DataStore Objects (advanced) is where you set-up Data Tiering Properties:

For each DataStore Object (advanced) you first choose the temperatures which are relevant and whether you would like to manage the individual data temperature by partition, or if the temperature applies to all data in the DataStore Object.

The next step is to define the planned temperature of each DataStore Object (advanced) or each partition of an DataStore Object (advanced). The SAP BW/4HANA Cockpit provides a suitable app for this purpose: DataStore Objects - Manage Data Tiering. The app provides all required functions to define temperature for the DataStore Objects (advanced) partitions or, depending on the general setup, for the entire DataStore Object (advanced), and to execute the data movement across temperatures.

In addition to the manual definition of the temperature of a partition or object, you can also define temperature change rules for selected objects. Rules can be defined for DataStore Objects (advanced) with tiering at partition level and with an SAP time characteristic or a field of data type DATS as the partition field. In the rule, starting from the current date, you can specify a relative time based condition for the various possible temperatures. For example, you could specify a warm temperature for data that is more than one year old, and cold temperature for data that is more than five years old. Using Process Chains to execute the jobs, you can completely automate the movement of data across temperatures.

The last step is to define if the COLD data should be considered in reporting. You do this by enabling the setting in the General properties of the BW Query definition that reads data from the DataStore Object (advanced).