Identifying Delta Process Properties

Generally, there are two ways to extract data from a source:

• Full load: There are sources, such as flat files, in which it's unknown which data has changed. The only way to get an update is to extract the available content, again, in its newest state.

• Delta load: SAP applications often offer to only extract new or changed records.

So, if we want to use delta loads for data from the SAP source system, how is the delta process implemented?

There are different ways to implement a delta process. First of all, the source system can provide the delta process. You have two options:

1. The extractor in the source system fills a delta queue based on a delta field, which can be a time stamp, date or sequence number.
2. The application fills a delta queue. No delta field is needed. The application may write storno records or additive delta records.

If the source system can’t provide the delta, SAP BW/4HANA can manage the delta mechanism. SAP BW/4HANA filters the data for different intervals of a delta field (time stamp, date, or sequence number.)

The delta process is a property of the DataSource that indicates how data is transferred. A DataSource with a valid delta process means that it can in some way send changes (Delta) to SAP BW/4HANA. As a formal attribute of the DataSource, it indicates exactly how the data in the DataSource is transferred to the data target.

Table ROOSOURCE contains DataSources and the assigned delta process. The details of a delta process are stored in table RODELTAM.

Some examples of delta processes are:

• ABR
• ADD
• ADDD
• AIE
• AIED

In table RODELTAM, every delta process contains a combination of values of the fields listed in the following table:

Let's look at the properties of a delta process.

A delta process has properties related to the following characteristics:

• Serialization (field DREQSER in table RODELTAM)
• Delta Type (field DELTATYPE in table RODELTAM)
• Record Mode (fields UPDM_NIM, UPDM_BIM, UPDM_AIM, UPDM_ADD, UPDM_DEL and UPDM_RIM in table RODELTAM)

Let's explain each of these characteristics.

Serialization defines to what extent the DataSource keeps the data packets and data records in chronological order.

The serialization property describes to what extent the sequence of individual data records within a delta process reflects the origin of these data records in the application. Serialization of the incoming records is necessary when updating a DataStore Object (advanced) with Activation option and the Overwrite Rule Type on the key figures. This is because you don't want to use the overwrite if the records lose serialization, as an older record can overwrite a newer one.

The possible values of serialization are as follows:

• (0) No serialization required.

• (1) Serialization of requests required: the data packages within a request do not need to be serialized.

• (2) Serialization of data packet required: the data packages in a request must also be serialized. For example, this is necessary if an extractor can only supply after images and several records with the same key can be sent within a request.

The delta type describes how the delta is staged within a delta process. Two of the most important values, delta type D and E, are only used with SAP source systems.

Watch this video to learn about possible values of delta type.

The record mode concept deals with the question of how changes to data records are recorded in the delta process.

Watch this video for an example of different record modes.

The record mode describes the type of change that a data record contains. The difference between the various delta processes is that each supports only a subset of the possible values. If a DataSource implements a delta process that uses several values, the field ROCANCEL (in which the record mode is saved) is automatically part of the DataSource. This field for the DataSource is assigned to the InfoObject 0RECORDMODE in the SAP BW/4HANA (or SAP BW) system.

For more details, the following link shows examples in the system: https://blogs.sap.com/2012/07/12/record-mode-concept-in-sap-bi/.

Let's look at the table about the key figure transformation behavior in loading scenarios, and the three main modeling options for DataStore Objects (advanced):

The model type of a DataStore Object (advanced) determines whether it can consistently use overwrite or summation for key figures. Therefore, you must model a DataStore Object (advanced) depending on what type of delta images were provided by your source of data, primarily DataSources.

Let's explain how the delta features of DataSources match up with DataStore Objects (advanced) that are modeled with Standard DataStore Object, Staging DataStore Object, or Data Mart DataStore Object.

The preceding figure describes the relationship between the delta types, possibly provided record modes, and the correct update to data targets for transactional data.

You want the correct values in the target. Thus, note the following considerations when using DataSources capable of providing delta figures in the modeling of a data flow:

• If the data that fills SAP BW/4HANA (or SAP BW) is an after image, the data can be consistently written to a DataStore Object (advanced) in which all key figures are updated using the MOVE setting in the corresponding transformation. This setting causes values to be overwritten and is only applicable for a Standard or Staging DataStore Object with Activation option. During request activation, the after image record replaces the old record. In fact, the DataStore Object (advanced) is in overwrite mode.
• If the data that fills SAP BW/4HANA (or SAP BW) is an additive image, the data can only be written correctly to a DataStore Object (advanced) in which all key figures are aggregated as SUM. In fact, the DataStore Object (advanced) is in sum mode. This is the case in a Data Mart DataStore Object, unless there are key figures with MIN or MAX aggregation type.
• If the DataSource sends both the before image (or reverse image in case of deletion) and the after image, no restrictions apply. This combination can be loaded correctly to a Standard or Staging DataStore Object with Activation option, or to a Data Mart DataStore Object. If a Standard or Staging DataStore Object with Activation option is in overwrite mode, only the after image (the last image) arrives in the activation queue of the DataStore Object (advanced). If there is at least one key figure for which the transformation is set to summation, both the before and after image (or reverse image) are necessary to load the data correctly to the target.
• If the DataSource only sends the after image records, this must first be updated to the Standard DataStore Object with Change Log, which is in overwrite mode.
• New image and the reverse image records can be processed by all targets.
• Delete images can only be processed by a DataStore Object (advanced) during Activation. A Data Mart DataStore Object cannot process deletions.

• If all record modes are additive image (a difference between new and old value), you must use SUM to update the key figure values. A pair of before image (a STORNO) and an after image (the new value) behaves in a similar way. Overwriting key figures (update method MOV) is only consistent when the source provides after images. Also, you keep the sequence of records with the same key in the data load process (within and across requests).

For more detail on exactly what each delta method outputs, follow this link:https://wiki.scn.sap.com/wiki/display/EIM/Dataflows+for+each+Extractor+Delta+Process+Type.

Let's compare two different SAP BI Content DataSources that have different delta processes. They are only similar in their use of serialization. Both delta processes provide the data records in the same sequence in which they were created in the source system.

The following is a comparison of the delta processes:

This is delta type D (push). Therefore, the delta data records from the application are posted to the delta queue before they are extracted from the operational delta queue as part of a delta update. If the DataSources are from the Leading Object (LO) Cockpit, this push takes place either with direct delta (directly from the application, V1–updating) or with queued delta (from the application to the extraction queue (V1)) and from there to the operational delta queue with a collective run.

The DataSource continues to provide after-image records, before-image records, and reverse-image records. For this reason, you can use Standard or Staging DataStore Objects with Activation option or Data Mart DataStore Objects.

Because this is a delta type (E) pull, the delta data records are determined during the delta update by the DataSource extractor. Then the records are updated to the operational delta queue, and passed on to SAP BW/4HANA directly from there.

For the DataSource that only returns the after images, do not update records directly into Data Mart DataStore Objects. All key figures must be set to overwrite in the transformation rules.

It is important to remember that a delta process does not indicate how the delta is determined in the application. This issue is addressed in different technical ways in the DataSource extractor or in the application itself, and depends on the environment and the data model of the specific application.

In our example, the DataSource 0FI_GL_4 determines its delta based on a time stamp field in the underlying source table: The changed data records can be selected in the extraction by the extractor of the DataSource using the field. However, the source tables of the DataSource 2LIS_02_ITM do not offer this precondition. Therefore, in this DataSource, the delta is logged at the moment of the update, for example, save in the dialog transaction, so that it can be extracted later in a delta update.