Working with Replication Flows

A replication flow is designed for mass data movement between a source system and a target system. It is the recommended tool for ingesting large volumes of data into SAP Datasphere from SAP and non-SAP source systems, as well as for outbound scenarios where data is replicated from SAP Datasphere to an external target. It is part of the ELT (Extract, Load, Transform) pattern. Replication flows replicate raw data and transformation flows can be used for transforming the data in a separate step.

Sources include SAP and non-SAP systems connected via the SAP Datasphere connection framework. For a complete list of possible connections which can be used as source or target for a replication flow, you can check this: https://help.sap.com/docs/SAP_DATASPHERE/be5967d099974c69b77f4549425ca4c0/94562426f30c475286f50a1e2b45e743.html?connection_overview-rfs=yes&connection_overview-rfs=via+SAP+ABAP+Connections&locale=en-US&version=LATEST

Replication flows can be used for the following three scenarios:

• Inbound data movement: Replicate data from SAP or non-SAP sources into SAP Datasphere and capture data changes occurring near real-time from the source to update the target table.

• Outbound data movement: Read data from SAP Datasphere and replicate it into supported SAP and non-SAP outbound targets while tracking data changes.

• Pass-through option: Use SAP Datasphere as a middleware and replicate data from supported sources into supported targets while tracking data changes occurring near real-time in that source.

A replication flow offers the following load types:

• Initial Only: Performs a one-time full load of all data from the source. No subsequent changes are captured after the initial load completes. You use this when you need a one-off snapshot of source data.
• Initial and Delta: Performs a full initial load, then continuously replicates changes only (inserts, updates, deletes) at a configurable interval. This is the most common load type for operational reporting or near-real-time analytics. Once started, the flow remains Active until stopped or paused.
• Delta Only: Replicates only changes that occur after the flow starts. No initial full load is performed. You use this when the target already contains a baseline copy and you only want to keep it up to date.

Replication flows are supported in spaces with storage type SAP HANA Database (Disk and In-Memory) and storage type SAP HANA Data Lake Files. When you use a space with storage type SAP HANA Data Lake Files, data is written to an inbound buffer first. A separate merge task must run to make data visible in the target local table (file). The option Merge Data Automatically option can automate this step.

In a replication flow, a container is the source-side grouping you select before choosing source objects. It acts as a logical namespace or configuration boundary that organizes the objects available for replication.

Let's explore the type of containers when connecting ABAP sources:

• CDS_EXTRACTION: for ABAP CDS views (C1-released, with @Analytics.dataextraction.enabled: true).

• SQL_SERVICE: for standard and custom CDS view entities exposed via the ABAP SQL service.

• ODP_BW: for ODP-based replication of SAP BW InfoProviders using the BW extraction context.

• ODP_SAPI: for ODP-based replication using the S-API extraction context, providing access to SAP standard extractors and DataSources released for ODP.

• SLT (Mass Transfer ID): for table-based replication via SAP Landscape Transformation Replication Server (SLT). The container corresponds to a pre-configured mass transfer ID in SLT, which must exist in SLT before creating the replication flow. It requires the DMIS add-on (DMIS 2018 SP06 / DMIS 2020 SP03 or higher recommended).

The table Container Types: SAP S/4HANA, SAP BW and SAP BW/4HANA shows which container types can be used for SAP S/4HANA, SAP BW and SAP BW/4HANA:

You can find more information about the use of SAP S/4HANA and other ABAP sources for replication flows here: https://help.sap.com/docs/SAP_DATASPHERE/c8a54ee704e94e15926551293243fd1d/3f70579c92434f4f88471bba2bd70893.html?locale=en-US&version=LATEST

In a replication flow you can define a Projection for a replication object. This offers the following options:

• Row-level filtering: Apply filters to restrict which rows are replicated from the source. Filters apply at the replication object level (per source table or CDS view).

  Note

  When filtering delta loads from ABAP CDS views, filters should be applied on primary key fields only. Delete operations are recorded in the ODQ (Operational Delta Queue) with key fields only, so filtering on non-key fields can cause issues with delta consistency.

• Column-level shaping: Remove columns, rename columns, reorder columns, change data type of columns, or add new columns (populated with a constant value or a predefined function (for example CURRENT_DATE)).

  Note

  For ABAP CDS view sources, you can go beyond the standard projection UI by implementing the BAdI BADI_DHAPE_RMS_EVENTS in the source ABAP system. This allows complex custom projection or filter logic that goes beyond what the SAP Datasphere UI supports.

Replication flows do not support complex transformations. For advanced transformation logic, you combine the replication flow with a transformation flow.

A replication flow can be started form the Data Builder editor or from the Data Integration Monitor.

The behavior of a run depends on the load type configured:

• Initial Only: Starts a batch run that completes when all data has been loaded. The flow reaches a Completed status.
• Initial and Delta: Starts an initial full load, then enters a continuous active state. The flow status becomes Active and remains until the flow is stopped or paused. Delta changes are replicated at the configured interval.
• Delta Only: Begins monitoring for changes immediately. No initial load is performed. The flow status is Active.

You can use the Data Integration Monitor for monitoring. If a replication flow run fails, it restarts where it failed (not from the beginning). You can pause a replication flow run at any time (for example during a system update) and resume it later. While paused, SAP Datasphere stops copying data to the target, but the source continues tracking changes. Upon resumption, all changes that occurred during the pause are replicated.

Replication flows support parallelization during initial load through automatic partitioning. Large datasets are sliced into smaller chunks that are replicated in parallel. The degree of parallelization is controlled by the number of assigned replication threads. Up to a maximum of 16 parallel jobs (80 threads) can be utilized for mass data replication during initial loads. For delta loads, parallel thread configuration is also available for CDS views and SLT tables.

Replication flows do not have a Schedule button in the Data Builder editor. Scheduling must be set up from the Data Integration Monitor.

Only if all objects in a replication flow have load type Initial Only, the flow can be assigned to a task chain. Flows containing any objects with load type Initial and Delta or Delta Only cannot be added to a task chain, because those flows have no defined end point.