Let's now explore how data flows can be used in SAP Datasphere. When extracting data from source systems, a data flow is used in the inbound layer. In the sales scenario of this course, the data flow is used within the harmonization layer.
Use of Data Flows
A data flow is a graphical ETL (Extract, Transform, Load) tool for loading and transforming data from SAP and non-SAP sources into a local table in SAP Datasphere. Data flows are used for scenarios requiring complex data transformations (including Python scripting), blending, and enrichment, especially when combining data from multiple sources. Local objects in SAP Datasphere (local tables, views, and Open SQL schema objects) can also be used as a source.
Note
The target of a data flow is always a local table within an SAP Datasphere space. For a list of available sources, you can check this documentation: https://help.sap.com/docs/SAP_DATASPHERE/be5967d099974c69b77f4549425ca4c0/94562426f30c475286f50a1e2b45e743.html?locale=en-US&version=LATEST.
Note
ODP-based sources (ODP_SAPI, ODP_BW extractors) are not supported in data flows. You can use a replication flow for ODP sources.
A data flow only supports Initial/Batch load. It does not offer delta load capability. On each run, the full dataset is processed and written to the target. The write behavior is controlled by the write mode configured at the target:
- Append:: Inserts all incoming records as new rows without affecting existing data.
- Truncate:: Deletes all existing records first, then loads the full dataset.
- Upsert:: Inserts new records and updates existing ones based on the primary key.
Data flows are only supported in spaces with storage type SAP HANA Database (Disk and In-Memory).
Creating Data Flows
Creating a data flow involves the following steps:
Add one or more sources.
You can choose from:
- Repository: local tables, views, and Open SQL schema objects within your space.
- Connections: tables and files from connected source systems (SAP and non-SAP).
Add transformation logic.
You can use SQL-based transformations (join, union, projection, filter, aggregation and calculated columns). Python script can be used as well.
Add a target table.
Every data flow requires exactly one target table, which must be a local table in the same space. Map the output columns from the last operator to the target table columns. If column names match, mapping is automatic. Set the write mode (append, truncate, upsert).
Running Data Flows
A data flow can be started form the Data Builder editor, the Data Integration Monitor or via a task chain.
A data flow run is a one-time batch event. The flow reads data from all configured sources, applies transformations, and writes to the target table. When the load completes successfully, the flow reaches a Completed status. If a data flow run fails, it restarts from the beginning on the next execution. There is no checkpoint-based resume capability. There is no pause/resume feature. The flow is either running or stopped.
Data flows have limited parallel processing capabilities compared to replication flows. The processing is executed in the SAP Data Intelligence embedded engine. There is no configurable thread or partitioning control equivalent to what replication flows offer. The Python script operator processes data in batches, not in parallel across the full dataset.
Data flows have a dedicated Schedule button available both in the Data Builder editor and in the Data Integration Monitor. You can also assign a data flow to a task chain.
Data flows support input parameters, which allow dynamic values to be passed at runtime. Parameters are defined on the flow canvas and can be set when running manually, via schedule, or via a task chain. This allows a single data flow to serve multiple use cases by varying the parameter values.
