Data flows contain the source, transform, and target objects that represent the key activities in data integration and data quality processes.
Data Flow Usage
Data flows determine how information is extracted from sources, transformed, and loaded into targets. The lines connecting objects in a data flow represent the flow of data with data integration and data quality processes.
- Extract, transform and load data.
- Determine the flow of data.
- Are closed operations.
- Are created in the Local Object Library or in the tool palette.
Each icon you place in the data flow diagram becomes a step in the data flow as shown in the figure.
You can use source and target objects and transforms as steps in a data flow. Make connections between the icons to determine the order in which Data Services completes the steps.
Data Flow Steps
Each step in a data flow, up to the target definition, produces an intermediate result. The intermediate result is called a data set.
The intermediate result is a set of rows from the previous operation and the schema in which the rows are arranged. This data set may be further processed and directed into another data set. A data set could be, for example, the results of a query containing a
WHERE clause, to filter some rows, that flows to the next step in the data flow that will cleanse those rows.
Data flows are closed operations, even when they are steps in a work flow. Any data set created within a data flow is not available to other steps in the work flow or job. So the only way for a data flow to generate data that could be further processed is to load the data into a table or file.