Outlining the Technical Architecture of a Data Product

Data products are at the heart of SAP BDC. They are the individual building blocks of the data platform and consist of curated business data and associated metadata that has been extracted from SAP and non-SAP applications.

For SAP source applications, SAP manages all aspects of the extraction, transformation and loading (ETL) process, from the source system to the SAP BDC object store where the data is landed. Customers simply consume the ready-to-use data products. A data product provides business-ready data with no data preparation tasks needed by a customer's IT organization. Customers can focus on adding value to the SAP-delivered data products through enrichment. Or simply use them as they are.

Each SAP source application provides the data to the data products using its own native technology. For example, SAP S/4HANA uses the ABAP technology where ABAP Core Data Services (CDS) views combine several ABAP tables into a harmonized view ready for extraction.

Data products are powerful, but if they cannot be discovered then they are useless.

SAP BDC data products are described and exposed using the SAP-developed, open standard Open Resource Discovery (ORD) protocol.

The ORD protocol was originally developed by SAP to expose APIs between systems. It was later extended to include SAP BDC data products. ORD is a well-established framework within the SAP Business AI Platform that is used by developers to discover APIs and events from SAP systems.

The goal of ORD in the context of SAP BDC is to collect information relating to data products from across all SAP source applications and to expose them using a central catalog. The provider must ensure the Data Product Definition file for each data product includes the key information that the aggregator requires through the provider API. The aggregator for SAP BDC is the central catalog.

The ORD protocol includes important discoverability attributes such as visibility, release status, and integration dependencies.

If you are interested in learning more about the ORD protocol, you can click here: Guide to Open Resource Discovery (ORD)

As well as the business data, SAP BDC data products also provide the valuable meta data that describes the business data in detail.

The meta data of a data product provides AI with essential information about the meaning of the business data. Without meta data, accurate and useful results from AI would be difficult to achieve.

Meta data includes:

• Descriptions / labels for each field in various languages - for example, the technical field R_001_10a has the label Gross Revenue in English and Revenu Brut in French.
• Identification of the field that provides the language for the display of labels and descriptions. For example, the country of the company code that generates the invoice.
• Nested association to other entities - for example, product is associated with supplier which is associated with country.
• Semantics that describe the precise meaning of each field in the data product - for example Revenue is a monitory value with the currency EUR.
• Rules to describe how a measures should and should not be aggregated - for example, Closing Stock Balance should never be aggregated using Sum for the same product across time dimensions and should instead be aggregated over time using Last.

The Core Schema Notation (CSN) protocol is used to define the meta data of SAP BDC data products.

CSN is a well-established format to describe entity and service models within the SAP ecosystem. CSN uses a JSON format to exchange metadata between systems.

If you want to learn more about CSN and see a detailed example, click here:

Foundation Services are often referred to as the backbone of SAP BDC.

Foundation Services play a key role in the generation and management of data products. The 'fuel' of SAP BDC.

Foundation Services are the behind-the-scene SAP-managed services that have multiple responsibilities including:

• Extracting data from SAP applications
• Transforming and unifying data from SAP applications
• Generating data products
• Setting up the initial and delta load processes
• Managing the storage of data products in the object store
• Sharing data products with consumer applications
• Governance and security of data products

Customers do not have access to the Foundation Services. What takes place is in the background and is invisible to customers. SAP constantly monitors all activities in the Foundation Services of a customers' SAP BDC landscape to ensure that the data flows from the source applications to SAP BDC without interruption.

Foundation Services is a mandatory component in a SAP BDC landscape and is part of the provisioned SAP BDC core component.

The object store of the Foundation Services is based on SAP HANA Cloud, Data Lake.

Let's follow the flow of the production of a data product.

1. To trigger the installation of a data product, you can install intelligent content which automatically installs the required data products, or you can manually install a data product from the SAP BDC Cockpit.
2. The Orchestration Layer of the Foundation Services is informed about the installation and starts the generation process. The Orchestration Layer checks the Data Product Registry where it looks up the Data Product Definition file to find out which source system to connect for each data product.
3. In the source system there is a mapping between each data product and the source tables / views. Data is collected by the source system and pushed to the ingestion layer (replicated) of the object store in the SAP BDC Foundation Services. The data is stored in the format of the source system.
4. The data from the replicated layer passes through transformations which are based on Spark pipelines, to generate consumption-ready, stable data products in the refined layer. Some data products might also pass through an additional layer to generate a derived data product from the refined layer to the enriched layer.
5. Finally the data product is exposed using the ORD protocol so that is can be discovered by consumers across the customer landscape.

The data products of SAP BDC are stored in the object store. The object store is a key component in the SAP BDC Foundation Services.

The object store is technically implemented with SAP HANA Cloud, Data Lake. In the data lake the data is managed using the Delta Table format which supports the delta share protocol. The data is stored in files - usually parquet files.

The data lake storage allows SAP BDC to efficiently manage very large volumes of data with features such as ACID transactions, scalable metadata handling, and unified streaming and batch data processing. This approach is cost-effective and highly scalable, supporting multi-terabyte storage and processing.

Sharing data instead of copying it means that you avoid creating uncontrolled data silos. Sharing a single data set across all applications means you maintain control of who has access to the data and you are always sure that the data can be trusted as the single version of the truth.

In SAP BDC, data products are shared using the delta sharing protocol. The delta share protocol is a widely adopted open-source technology that enables data access:

• without moving or copying data. This is achieved using a zero-copy approach.
• through a variety of methods to support most consumption tools.
• with a central governance that ensures data security.
• in a highly scalable way.

The data provider shares selected data and manages access through a sharing server that uses the delta sharing protocol.

The data consumer must use one of the many delta sharing clients that supports the protocol. Some open-source connectors have been released, for example, for Apache Spark and Python.

The protocol ensures that the client authenticates to the sharing server and makes sure it's allowed to access the data requested in the query. Then, it logs the request and determines which data to send back. The sharing server then creates temporary URLs for the client to download files directly from the cloud provider. This process allows for fast, large-scale data transfer without going through the sharing server, making it efficient and cost-effective.

You can learn more about delta sharing here: Deep Dive on Delta Share

• You've learned how data products are self-describing using the ORD protocol.

• You've learned how CSN files provide the meta data of the data products.

• You've learned how Foundation Services manage the data products and provide a storage component known as the object store, which is based on a data lake architecture.