In this lesson, you'll learn more about the data products and Foundation Services provided by SAP Business Data Cloud.
Explaining Foundation Services and Data Products
Objective
After completing this lesson, you will be able to explain Foundation Services and Data Products.
Introduction
Technical Aspects of Data Products
We already covered the basic concept of a data product, so let's go a little deeper and cover some technical aspects of data products.
Data products are managed by the provider. In the case of SAP-managed data products, SAP is the provider and takes care of ensuring the data products are always up to date. For example, if new fields are added to the underlying tables in the source system, those fields should also be available to the data products.
Data products from all SAP source applications expose themselves to SAP Business Data Cloud so that they appear in the catalog. This means a business user could explore the data product before deciding if it should be activated. After activation, the data product is managed in the Foundation Services of SAP Business Data Cloud so that it can be shared with consumers such as SAP Databricks and SAP Datasphere.
A lot of background work goes into making a data product available for consumption by the business user. This includes the extraction, harmonization, and joining of the different source tables and setting up the initial and delta data replication processes. This is all handled in the background by SAP.
SAP Business Data Cloud is built to scale huge amounts of data so data products are optimized towards high volume, intensive reads. Data products are read-only and cannot be changed by the customer.
If an SAP source application has been extended with additional custom fields, these are included in the data extraction and land in the SAP Business Data Cloud as part of the SAP-managed data product. However, custom fields are ignored by the SAP provided data models that sit on top of the standard SAP-managed data products. Customers should develop their own custom data models if they want to include the custom fields of the data product in their analytical applications.
Each SAP source application provides data products using its own technology. For example, data products from SAP S/4HANA use ABAP Core Data Services (CDS) views to combine several tables into a harmonized view.
It is the data product's responsibility to describe and expose itself. This is achieved using the Open Resource Discovery (ORD) protocol.
ORD is an open, industry standard protocol that was originally developed by SAP. The ORD protocol was originally developed for the discovery and exposure of different types of APIs across SAP systems. SAP has been adding more and more object types to ORD that can be discovered, including events. When SAP Business Data Cloud was developed, SAP added data products to the ORD protocol. Using ORD makes it possible for any compliant provider to expose data products and for consumers to discover them. The concept behind ORD is that the source systems should describe their data products. It s not the responsibility of the consumer to describe the data product.
Foundation Services and Delta Share
The Role of Foundation Services
Foundation Services are responsible for managing SAP-managed data products. Foundation Services provide data pipelines to connect to SAP applications and replicate data to the object store of SAP Business Data Cloud.
Watch this short video to learn more about the role of the Foundation Services of SAP Business Data Cloud.
Using the Open Resource Discovery (ORD) protocol, Foundation Services can discover all the data products available from the different data sources.
You can explore the available data product's technical definitions and business descriptions and if you want to go ahead, you can then install the data product in SAP Business Data Cloud.
Technically, the data is replicated to the SAP HANA Cloud Data Lake object store. This read-only data is managed in data lake files that are storage and access-efficient.
A Data Lake stores data in files, not tables. SAP Business Data Cloud uses the SAP HANA Cloud Data Lake for storage of data in the Foundation Services.
Storing data in files instead of tables also widens the range of tools that will be able to consume your data.
To ensure security, the data in the Foundation Services is managed by SAP and is not directly accessible by users. To access the data stored in the Foundation Services, users need to install the required data product into SAP Datasphere or SAP Databricks.
Note
Foundation Services uses a dedicated object store to manage data. The object store of Foundation Services is only for SAP-managed data products. For custom data products, the SAP Datasphere object store is used.
Introducing Data Lake and Delta Share
Data products are shared using the Delta Sharing protocol. The delta share protocol is a widely adopted open-source technology that enables data access:
- without moving or copying data. This is achieved using a zero-copy approach.
- through a variety of methods to support most consumption tools.
- with a central governance that ensures data security.
- in a highly scalable way.
The data provider shares selected data and manages access through a sharing server that uses the Delta Sharing protocol.
The data consumer just needs one of the many Delta Sharing clients that supports the protocol. Some open-source connectors have been released, for example, for Apache Spark and Python.
The protocol ensures that the client authenticates to the sharing server and makes sure it's allowed to access the data requested in the query. Then, it logs the request and determines which data to send back. The sharing server then creates temporary URLs for the client to download files directly from the cloud provider. This process allows for fast, large-scale data transfer without going through the sharing server, making it efficient and cost-effective.
You can learn more about Delta Sharing here: https://delta.io
Note
The Delta Sharing concept is different from the concept of sharing in the context of SAP Datasphere Spaces or SAP Datasphere Marketplace.
Let's Summarize What You've Learned
In this lesson, you've learned about data products and the role of Foundation Services of SAP Business Data Cloud.
Data Products: Predefined, self-describing datasets integrate raw business data and metadata, optimized for large-scale, read-only analytical use.
Foundation Services: Generate SAP-managed data products using Open Resource Discovery (ORD) and Delta Sharing for integrated, high-quality datasets.