Using the Catalog and Marketplace Features in SAP Datasphere

There are many challenges to getting value out of your data:

• It can be hard to know if a data asset or data product already exists in an organization.

• The meaning and intent of data assets or data products can be hard to discern without complete and consistent documentation.

• Efforts to build data models or dashboards can be duplicated, making it difficult to know the correct version to consume.

• Objects and measures can be misunderstood or unreliable for current use.

• It can be difficult for data creators to easily control which consumers have access to data and meta data.

An effective data-governance strategy helps you to address these challenges. User management and role management is just a part of such a data-governance strategy. It addresses authorization challenges, but cannot guarantee that users find, use, and interpret the objects in the right way. The catalog is designed to bring the objects and their meta data together. It provides a one-stop-shop seamless experience for those working with data in your organization. With the catalog, you can map out a complete and connected view of all your data sources, build and apply data-governance standards and processes, and offer trusted, self-service data discovery to all your users. Data-lineage information can be made available for reviewing the data provenance. Tags and a glossary with terms can be defined to add meaning to the objects and its elements.

SAP Datasphere combines ideas from two former SAP products called SAP Data Intelligence and SAP Data Warehouse Cloud, from specific SAP partner solutions such as Databricks, Confluent, DataRobot, and Collibra, and provides new capabilities. The idea of creating glossaries with specific terms and definitions has been integrated form SAP Data Intelligence and enhanced with other capabilities, such as data lineage and a tag hierarchy. SAP Datasphere provides standard roles for catalog management and usage.

Users with the Catalog Administrator role are responsible for the overall setup and data governance implementation for the catalog. They maintain an inventory of all data sources, build trust, and make that trusted data available to users. Depending on the amount of data and size of the organization, there could be one or several catalog administrators.

Catalog administrators are responsible for the following tasks:

• Connect: Add many different data-source systems in the same landscape to the catalog and extract and asset meta data. Monitor the meta data.
• Classify and enrich: Create and apply consistent business terms to an organization's data, and enrich the data with descriptions, tags, and other information in one place.
• Publish: Prepare and make available the governed data to all data users in an organization.

Catalog users are data modelers looking for the single source of truth for certain domain data to build reusable business models, or they are business users looking for stories for specific data or key-performance indicators (KPIs). Anyone who uses the catalog needs to find and understand data within an organization.

By using the catalog, users can work with data in several ways:

• Discover: Search, filter, and explore the data items that are available in the catalog.
• Evaluate: View descriptions and other meta data about a particular data item to make an informed decision about whether it's the item wanted and how to use it. View its impact and lineage to see information about its source, the transformations it goes through, its final state, and objects affected by changes made to it.
• Use: After finding and evaluating a data item, users can do one of the following:
  • Open the data asset to view the asset in its source system.
  • Try the sample data from the marketplace data product.
  • Install an API from an SAP Business Data Cloud data product.
  • Use the data item as part of a data project to build something new.

The following concepts are part of the data catalog.

• The catalog is the central place to discover, classify, profile, understand, and prepare all the data in your enterprise.

• An asset is any data or analytic object made available in the catalog (for example, a local table or view sourced from SAP Datasphere, an analytic model, or a story from SAP Analytics Cloud). Assets are the objects that you provide governance around and publish to the catalog for users to discover, evaluate, and ultimately open and start using for their work.

• Data providers can be a person or a company that offer data products in the catalog. Data providers are responsible for developing the data product, maintaining its lifecycle, and upgrading versions.

• Discovery is the act of searching for and finding an asset of interest. The catalog search page with filter options is the primary place for users to discover the data and assets that you've prepared and published.

• A business glossary can promote a common, consistent understanding of business terms within your organization. The definitions provide context for the terms, and the term relationships provide additional meaning. Aim to use the business glossary enterprise-wide or division-wide and make the terms as clear and understandable as possible

• Term relationships and KPI relationships are links to other terms and data assets.

• Data governance is the set of processes, standards, roles, and measures by which an enterprise manages its data assets. Having a well-defined data-governance strategy allows your organization to ensure that the data assets that employees use are trustworthy, consistent in the use of business language, well-documented, and secured.

• A KPI measures progress towards a result such as a goal or objective. A goal or objective is the outcome that you want to achieve. A KPI is a metric that indicates your progress in achieving the goal. Use KPIs to provide an analytical basis for decision-making.

• The Impact and Lineage Analysis diagram enables you to understand the lineage (also known as data provenance) and impacts of a selected asset in the catalog. Impact and lineage contain information about the source of the data, the transformations it goes through, its final state, and objects affected by changes made to it.

• A data product is a self-contained set of tables containing data exposed for consumption outside the producing application or service via APIs. Marketplace data products are only available in the SAP Datasphere catalog. SAP, customers, and partners produce other data products that can be acquired from the SAP Datasphere Marketplace.

• Meta data is the information about the asset itself: the name, description, source system where it originated from, date when it was created on, and much more. Meta data provides users with all the information they need to make an informed decision about the utility of an asset they find in the catalog.

• Meta-data extraction is the process of pulling information about assets from source systems to the catalog. When users search for and find an asset, what they are seeing is the representation of that underlying data object or analytics content in the catalog, made up of all its meta data. Much of this information is pulled from the underlying source system and displayed in the catalog.

• Publishing is the act of making assets visible and available to anyone who uses the catalog. Unpublished assets are available only to users who administer the catalog. Before an asset is published to the catalog, a user with the Catalog Administrator role can edit and enrich the meta data, apply appropriate classifications, and ensure the overall quality of the asset.

• Tags allow you to classify your data assets. Classification is the process by which you can organize your assets into clear categories. The catalog provides a hierarchical tagging system that helps you organize and manage your tags. For example, let's say you have an inventory of bike products that you want to classify.

You can define tags that can be used to organize the data assets efficiently, organize information, and narrow search results for an easier data discovery. You can, for example, add tags like 'approved' or 'in quality check' that give a hint to users how reliable the asset data is. You could also add tags like 'sales data' or 'purchase data' to indicate the topic. Tags can be linked to terms, data sets, or columns within a data set. You can combine as many tags as you like. Tags can be part of a hierarchy.

You could, for example, create a category called 'Bike Products' and add these tags to your tag hierarchy:

• • Bicycles
    • Road
    • Mountain
    • Hybrid
  • Motorcycles
    • Road
    • Motocross

Administration and monitoring is important for a governed publication and approval process. Users with role Catalog Administrator can see, create, and enrich all elements in the catalog, such as assets, terms, and KPIs. However, users with only the Catalog User role (or a subset of its authorizations) do not find them unless they are published. A catalog administrator can publish them to the catalog as part of a governed process to determine what content is available.

In SAP Business Data Cloud, the Catalog Administrator and the BD Viewer roles applied together, for example, grant these privileges. In SAP Datasphere, the Catalog Adminstrator, the DW Viewer role template (used directly as a global role), and the DW AI consumer-scoped role template applied together, for example, grant these privileges.

Publishing a catalog object has the following effects:

• The object is set to status 'Published'.
• The object can be discovered by all users in the catalog.
• The date and time when the element is published appears in the header under the catalog activity section.

You can set source systems to automatically publish assets, or you can be more involved and manually publish individual assets from the asset details page or manually publish small groups of assets from the catalog search page.

Publishing has no effect on tags, links, or other properties inside the object. Even published objects can be changed by a user with the Catalog Administrator role.

In 2023, templates were introduced to reduce the steps in creation and editing workflows. A template can provide fields that must be filled, such as data quality. This guarantees some degree of information and consistency.

Moreover, if you look for a glossary term and it is not available, you can directly attach the term to the glossary as a new term and add a definition for it. You can link terms and KPIs from the details view of an asset.

Users with a catalog administrator role can open the Remote Systems page to create connections to a variety of data sources in the same landscape. This page also includes tools for monitoring and managing data sources, allowing meta data for objects to be extracted and added to the catalog assets. You can connect up to three SAP Analytics Cloud tenants to the catalog to allow you to publish stories and other analytic assets.

You can connect the local SAP Datasphere tenant and multiple external SAP Datasphere tenants to a single catalog. On the Remote Systems page, you can create a connection to the local or an external SAP Datasphere tenant. Once connected, the catalog will start monitoring the source system for new, updated, or deleted assets. Then, you can extract meta-data content from multiple tenants in a centralized catalog. If the tenants are deployed for different regions, you can effortlessly categorize catalogs by region.

On the Details tab, you can find meta-data information about an asset, such as local tables, remote tables, views, intelligent lookups, and analytic models. It lists the columns, attributes, measures, dimensions, and properties of the asset. The details tab is also available for SAP Analytics Cloud perspectives, analytic models, and planning models with different options, depending on the type of object.

You can manage glossary terms, tags, and KPIs, and link them to assets as well as to individual attributes, measures, or dimensions of an asset. On the asset details page, you see the relationship between attributes and glossary terms, tags, or KPIs.

This feature helps users to understand the assets based on a better understanding of its elements.

To start finding an asset, proceed as follows.

1. In the side navigation area, choose Catalog & Marketplace → Search.
2. On the catalog search page, choose Assets as the scope.
3. Use the filters or the search to find the asset you want and select it. For more information, see Searching for Data Products and Assets in the Catalog.

Filter options are, for example, the type of asset, its ownership, and status.

A KPI can be defined and assets or its measures can be used as related objects. Pre-defined KPI types (qualitative or quantitative) and measure types exist.

When you want to edit and enrich an asset, proceed as follows:

1. On the asset details page, find the header toolbar and select Edit. The Edit Catalog Details dialog opens.
2. Change the asset name to make it more understandable.
3. Enter a custom summary description, or select the Generate Summary button to use the SAP Business AI feature to generate a summary description. AI-generated content may contain inaccuracies. Verify results before use. If you leave the summary description empty and save the dialog, the description from the source system is the default that appears in the summary description to all catalog users.
4. Save your changes. In the header of the asset details page, the name of the asset and summary description are updated.
5. Select the Documentation tab.
6. Select Edit Content.
7. Enter and save a custom description, or select the Generate button to use SAP Business AI feature to generate a description. You can add multimedia content (images and links to videos).

In the Overview tab, you now find the asset name, business name, meta data, and container or space.