Enriching Customer Data Ingestion using Extensions in the SAP Customer Data Platform

Objective

After completing this lesson, you will be able to program and configure an Event to enrich Customer Data using Extensions

Introducing Extensions

In this lesson, you will learn when to apply, and how to setup and use Extensions.

Extensions allow two types of essential server-side customization use-cases related to data ingestion in SAP Customer Data Platform. It allows validation and optional data-enrichment of individual event records as part of the beginning of the ingestion process pipeline.

A flowchart illustrating the process of ingesting records in the SAP Customer Data Platform. Steps: 1. A record enters the event ingestion pipeline. 2. An extension request is sent to a customer's server running server-side extension code. 3. The SAP Customer Data Platform waits for a response. 4. If the response status is OK, the record is ingested, possibly with optional enrichment data. 5. If the ‘fail’ response status is received, the record is not ingested.

The Extension’s request payload is originated during the ingestion using the Event configuration. An example of such payload can be:

JSON
12345678910111213
{
    "callID": "123abc",
    "businessUnitId": "4_abc-d123D_456XYZ",
    "applicationId": "HB9yTnq1rdm1J3fT3RgJlw",
    "dataEventId": "HKabcD",
    "extensionPoint": "preProcess",               
    "data": {
	"firstName": "Joseph",
	"lastName": "Asif",
	"age": 32	
    }		
}

As you can see, it provides you with some metadata indicating the Business Unit, Application, and the Event that triggered the Extension execution. The data object contains one input Event record from the original Event input payload provided during ingestion.

When your REST-compliant extension code receives this request payload, it must provide a timely response that validates and optionally enriches the event record with additional customer data. As this is custom server-side code, anything is possible in terms of validation, including connecting to your SAP S/4HANA Cloud, SAP BTP, or any other systems in your IT infrastructure. It is up to your developers to code the script that will handle the validation and enrichment. The possible scenarios are:

  1. Validation of individual event record: It’s up to your REST-compliant server-side code to decide if record ingestion should happen, then return a status to tell SAP Customer Data Platform whether or not to ingest the customer data.

    For the records you want to indicate have passed Extension validation, set "OK" as the status attribute. Here is an example of such an Extension response payload:

    JSON
    12345
    {
    "status": "OK",
    "errorMessage": ""
}

    For the records you want to indicate have failed Extension validation, set the status attribute to "fail". It is also good practice to provide a relevant errorMessage telling the SAP Customer Data Platform why this Event record ingestion should fail, allowing this to be checked further in the Ingestion Monitoring logs. Here is an example of such Extension response payload:

    JSON
    12345
    {
    "status": "fail",
    "errorMessage": "The customer’s country needs to be part of the EU"
}
  2. Optional data enrichment: for situations when you return an "OK" status, you can optionally enrich the original Event individual record with extra attribute values that will be merged into it.

    Add a data.extension object to the Extension’s response payload with the additional data you want to add to the original Event record. That will be merged into the event record, and also ingested as long as it is both part of the event model schema and linked to the customer schema using the schema mapping configuration. On top of that, the event model and schema validations are also applied, as usual. Here’s an example:

    JSON
    1234567891011
    {
    "status": "OK",
    "data": {
        "extension": {
            "familyStatus": "married",
	     "age": 35
        }
    },
    "errorMessage": ""
}

    The Extension request needs to be authenticated using an OAuth 2.0 compliant JWT bearer token. The issuer attribute (part of the bearer token payload) also needs to be validated.

Note

Even though this is out of scope for this lesson, we include Python sample code showing how to perform such authentication validations. In case you need to code your own server-side REST endpoint in another language or platform, please refer to the manual on how to do that. The ready-to-use executable Python sample code here will then serve as a pseudo-code reference covering the minimal concerns you need to address in your own Extension code.

Implementing an Extension

In this section, we will explain how to implement an Extension in two main steps:

  1. Configuring an Extension on the Console
  2. Coding an Extension
Before starting:
  • Copy the code phrases, keys and other placeholder values to a text file.
  • Use the copy icon located on the right side of each textbox to ensure you've copied the full value of the corresponding key.
  • Familiarize yourself with Python language use and syntax, and how to host a public HTTPS REST Endpoint.
  • Ensure you have access to the SAP Customer Data Platform Console and create or use a Business Unit already loaded with a Customer Profile.
Practice system options:
SAP Practice System

Step 1. Configuring an Extension on the Console

1.1 he extension setup is part of the Event configuration, performed using the Console. When you create or edit a Source Application Event, you will find the Extension button at the top right of your Event configuration screen. On the screenshot below, we can see it once we’ve reached step 2 (Model) of the Event configuration.

The Event configuration screen with the Model step and Extension button highlighted. The screenshot also shows an Event Model containing firstName, lastName, masterDataId, timestamp, country, and zip attributes.

1.2 After clicking the Extensions button, you land on the Extension configuration page. Here you give a name to your Extension, set the external REST endpoint URL, enable or disable it, set a timeout (amount of time SAP Customer Data Platform will wait for the Extension’s response), and whether the entire ingestion will continue or stop if the Extension fails (remember that an ingestion is usually made of multiple individual customer data records). In this case, choose Stop.

Point the URL to your server-side REST endpoint. Sample code will be provided in the next step of this lesson. Save your changes.

Extension configuration screen containing Name, URL, Enabled toggle, Timeout, and Failure Policy, which has two options: Continue or Stop. The Save and Cancel buttons are also displayed.

Step 2. Coding an Extension

2.1 Let’s now code a simple Extension. For this scenario, let’s say we require the customer zip code to be '12345', otherwise the extension will fail. If the validation passes, the extension will also enrich the customer profile with country 'US'.

We will code our Extension in Python. And we will cover only the blocks of code relevant to our requirements. The full code will be provided at the end of this section.

2.1.1 First we retrieve the JWT bearer token from the HTTP request headers

Python
1234567

    auth_header = request.get_header('Authorization')
    if auth_header and auth_header.startswith('bearer '):
        bearer = auth_header[7:] # Remove 'Bearer ' prefix
    else:
        return err('Bearer token absent or invalid!')

2.1.2 Then we recover the Key ID (kid attribute) from the bearer token’s header.

Python
1234567

    try:
        header = jwt.get_unverified_header(bearer)
        kid = header['kid']
    except jwt.exceptions.PyJWTError as e:
        return err('Invalid token header: ' + str(e))

2.1.3 We obtain the SAP Customer Data Platform public key list in JWK format. In that list, we find the key corresponding to the kid in the previous step.

Python
123456789101112131415
    
    try:
        response = requests.get('https://accounts.us1.gigya.com/accounts.getJWTPublicKey?v2=true')
        keys = response.json()
    except requests.exceptions.RequestException as e:
        return err('Unable to retrieve SAP CDP Public Keys: ' + str(e))

    for key in keys['keys']:
        if (kid == key['kid']):
            public_key = jwt.algorithms.RSAAlgorithm.from_jwk(json.dumps(key))
            break

    if not public_key:
        return err('No SAP CDP Public Key for kid ' + kid)

2.1.4 We now validate the bearer token. This step is essential for security, blocking any request that did not originate from the SAP Customer Data Platform servers.

Python
123456
    
    try:
        token_payload = jwt.decode(bearer, key=public_key, algorithms=['RS256'])
    except jwt.exceptions.PyJWTError as e:
        return err('Invalid bearer token ' + bearer + '. Reason: ' + str(e))

2.1.5 Validate that the Issuer (iss attribute from the bearer token’s payload) matches the Business Unit ID. Replace <BU_ID> placeholder by your own Business Unit ID.

Note: Business Unit ID can be found in the browser address bar on the URL path between slashes after the /business_unit/ part.

Python
123456
    
    EXPECTED_ISS = 'https://fidm.gigya.com/jwt/<BU_ID>/'
    iss = token_payload['iss']
    if iss != EXPECTED_ISS:
        return err('Invalid iss ' + iss)

2.1.6 This next block implements our business requirement. In the Extension request payload, find zip code in the attribute data.zip. If the zip code is not '12345', fail the Extension execution.

Python
123456

    request_payload = request.json
    zipcode = request_payload['data']['zip']
    if zipcode != '12345':
        return err('Invalid zip ' + zipcode)

2.1.7 Lastly, return an HTTP 200 OK status (success). The Extension response payload has status OK, and enriches the original Event record with country 'US' attribute value (data.extension.country).

Python
12345678910111213141516

    status = 200
    message = {
        'status': 'OK',
        'data': {
            'extension': {
                'country': 'US'
            }
        },
        'errorMessage': ''
    }
    headers = {
        'Content-Type': 'application/json'
    }
    return HTTPResponse(body=json.dumps(message), status=status, headers=headers)

2.1.8 You may have found the helper function err being called at times when validation fails. Let’s take a look at its code. It basically returns an HTTP 200 OK status (success), the status as fail, and errorMessage as the function argument msg (provided by the function caller).

Python
123456789101112

def err(msg):
    status = 200
    headers = {
        'Content-Type': 'application/json'
    }
    message = {
        'status': 'fail',
        'errorMessage': msg
    }
    return HTTPResponse(body=json.dumps(message), status=status, headers=headers)

Note

Note: you may be wondering why the HTTP status is still 200 if we are failing the Extension’s response. That’s because it’s important to always return an HTTP 200 status, regardless of the actual Extension’s response status. The HTTP 200 status tells SAP Customer Data Platform that the REST Endpoint is healthy, and the validation failure is not at the system or infrastructure level, but instead at the application level (the Extension server-side code).

2.1.9 Please see the full Extension code here. If you decide to copy and paste, just make sure to use an iss URL that corresponds to your own Business Unit ID (see step 2-e above).

Note

Expose this REST endpoint over a public HTTPS server. This code is served on HTTP port 8080, but you can change that in the code. You will also need a proper reverse proxy, a registered public domain, and a valid SSL certificate set up on it. You can leverage SAP BTP Cloud Foundry or Kyma Serverless module, please refer to SAP Business Technology Platform (SAP BTP) on how to do that
Python
1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283848586878889909192

import requests
import json
import jwt
import bottle
from bottle import HTTPResponse

# fails with a reason message
def err(msg):
    status = 200
    headers = {
        'Content-Type': 'application/json'
    }
    message = {
        'status': 'fail',
        'errorMessage': msg
    }
    return HTTPResponse(body=json.dumps(message), status=status, headers=headers)

def main(url):
    request = bottle.request

    # retrieve the bearer token
    auth_header = request.get_header('Authorization')
    if auth_header and auth_header.startswith('bearer '):
        bearer = auth_header[7:] # Remove 'Bearer ' prefix
    else:
        return err('Bearer token absent or invalid!')

    # retrieve kid from bearer's header
    try:
        header = jwt.get_unverified_header(bearer)
        kid = header['kid']
    except jwt.exceptions.PyJWTError as e:
        return err('Invalid token header: ' + str(e))

    # retrieve public keys from SAP CDP
    try:
        response = requests.get('https://accounts.us1.gigya.com/accounts.getJWTPublicKey?v2=true')
        keys = response.json()
    except requests.exceptions.RequestException as e:
        return err('Unable to retrieve SAP CDP Public Keys: ' + str(e))

    # retrieve the kid public key from the list of SAP CDP public keys
    for key in keys['keys']:
        if (kid == key['kid']):
            public_key = jwt.algorithms.RSAAlgorithm.from_jwk(json.dumps(key))
            break

    if not public_key:
        return err('No SAP CDP Public Key for kid ' + kid)

    # validate bearer token using its public key
    try:
        token_payload = jwt.decode(bearer, key=public_key, algorithms=['RS256'])
    except jwt.exceptions.PyJWTError as e:
        return err('Invalid bearer token ' + bearer + '. Reason: ' + str(e))

    # validate bearer token payload's iss
    # replace the last part of the EXPECTED_ISS URL by your own Business Unit ID
    EXPECTED_ISS = 'https://fidm.gigya.com/jwt/4_iNfbhGDrBGciUrxckeBfJA/'
    iss = token_payload['iss']
    if iss != EXPECTED_ISS:
        return err('Invalid iss ' + iss)

    # sample validation for zipcode received from extension's request payload
    # fails if zip is not '12345'
    request_payload = request.json
    zipcode = request_payload['data']['zip']
    if zipcode != '12345':
        return err('Invalid zip ' + zipcode)

    # build and return a sample response payload that enriches country as US
    status = 200
    message = {
        'status': 'OK',
        'data': {
            'extension': {
                'country': 'US'
            }
        },
        'errorMessage': ''
    }
    headers = {
        'Content-Type': 'application/json'
    }
    return HTTPResponse(body=json.dumps(message), status=status, headers=headers)

if __name__ == '__main__':
    bottle.route('/<url:re:.+>', ['GET', 'POST'], main)
    bottle.run(host='localhost', port=8080, debug=True)

Step 3. Testing an Extension

Let’s now simulate two ingestions to test our Extension. On the first test scenario we will ingest a customer with zip code '12345'. That will pass the Extension validation, enrich the Event profile data with a country attribute set to 'US', and create a new customer profile. On the second test scenario, we will ingest a customer with a zip code that is not '12345'. That will fail the Extension validation and prevent the Ingestion of a new customer profile.

While we can perform customer profile data ingestion using different types of Application Source Events, or using the SAP Customer Data Platform REST APIs, we will resort to the Event Playground for the sake of simplicity. We will also use the Customer Search feature to verify the new customer profile data resulting from the Extension execution.

Make sure your customer Profile schema and the Event model schema matches the following JSON schema:

JSON
12345678910111213141516171819202122232425262728

{
    "type": "object",
    "properties": {
        "firstName": {
            "type": "string"
        },
        "lastName": {
            "type": "string"
        },
        "masterDataId": {
            "type": "string"
        },
        "timestamp": {
            "type": "string",
            "format": "date-time"
        },
        "country": {
            "type": "string"
        },
        "zip": {
            "type": "string"
        }
    },
    "title": "Import Profiles",
    "additionalProperties": true
}

Remember that the zip attribute is used by the Extension code validation, and the country attribute is enriched if the Extension passes validation.

3.1 Access the Event Playground by going to any existing Event inside an existing Source Application, and choosing the Event Playground on the Event card’s Action menu.

Source Application showing its existing Event configurations, one named Import Orders, another named Import Profiles. The action menu inside Import Profiles is selected and highlighted, and the option Playground is highlighted

3.2 Let’s now send Event data that will trigger the Extension’s first scenario: pass validation when zip code is '12345' and set the country to 'US'. Make sure you write down the first and last name (so we can check this profile later on using the Customer Search feature), set the zip to '12345' and make sure country is not 'US'. Your screen should look something like the screenshot below. Then choose Send Event.

Event Playground form showing fields corresponding to the event model such as firstName, lastName, masterDataId, timestamp, country, and zip (highlighted). Zip is set to 12345. The Send Event button is also highlighted.

3.3 After landing on the resulting screen, head to the Customer Search main menu option, and search using the name of the customer that was just ingested. The system will confirm the ingestion took place, meaning the extension validated the Event record.

Customer Search screen showing a single Customer Card for a customer named Cordelia Greenfelder. The search box contains the customer’s first name, Cordelia, providing the single search result. Most of the other customer attributes shown in the card are empty at this time.

3.4 Choose the customer card, then pick the Details tab. Verify that the country was changed to "US" (on the original ingestion payload, it was set to "Oman"), confirming the data enrichment defined by the Extension’s response.

Customer information for Cordelia Greenfelder. Customer attributes are shown in the card’s header, and most of them are currently empty, except for Master Data ID. The Details tab is selected and showing some attributes. The highlighted attribute, country, has the value US.

3.5 Go back to the Event Playground. Make sure the zip code is not '12345'. Choose Send.

Event Playground form showing fields corresponding to the event model such as firstName, lastName, masterDataId, timestamp, country, and zip (highlighted). Zip was set to a value that is not 12345. The Send Event button is also highlighted.

3.6 The result screen shows the ingestion pipeline failed on the Extension step.

The result screen of Event Playground showing the ingestion pipeline steps ingest and extension. The extension step is highlighted and marked with an alert icon, indicating a problem.

3.7 Head back to the Customer Search. Search for the customer’s first name in the ingestion attempt in the previous step. No customer should be found.

Result screen of Customer Search showing No Results when filtered by customer name Lilyan.

Coding an Extension – Video

In this video, we'll guide you through the steps necessary to code an extension, using the Python language as an example. The video will echo the steps in the previous demonstration; by the end, we hope you’ll feel confident you can code your own extension when your requirements call for one.

Log in to track your progress & complete quizzes

Learning

SAP FacebookSAP YoutubeSAP LinkedIn