Ingesting Customer Data with the Ingestion API in the SAP Customer Data Platform

Before we can dive into the specifics of ingesting data using the Ingestion API, it's imperative to properly prepare your data for a smooth and error-free ingestion process.

Organize Data Sources

Collect all data sources you plan to ingest into the SAP Customer Data Platform. Make sure your data is clean, consistent, and formatted according to the ingestion requirements. This often involves normalizing the format of customer data fields like names, addresses, emails, and phone numbers.

Validate Data Structure

Ensure your data corresponds to the expected schema of the target events within the SAP Customer Data Platform to avoid format errors that may arise during ingestion.

Prepare Business Unit, Customer Schema, Server Application and Events in the SAP Customer Data Platform Console

To persist the customer data ingested from the events, we need to prepare the business unit and customer schema. The custom attributes we add in the customer schema will be mapped to the attributes on the event model we build for the server application. Let’s assume the server application is created, and the event model is properly mapped to the customer schema.

In SAP Customer Data Platform, each event is identified using either of two unique identifiers: dataEventId or externalId. Ensure that you copy the appropriate identifier of the event you plan to ingest.

In this section, we will explain how to ingest customer event data using the Ingestion APIs in three main steps:

1. Locate the Common Ingestion API Parameters
2. Call the Ingestion API to Ingest a Single Event by ID
3. Call the Ingestion API to Ingest Multiple Events By ID

Before starting:

1. Access your SAP Customer Data Platform tenant or the SAP practice system
2. Create or use an existing Business Unit
3. Create or use an existing Server Application and Event

In this section, we will identify the common Ingestion API parameters we will use to build REST API calls to SAP Customer Data Platform.

1.1 First comes the BASE_URL, which prefixes the Endpoint URL for our REST API calls. For SAP Customer Data Platform, this value is https://cdp.eu5.gigya.com/api.

1.2 Next, let’s find the value of the Ingestion API call parameters, the BUSINESS_UNIT_ID, APPLICATION_ID, and EVENT_ID.

BUSINESS_UNIT_ID is the ID of the business unit in your SAP Customer Data Platform tenant. APPLICATION_ID and EVENT_ID are the ID values of the source application and event that SAP Customer Data Platform uses to ingest the customer data.

On the SAP Customer Data Platform Console main menu, choose Sources under the Connect menu group. On the Sources screen, choose the source application from which you want to ingest customer data. Then, on the source application screen, click the vertical three dot action menu of the target event and choose Edit.

1.3 Copy the value of the business-unit path from the browser address bar. Let’s call this the BU_ID. In the screenshot below, this value is 4_A4hMWN7E7TuEHFOczsisgg.

1.4 Copy the value of the sources path parameter from the browser address bar. We’ll refer to this value as APPLICATION_ID. In the screenshot below, this value is HGGLC_Gbr1OWyLknFVPmEg.

1.5 Copy the value of the event path parameter from the browser address bar. We’ll call that the EVENT_ID. In the screenshot below, this value is HJMZmEHYXAgx3_mM3Y3uDg.

You can reference Lesson 1 Creating Admin Credentials for API Access in the SAP Customer Data Platform for how to find the USER_KEY and SECRET_KEY in the Server application configuration.

We now have all the common requirements for using the Ingestion APIs: the Base URL, USER_KEY, SECRET_KEY, BU_ID, APPLICATION_ID, and EVENT_ID.

Each of the REST API Calls will vary on the last parts of the URL Endpoint after APPLICATION_ID, depending on whether we ingest customer data by event ID or Name. In the case of event ID, the last parts of the URL Endpoint will be dataevents/{dataEventId}/, and for event name, the last parts of the URL Endpoint will be external/{externalId}/.

For the HTTP Verb, we generally use POST for data ingestion APIs.

1.6 Another way to easily locate the Ingestion API request parameters is by checking out the event listener URL, displayed on the Listener tab (the last step) of event configuration.

1.7 Open the Event Listener tab of the source application in SAP Customer Data Platform Console. As shown on the above screenshot, you can grab the values of Business Unit, Application Id and Event Id from the highlighted listener URL.

In this case, the event listener URL is:

1
https://cdp.EU5-prod.gigya.com/api/businessunits/4_A4hMWN7E7TuEHFOczsisgg/applications/HGGLC_Gbr1OWyLknFVPmEg/dataevents/HJMZmEHYXAgx3_mM3Y3uDg/event

Using this event listener URL, we can easily locate the BU_ID, APPLICATION_ID, and EVENT_ID.

Result: You have successfully located the Common Ingestion API Parameters.

2.1 Let’s now build the SAP Customer Data Platform REST API call that ingests a single customer Order event. The CURL command template is as follows:

12345678
curl  -X POST \ 
'BASE_URL/businessunits/BU_ID/applications/APPLICATION_ID/dataevents/EVENT_ID/event' \ 
  --url-query 'userKey=USER_KEY' \
  --url-query 'secret=USER_SECRET' \
  --header 'Accept: */*' \ 
  --header 'Content-Type: application/json' \ 
  --data-raw 'JSON_PAYLOAD'

2.2 The actual Order event JSON payload is passed as the request body; you can use the Event Playground feature to generate a sample payload.

2.2.1 Navigate to Sources under Connect on the left side menu.

2.2.2 On the Sources screen, choose the desired Server Application.

2.2.3 On the Server Application screen, choose Playground from the three-dot action menu of the desired Event.

2.2.4 On the Event Playground screen, choose the JSON tab and copy everything in the example JSON payload textbox underneath, for example:

12345678910
{
    "currency": "laborum quis cillum",
    "amount": 54196764.42036781,
    "tax": -78799498.56686981,
    "id": "22a63c68-88ce-47c6-9aa0-8c368257fd1b",
    "crmId": [
        "est"
    ]
}

2.2.5 In the above payload, to find the crmId value of an existing customer profile, navigate to Search under Customers on the left side menu.

2.2.6 On the Customers dashboard, find the desired customer profile, hover the mouse over the CRM ID attribute value and choose the copy icon to save the CRM ID attribute value to the system clipboard.

Now based on the auto-generated payload structure, we can replace the attribute values as needed, including the CRM ID value we just copied. In this demo, we’ll use the following JSON payload:

12345678910
{
    "currency": "EUD",
    "amount": 199.50,
    "tax": 10.5,
    "id": "002",
    "crmId": [
        "1000202"
    ]
}

2.3 With everything ready, you can build and send the API call using your preferred REST-compliant client applications. For the sake of accessibility, we’re going to use the CURL command for exercise or demo purposes.

After replacing the endpoint URL, the request parameters (as parts of the URL), and JSON payload in the CURL command template, we can go to the Terminal or Command Line Interface (CLI) window and execute the following CURL command to send the event ingestion API call.

12345678910111213141516
curl  -X POST \
'https://cdp.eu5.gigya.com/api/businessunits/4_A4hMWN7E7TuEHFOczsisgg/applications/HGGLC_Gbr1OWyLknFVPmEg/dataevents/HJMZmEHYXAgx3_mM3Y3uDg/event?' \
  --url-query 'userKey=AIcBrfeDDfPs' \
  --url-query 'secret=ioaf2BbChDOmh8/vcmUlIoL12chqLqMg' \
  --header 'Accept: */*' \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "currency": "EUD",
    "amount": 199.50,
    "tax": 10.5,
    "id": "001",
    "crmId": [
        "1000202"
    ]
}'

You should receive an empty JSON response {} in the Terminal or CLI window, indicating the ingestion API call was successful.

2.4 To verify the result, navigate to the Customer Dashboard for the same customer profile, and on the Activities tab, choose the latest Order activity on the Activity Timeline.

To ingest multiple events in a batch, we can use the Ingest Multiple Events by ID API. We can use the same data we used in the Single Event Ingestion API, changing only the endpoint URL and JSON payload.

3.1 Let’s now build the SAP Customer Data Platform REST API call that ingests multiple customer Order events. The CURL command template is as follows:

12345678
curl  -X POST \ 
'BASE_URL/businessunits/BU_ID/applications/APPLICATION_ID/dataevents/EVENT_ID/events' \
   --url-query 'userKey=USER_KEY' \
  --url-query 'secret=USER_SECRET' \
  --header 'Accept: */*' \ 
  --header 'Content-Type: application/json' \ 
  --data-raw 'JSON_PAYLOAD'

As you see, the endpoint URL differs from the Single Event Ingestion API at the end: instead of event, we’ll now use events to ingest multiple events.

3.2 The payload structure is also different. Instead of providing a single event object, we can pass an array of event objects in JSON format. Here is an example JSON payload for multiple Order events:

12345678910111213141516171819202122
{
  "events": [
    {
      "currency": "USD",
      "amount": 399.5,
      "tax": 20.5,
      "id": "007",
      "crmId": [
        "1000202"
      ]
    },
    {
      "currency": "EUD",
      "amount": 599.5,
      "tax": 30.5,
      "id": "008",
      "crmId": [
        "1000202"
      ]
    }
  ]
}

3.3 After replacing the endpoint URL, the request parameters (as parts of the URL), and the JSON payload with two Order events in the CURL command template, we can go to the Terminal or Command Line Interface (CLI) window and execute the following CURL command to send the event ingestion API call.

12345678910111213141516171819202122232425262728
curl  -X POST \
'https://cdp.eu5.gigya.com/api/businessunits/4_A4hMWN7E7TuEHFOczsisgg/applications/HGGLC_Gbr1OWyLknFVPmEg/dataevents/HJMZmEHYXAgx3_mM3Y3uDg/events' \
  --url-query 'userKey=AIcBrfeDDfPs' \
  --url-query 'secret=ioaf2BbChDOmh8/vcmUlIoL12chqLqMg' \
  --header 'Accept: */*' \
  --header 'Content-Type: application/json' \
  --data-raw '{
  "events": [
    {
      "currency": "USD",
      "amount": 399.5,
      "tax": 20.5,
      "id": "007",
      "crmId": [
        "1000202"
      ]
    },
    {
      "currency": "EUD",
      "amount": 599.5,
      "tax": 30.5,
      "id": "008",
      "crmId": [
        "1000202"
      ]
    }
  ]
}'

Again, you’ll receive an empty JSON response {} if the call was successful.

3.4 To verify the result, navigate to the Customer Dashboard for the same customer profile, and on the Activities tab, choose the latest Order activity on the Activity Timeline. You’ll see two Order activities created on the timeline.

In this video, we demonstrated how to ingest customer event data via the Ingestion API in SAP Customer Data Platform. We walked you through setting up the API call using the cURL command. You learned how to locate the request parameters, configure the API endpoint, structure the JSON payload, and execute the API request to successfully ingest customer data into the platform.