Introduction to Orchestration

Orchestration combines content generation with a set of functions that are often required in business AI scenarios.

Key Functions:

Templating
Content filtering
Data Masking
Orchestration Workflow
Configuration and Execution
Harmonized API

Orchestration Workflow

In a basic orchestration setup, you can combine different modules into a pipeline executed with a single API call. Each module’s response serves as the input for the next module.

Grounding

Grounding integrates external, contextually relevant, domain-specific, or real-time data into AI processes. This data supplements the natural language processing capabilities of pre-trained models, which are trained on general material.

Grounding is a service designed to handle data-related tasks, for example grounding and retrieval, using vector databases.

The Pipeline API is proxied through the SAP AI Core generative AI hub, and incorporates vector stores, such as the managed SAP HANA Cloud database.

Grounding provides specialized data retrieval through vector databases, grounding the retrieval process using your own external and context relevant data. Grounding combines generative AI capabilities with the capacity to use real-time, precise data to improve decision-making and business operations, for specific business AI driven solutions.

Templating

The templating module is mandatory. It enables you to compose prompts and define placeholders. It then generates the final query that is sent to the model configuration module. Any placeholders that you define in your prompt can be filled at runtime. For example, the following template places the input text into the {{ ?input }} placeholder, for example:

Code Snippet

12345678
"templating_module_config": {
  "template": [
    {
      "role": "user",
      "content": "{{ ?input }}"
    }
  ]
}

The {{ ?input }} placeholder is filled using the contents of the input_params.input parameter:

Code Snippet

123
"input_params": {
  "input": "A sample prompt to be sent to the model"
}

You can also set default values for the placeholders. For example, to request either 5 paraphrases or a user-specific number of paraphrases for a given phrase, you can use the following configuration:

Code Snippet

12345678910111213141516
"templating_module_config": {
  "template": [
    {
      "role": "user",
      "content": "Create {{ ?number }} paraphrases of {{ ?phrase }}"
    }
  ],
  "defaults": {
    "number": 5
  }
},
//... other configuration
"input_params": {
  "number": "3",
  "phrase": "Please respond as soon as possible, thanks."
}

Data Masking

The data masking module is optional. It anonymizes or pseudonymizes personally identifiable information from the input.

Anonymized data can not be unmasked as information about the original data is not retained. Personally identifiable information is replaced with a MASKED_ENTITY placeholder.
Pseudonymized data can be unmasked in the response. personally identifiable information is replaced with a MASKED_ENTITY_ID placeholder.

The module currently supports the following anonymization services:

SAP Data Privacy Integration - Anonymization

Caution!

Input Filtering

The content filtering module is optional. It lets you filter the input and output based on content safety criteria.

The module supports the Azure Content Safety classification service. This service recognizes four distinct content categories: Hate, Violence, Sexual, and SelfHarm. For more information, see Harm categories in Azure AI Content SafetyInformation published on non-SAP site. Text can have more than one label (for example, a text sample can be classified as both Hate and Violence). The returned content categories include a severity level rating of 0, 2, 4, or 6. The value increases with the severity of the content.

If no content filter is configured in the orchestration configuration, the input is passed to the model configuration without filtering. Similarly, the response is returned without filtering. If no severity levels are set in the configuration (i.e. the config key is not given), the default level of 2 (low) is used for all categories.

Model Configuration

The model configuration module is mandatory. It lets you inference the large language module by making a call to the specified LLM and returning its response. You can configure the module by passing the following information:

Model Name: The name of the model to be used.
Model Parameters: The parameters to be applied to the model.
Model Version: The version of the model to be used, which can either be latest or a specific version number.

Output Filtering

The content filtering module is optional. It lets you filter the input and output based on content safety criteria.

The module supports the Azure Content Safety classification service. This service recognizes four distinct content categories:

Hate
Violence
Sexual
SelfHarm.

Harmonized API

The harmonized API lets you use different foundation models without the need to change the client code. It does so by taking the OpenAI API as the standard and mapping other model APIs to it. This includes standardizing message formats, model parameters, and response formats. The harmonized API is integrated into the templating module, the model configuration, and the orchestration response.

In the OpenAI format, a prompt and its response can contain a list of messages with a role and content. The role can be system, user, or assistant and the content is the text of the message. The response can contain a list of choices with index, messages, and a finish_reason.

For more information see Harmonized API

Next lesson