Getting Started with Generative AI Hub

To access the generative AI hub, begin by setting up your SAP Business Technology Platform (BTP) environment and provisioning the required AI services, ensuring a secure and effective foundation for interacting with generative AI hub.

1. Set Up Your SAP BTP Global Account: The first prerequisite is to have access to a global SAP BTP account. This serves as your central point of entry and management for all services on SAP BTP. It's the administrative umbrella under which your AI capabilities will reside.

   To access your SAP BTP Enterprise Account’s global account and make the initial configurations refer to Setup your Global Account of your SAP BTP Enterprise Account

2. Provision SAP AI Core from the SAP BTP Cockpit: The generative AI hub operates within SAP AI Core, which is a service you provision directly from your SAP BTP cockpit. This provisioning process generates a service key. This service key contains the necessary URLs and credentials that authenticate and authorize your access to your dedicated SAP AI Core instance.

   See the following documentation for this process: Initial setup for SAP AI core

   To set up generative AI hub in SAP AI Core to start consuming LLMs, refer to Set up Generative AI Hub in SAP AI Core.

3. Connect to SAP AI Core Tools: Once SAP AI Core is provisioned, you need to establish a connection from your preferred tools. Generative AI hub can be primarily accessed through SAP AI Launchpad, a user interface for managing and monitoring your AI assets. For developers, programmatic access is also vital, allowing you to connect using tools like Bruno for API testing or Python SDKs for integrating AI capabilities directly into your applications.

   To create a connection between SAP AI Core and the tool of your choice (SAP AI Launchpad, API clients like Postman, SDK for languages like Python ) refer to Set Up Tools to Connect With and Operate SAP AI Core.

4. Securing Access via Roles and Authorizations: Access to the generative AI hub via SAP AI Launchpad is managed through specific roles and authorizations. The generative AI hub is available in SAP AI Core exclusively with the extended service plan.

   To utilize the generative AI hub via SAP AI Launchpad, users must be assigned one of the following roles or have a role collection that includes one of these roles:

   genai_manager

   prompt_manager

   genai_experimenter

   prompt_experimenter

For additional details, refer to Roles and Authorizations.

Once you have accessed generative AI hub, the next step is to deploy the specific LLMs you intend to use. Deployment in the generative AI hub means instantiating a use-case-specific LLM configuration that your applications can then consume.

You can start with an orchestration deployment. There is no need to deploy individual models and their configuration. You can use a single orchestration deployment to configure and consume models in generative AI hub.

Select the Configuration section for your resource group.

Fill in Deployment Details, under Configurations, with the following details:

• Name: "orchestration"
• Scenario: "orchestration"
• Version: "0.0.1"
• Executable: "orchestration"

Click Next.

When prompted, click Create Deployment. Continue through the setup by clicking Next until you receive the deployment confirmation.

Once the deployment begins, continue to the status page. Verify that the Deployment Status changes to Running (refer to following screenshot for reference).

You can explicitly control which LLMs are used for orchestration deployment. This is crucial for enforcing internal standards, compliance, and cost management, ensuring that approved models are accessible for use cases.

To restrict model access, include modelFilterList and modelFilterListType within the configuration.

• modelFilterList: This is a list of specific modelNames and, optionally, modelVersions that you want to either allow or deny. If modelVersions is not defined for a modelName, all versions of that model are considered.
• modelFilterListType: This parameter controls how the modelFilterList should be interpreted:
  • deny: Excludes the models and defined versions from use within this orchestration deployment. Any LLM in the modelFilterList will not be available.
  • allow: Only allows the models and defined versions in the modelFilterList to be used within this orchestration deployment. Any LLM not in the modelFilterList will be unavailable.

This mechanism enables you to implement internal policies, such as only allowing certain LLMs that have been pre-approved for specific types of data or performance characteristics.

It uses the following JSON list in modelFilterList.

12
[{"modelName": "gpt-4.1-nano"},{"modelName": "gpt-4o-mini"},{"modelName": "gemini-2.0-flash-lite"},{"modelName": "mistralai--mistral-small-instruct"}]

This instance is restricted to the specified models. Since no version is specified, the latest available version from the generative AI hub will be used.

This is a powerful way to enforce internal usage policies and temporarily allow or restrict models that are not yet approved for certain tasks.

Specific models can also be deployed according to your requirements. For example, you can deploy models like cohere--command-a-reasoning , which is not available in the orchestration deployment.

1. Create a Deployment: This step involves creating a deployment, which can be done either programmatically through APIs or via the user-friendly interface of the SAP AI Launchpad. When creating a deployment, you will reference a model provider-specific executable – for example, models provided through models from OpenAI (via Azure), GCP Vertex AI, Amazon, Anthropic (via AWS Bedrock), and Meta. You will also configure essential parameters such as the model name (e.g., gpt-4o, Gemini 2.2 Pro, or Claude 3.5 Sonnet) and its version.
2. Access the Deployed LLM: Upon successful deployment, SAP AI Core provides a unique, secure URL for each deployment. This URL acts as the endpoint your applications will call to interact with the deployed LLM. This abstraction simplifies how your applications connect to and utilize these powerful models, regardless of the underlying model provider.

Once your LLMs are deployed, you’ll need a strategy for managing new model versions as they become available. This is a crucial part of the LLM lifecycle for your applications. You typically choose one of two main strategies when configuring your deployment: auto upgrade or manual upgrades. This decision impacts control, effort, and application stability.

This strategy tells your deployment to always use the newest supported version of a specified model.

• How it Works: When setting up your LLM configuration (whether for an orchestration workflow or a direct model deployment), you set the modelVersion parameter to latest. This will automatically use the most recent version of the specified model that is supported.
• Benefits:
  • Less Manual Effort: Your deployment automatically upgrades to a newer version, potentially giving you access to model improvements without needing your intervention.
  • Continuous Updates: You automatically receive the latest bug fixes, performance enhancements, and new capabilities for that model.
• Risks and Limitations:
  • Less Control: Your application might suddenly start using a new model version you haven’t tested. This could introduce subtle changes in output behavior or performance.
  • Potential for Instability: If your application relies on highly consistent LLM output, unexpected behavioral shifts from a new version can cause issues.
  • Same Model Only: This strategy only updates versions of the same model (for example, from an older gpt-4o to a newer gpt-4o version). It does not automatically switch to a completely different model (for example, from gpt-4o to gpt-5).

This strategy gives you precise control over which model version your deployment uses.

• How it Works:: When creating your orchestration or model configuration, you specify an exact modelVersion, such as "2024-05-13". As you prepare to move to a newer model version, whether because the current version is retiring or you choose to upgrade, follow these steps:
  • Research: Identify a suitable newer model version. You can use the SAP Note 3437766, the Discovery API, or the Model Library in SAP AI Launchpad for benchmarking and information.

  • Use the following ways to manage model version:

    a. Update Configuration: Easily change your configuration to the new, specific model version using the harmonized API in orchestration.

    b. Apply (if direct deployment): If you are managing your own custom endpoint for a direct LLM deployment, you might then patch your existing deployment to reference this new configuration ID.

• Benefits:
  • Full Control: You decide precisely which model version your application uses and when to make changes.
  • Comprehensive Testing: Allows you to test new model versions comprehensively in a controlled environment before deploying them to production.
  • Predictable Behavior: Essential for production systems where consistent LLM behavior is critical, mitigating risks from unexpected changes.
• Risks - More Manual Effort: You must actively monitor model retirement dates and perform the upgrade steps yourself.

The ideal upgrade strategy depends on your application’s priorities:

• For most production applications, especially those sensitive to potential behavioral changes, specifying a fixed modelVersion (Manual Upgrade) is the recommended best practice. This provides the necessary control and predictability. Most application owners prefer this approach to avoid unexpected behavior shifts that can occur with new LLM versions. Fixing the version allows you to plan and test updates proactively.
• For non-critical or exploratory applications where minimizing operational overhead is preferred, using Auto Upgrade (modelVersion: latest) can be suitable. However, always be aware that behavior might change with new versions, and you’ll still need to manually manage transitions if you decide to switch to a different model entirely.

You have successfully navigated the foundational steps for getting started with the generative AI hub. You now understand the prerequisites for gaining access and the step-by-step procedure for deploying the Orchestration Service, which provides the central endpoint for all your orchestrated LLM interactions. You also learned how to explicitly restrict model access within deployments and the considerations for managing model upgrades, balancing the need for control with the desire for automation.

This comprehensive setup is a prerequisite for developing powerful and secure AI-driven applications. It allows you to focus on crafting intelligent prompts and solving business problems without worrying about the underlying infrastructure management. You are now ready to begin leveraging the full power of the generative AI hub.