The Prompt Optimizer

Objective

After completing this lesson, you will be able to use the Prompt Optimizer to test and refine prompts at scale against real labeled data.

The Prompt Optimizer

Introduction

SAP Generative AI Hub has added three powerful capabilities since you last certified. This unit covers each one: a tool for testing and optimizing prompts at scale, a framework for evaluating models systematically, and a new model type purpose-built for structured business predictions. By the end, you will know what each capability does, how to use it, and what to watch out for.

You shipped a prompt that looked great on five test cases. Three weeks into production, support tickets start coming in. The model is giving inconsistent outputs on inputs it has never seen at scale. You go back and adjust the prompt - but you are adjusting against your gut, not data. This is the problem the Prompt Optimizer solves.

The Prompt Optimizer automates what you have been doing manually. Give it a base prompt, a labeled dataset, and a scoring metric. It generates multiple prompt variants, tests each against your dataset, scores them, and saves the best-performing version per target model back to the Prompt Registry. You review the results. You save what works.

Why it matters

Performance is measured against real data, not your impression of a few examples.

A prompt optimized for one model does not work the same on another. The Optimizer runs independently per target model.

When a model is deprecated, run the Optimizer against the replacement instead of re-engineering the prompt from scratch.

What you need before you start

  • A base prompt template saved in the Prompt Registry

  • A labeled dataset: real inputs with correct expected outputs. Minimum 25 records, maximum 200. JSON format.

  • An evaluation metric that defines what a better response looks like for your task

How to Run it

Steps 1–3 are one-time setup. Steps 4–7 are the optimization loop you will repeat each time you want to test or improve a prompt.

  1. Register your object store secret in SAP AI Launchpad.
  2. Upload your labeled dataset to the object store and register it as an artifact in SAP AI Core.
  3. Confirm your base prompt template is saved in the Prompt Registry
  4. Create an optimization configuration: specify your template, dataset, metric, and target models.
  5. Trigger the execution via ML Operations→Executions.
  6. Review results in the ML Tracking Service. Look for the variant with the highest score on your metric.
  7. Save the best-performing variant as a new version of your template in the Prompt Registry.

Don't make this mistake

Running the Optimizer with a labeled dataset that does not reflect your real production inputs. The Optimizer is only as good as the examples you give it. If your dataset is too clean, too small, or does not cover the edge cases your users will actually send - you will get a well-scored prompt that still fails in production. Garbage in, garbage out.

Note

Requires the SAP AI Core Extended service plan. Not available on Free or Standard tiers.