Using SAP Cloud SDK for AI to Leverage the Power of LLMs

Before developing prompts using SAP Cloud SDK for AI (Python) - generative, you need to enable the use of AI models in generative AI hub. This is done through helper functions. You can see detailed functions in the repository here.

One of the function interfaces with the AI Core SDK to manage deployment tasks efficiently. It includes a spinner function to provide real-time updates during long operations, maintaining user engagement. The function checks for an existing deployment and either retrieves it or creates a new one, coordinating configurations and ensuring smooth execution within a specified timeout. Refer to the detailed code in the repository.

In case of any errors or issues while executing the code, you may refer some of the troubleshooting tips.

The following function sets up and sends requests to an AI orchestration service.

1234567891011121314151617181920212223242526272829303132
import pathlib
import yaml

from gen_ai_hub.proxy import get_proxy_client
from ai_api_client_sdk.models.status import Status

from gen_ai_hub.orchestration.models.config import OrchestrationConfig
from gen_ai_hub.orchestration.models.llm import LLM
from gen_ai_hub.orchestration.models.message import SystemMessage, UserMessage
from gen_ai_hub.orchestration.models.template import Template, TemplateValue
from gen_ai_hub.orchestration.service import OrchestrationService

client = get_proxy_client()
deployment = retrieve_or_deploy_orchestration(client.ai_core_client)
orchestration_service = OrchestrationService(api_url=deployment.deployment_url, proxy_client=client)

def send_request(prompt, _print=True, _model='meta--llama3.1-70b-instruct', **kwargs):
    config = OrchestrationConfig(
        llm=LLM(name=_model),
        # template=Template(messages=[SystemMessage(prompt),UserMessage("{{?input}}")
        template=prompt
    )
    template_values = [TemplateValue(name=key, value=value) for key, value in kwargs.items()]
    answer = orchestration_service.run(config=config, template_values=template_values)
    result = answer.module_results.llm.choices[0].message.content
    
    if _print:
        formatted_prompt = "\n".join([t.content for t in answer.module_results.templating])
        print(f"<-- PROMPT --->\n{formatted_prompt}\n<--- RESPONSE --->\n{result}")

    return result

It imports necessary libraries and modules, sets up a proxy client, and retrieves or deploys an orchestration configuration. It defines a function, "send_request", which configures and sends prompts to an AI model, then formats and prints the response. The function can access all the configured models in your generative AI hub instance using orchestration deployment. This setup aims to streamline AI-driven tasks using a consistent and reusable approach.

We will now find the urgency and sentiment in an incoming mail.

Use the following code to read a random mail with the ID assigned earlier.

1
mail = dev_set[EXAMPLE_MESSAGE_IDX]

As we saw earlier, developing a prompt to solve a business problem is an iterative process. This process is explained in the following steps:

1. Develop a basic prompt: We'll start with a basic prompt and then develop the prompt, finding an output that can be used by applications within the company. The first prompt is:
   Python

   Copy codeSwitch to dark mode

   12345678910111213141516171819
   from gen_ai_hub.orchestration.models.message import SystemMessage, UserMessage
   from gen_ai_hub.orchestration.models.template import Template, TemplateValue
   
   prompt_1 = Template(
       messages=[
           SystemMessage(
               """You are an intelligent assistant. Your task is to extract:
               - urgency
               - sentiment
   Giving the following message:"""
           ),
           UserMessage("{{?input}}")
       ]      
   )
   
   f_1 = partial(send_request, prompt=prompt_1)
   
   response = f_1(input=mail["message"])
   

   This code imports necessary classes from the gen_ai_hub.orchestration.models package to create a structured prompt for an intelligent assistant. It defines the system and user messages. The assistant is tasked with extracting urgency and sentiment from a given message. The partial function sets up the request with the prompt, and the code sends the request using the input message from mail["message"].

   You can see the following output:

   

   You can utilize different models using the same function. This is a benefit of the orchestration feature of generative AI hub.

   The current lengthy response is not useful for our objective of assigning urgency and sentiment to emails for software input, so we should continue to develop it.

   Note

   You may get slightly different responses to the one shown here and in all the remaining responses of models shown in this learning journey.

   When you execute the same prompt in your machine, an LLM produces varying outputs due to its probabilistic nature, temperature setting, and non-deterministic architecture, leading to different responses even with slight setting changes or internal state shifts.

2. Assign values to urgency and sentiment: We'll now assign some basic urgency and sentiment values and execute the prompt.
   Python

   Copy codeSwitch to dark mode

   12345678910111213141516
   prompt_2 = Template(
       messages=[
           SystemMessage(
               """You are an intelligent assistant. Your task is to extract:
   - "urgency" as one of {{?urgency}}
   - "sentiment" as one of {{?sentiment}}
   Giving the following message:"""
           ),
           UserMessage("{{?input}}")
       ]      
   )
   
   f_2 = partial(send_request, prompt=prompt_2, **option_lists)
   
   response = f_2(input=mail["message"])
   

   The code defines a prompt for extracting "urgency" and "sentiment" from a given message. It uses the "partial" function to configure a request with predefined options. Finally, it processes the email message by passing it through the configured request to get the needed details. This ensures consistent and accurate data extraction from messages.

   You get a response. You will notice that the values for urgency and sentiment are based on assigned values.

3. Generate JSON output: A software requires input in a specific format, and JSON is chosen for this purpose. JSON is a suitable choice because it's a lightweight, language-independent, and easy-to-parse format that enables efficient data exchange and simplifies development.

   We use the following code:

   Python

   Copy codeSwitch to dark mode

   12345678910111213141516
   prompt_3 = Template(
       messages=[
           SystemMessage(
               """You are an intelligent assistant. Extract and return a json with the follwoing keys and values:
   - "urgency" as one of {{?urgency}}
   - "sentiment" as one of {{?sentiment}}
   Your complete message should be a valid json string that can be read directly and only contain the keys mentioned in the list above.
   Giving the following message:"""
           ),
           UserMessage("{{?input}}")
       ]      
   )
   f_3 = partial(send_request, prompt=prompt_3, **option_lists)
   
   response = f_3(input=mail["message"])
   

   This code prepares and sends a request to extract specific information from a message. It uses a predefined prompt template to instruct the system to extract "urgency" and "sentiment" and return them in JSON format. The "partial" function sets up this request with fixed options, and then it sends the request using the message content.

   You will see a JSON output.

4. Ensure that the JSON formatting is correct: This step is needed when the JSON output format is not clear in the previous prompt. You can make it clearer using the following code:
   Python

   Copy codeSwitch to dark mode

   123456789101112131415161718
   prompt_4 = Template(
       messages=[
           SystemMessage(
               """You are an intelligent assistant. Extract and return a json with the follwoing keys and values:
   - "urgency" as one of {{?urgency}}
   - "sentiment" as one of {{?sentiment}}
   Your complete message should be a valid json string that can be read directly and only contain the keys mentioned in the list above. Never enclose it in ```json...```, no newlines, no unnessacary whitespaces.
   
   Giving the following message:"""
           ),
           UserMessage("{{?input}}")
       ]      
   )
   
   f_4 = partial(send_request, prompt=prompt_4, **option_lists)
   
   response = f_4(input=mail["message"])
   

   This code defines a prompt template called "prompt_4" that specifies instructions for extracting "urgency" and "sentiment" from an input message and returning it as a JSON string. This prompt gives clear instruction to provide a clean format without any quotes or white spaces.

   You will see that the JSON output is clear now. This is ready for software consumption.

5. Simple categories based on business functions: We have associated urgency and sentiment to a mail. However, we also need more tags for each message to categorize them for business needs. We can start with a simple code.
   Python

   Copy codeSwitch to dark mode

   1234567891011121314
   prompt_5 = Template(
       messages=[
           SystemMessage(
               """You are an intelligent assistant. Assign a list of matching support category to the message.
   
   Giving the following message:"""
           ),
           UserMessage("{{?input}}")
       ]      
   )
   f_5 = partial(send_request, prompt=prompt_5)
   
   response = f_5(input=mail["message"])
   

   This code sets up a template for assigning support categories based on user messages. It prepares a system message stating the assistant's role, incorporates user input, and defines a partial function to send the request. Finally, it uses this function to process the message and generate a response, helping categorize support queries efficiently.

   You will see that categories are assigned to the message. The response assigns the support categories to the original message, streamlining customer support processes.

6. Assigning values to categories from a list: The categories generated above have overlapping values, such as urgency, and should be aligned with company-defined values to address business needs effectively.
   Python

   Copy codeSwitch to dark mode

   1234567891011121314151617181920212223
   prompt_6 = Template(
       messages=[
           SystemMessage(
               """You are an intelligent assistant. Assign a list of best matching support category tags to the message:
   {{?categories}}
   
   Giving the following message:"""
           ),
           UserMessage("{{?input}}")
       ]
   )
   
   # Prepare the options
   option_lists = {
       'categories': ', '.join(f"`{entry}`" for entry in categories),
   }
   
   # Create the partial function
   f_6 = partial(send_request, prompt=prompt_6, **option_lists)
   
   # Call it
   response = f_6(input=mail["message"])
   

   This code sets up a template for classifying messages by support categories. It defines the system and user messages, prepares a list of category options, and creates a function to send the classification request using these options. Then, it calls the function with the input message to get the response, effectively assigning suitable tags based on the content of the message.

   You will see that categories are assigned to values from the list. These are streamlined for business processing.

7. Generate JSON output for categories values: Similar to step 3 earlier, we need JSON output for processing these values in a software. We also ensure that the format of JSON output is without any unnecessary symbols or space.

   We use the following code:

   Python

   Copy codeSwitch to dark mode

   12345678910111213141516
   prompt_7 = Template(
       messages=[
           SystemMessage(
               """You are an intelligent assistant. Extract and return a json with the follwoing keys and values:
   - "categories" list of the best matching support category tags from: {{?categories}}
   Your complete message should be a valid json string that can be read directly and only contain the keys mentioned in the list above. Never enclose it in ```json...```, no newlines, no unnessacary whitespaces.
   
   Giving the following message:"""
           ),
           UserMessage("{{?input}}")
       ]      
   )
   f_7 = partial(send_request, prompt=prompt_7, **option_lists)
   
   response = f_7(input=mail["message"])
   

   This code creates a precise prompt template to extract specific category tags from an input message and return them in a valid JSON format. It then uses a predefined function, "send_request", with the constructed prompt and some options to generate the necessary response from the given input message. This ensures consistency and accuracy in data extraction.

   You will see categories in JSON output that can be processed in a software.

8. Combining all the steps: We have used SAP Cloud SDK for AI (Python) - generative to arrive at proper values of urgency, sentiment, and categories in JSON format step-by-step. Now, let's combine all these steps into a single prompt.
   Python

   Copy codeSwitch to dark mode

   123456789101112131415161718192021222324
   prompt_8 = Template(
       messages=[
           SystemMessage(
               """You are an intelligent assistant. Extract and return a json with the follwoing keys and values:
   - "urgency" as one of {{?urgency}}
   - "sentiment" as one of {{?sentiment}}
   - "categories" list of the best matching support category tags from: {{?categories}}
   Your complete message should be a valid json string that can be read directly and only contain the keys mentioned in the list above. Never enclose it in ```json...```, no newlines, no unnessacary whitespaces.
   
   Giving the following message:"""
           ),
           UserMessage("{{?input}}")
       ]      
   )
   
   option_lists = {
       'urgency': ', '.join(f"`{entry}`" for entry in urgency),
       'sentiment': ', '.join(f"`{entry}`" for entry in sentiment),
       'categories': ', '.join(f"`{entry}`" for entry in categories),
   }
   f_8 = partial(send_request, prompt=prompt_8, **option_lists)
   
   response = f_8(input=mail["message"])
   

   The code combines all the previous steps in one prompt. It helps processing an email message to identify and return key information. By using a predefined template, it extracts details like urgency, sentiment, and categories, then formats them as a JSON string. It ensures that the output is clean and directly usable by invoking a function with the given parameters.

   The following screenshot shows the output of this code.

   

   You can see the consolidated output that assigns urgency, sentiment, and categories to customer messages that can be used in software.

You have successfully developed a prompt to move towards a solution of the problem using SAP Cloud SDK for AI (Python) - generative

You’ve successfully navigated the iterative process of prompt development, from a basic request to a highly refined, application-ready output. By leveraging the SAP Cloud SDK for AI and the generative AI hub’s orchestration capabilities, you’ve extracted complex information like urgency, sentiment, and categories from an email, formatting it into clean JSON that is directly consumable by software. This process highlights how continuous prompt refinement is key to solving real-world business problems with Generative AI simplified by using SAP Cloud SDK for AI.