Using Generative-AI-hub-SDK to Leverage the Power of LLMs

You've installed and configured generative-ai-hub-sdk. You've also configured orchestration services. We begin with installing packages and then loading data.

1
!pip install -U "generative-ai-hub-sdk>=3.1" tqdm

This code installs the necessary software packages. By using !pip install -U "generative-ai-hub-sdk>=3.1" tqdm, it ensures you have the latest version of the generative-ai-hub-sdk (version 3.1 or later) and the "tqdm" library. The "tqdm" package is a progress bar library that helps display the progress of tasks in the console.

12345
from typing import Literal, Type, Union, Dict, Any, List, Callable
import re, pathlib, json, time
from functools import partial
EXAMPLE_MESSAGE_IDX = 10

This code begins by importing necessary modules and types from various Python libraries, including typing, re, pathlib, json, time, and functools. It then defines a constant, EXAMPLE_MESSAGE_IDX set to 10 to select a random 10^th message from a dataset.

Next, you need to load and preprocess a dataset of emails stored in a JSON file. You can split dataset into development and testing sets with a smaller test set created for more focused evaluation. You can also process email data, to extract and organize categories, urgency, and sentiment into sets for further analysis.

Before developing prompts using generative-ai-hub-sdk, you need to enable the use of AI models in generative AI hub. This is done through helper functions. You can see detailed functions in the repository here.

One of the function interfaces with the AI Core SDK to manage deployment tasks efficiently. It includes a spinner function to provide real-time updates during long operations, maintaining user engagement. The function checks for an existing deployment and either retrieves it or creates a new one, coordinating configurations and ensuring smooth execution within a specified timeout. Refer to the detailed code in the repository here.and the relevant readme file.

In case of any errors or issues while executing the code, you may refer some of the troubleshooting tips.

The following function sets up and sends requests to an AI orchestration service.

123456789101112131415161718192021222324252627282930
import pathlib
import yaml

from gen_ai_hub.proxy import get_proxy_client
from ai_api_client_sdk.models.status import Status

from gen_ai_hub.orchestration.models.config import OrchestrationConfig
from gen_ai_hub.orchestration.models.llm import LLM
from gen_ai_hub.orchestration.models.message import SystemMessage, UserMessage
from gen_ai_hub.orchestration.models.template import Template, TemplateValue
from gen_ai_hub.orchestration.models.content_filter import AzureContentFilter
from gen_ai_hub.orchestration.service import OrchestrationService

client = get_proxy_client()
deployment = retrieve_or_deploy_orchestration(client.ai_core_client)
orchestration_service = OrchestrationService(api_url=deployment.deployment_url, proxy_client=client)

def send_request(prompt, _print=True, _model='meta--llama3-70b-instruct', **kwargs):
    config = OrchestrationConfig(
        llm=LLM(name=_model),
        template=Template(messages=[UserMessage(prompt)])
    )
    template_values = [TemplateValue(name=key, value=value) for key, value in kwargs.items()]
    answer = orchestration_service.run(config=config, template_values=template_values)
    result = answer.module_results.llm.choices[0].message.content
    if _print:
        formatted_prompt = answer.module_results.templating[0].content
        print(f"<-- PROMPT --->\n{formatted_prompt if _print else prompt}\n<--- RESPONSE --->\n{result}")   
    return result

It imports necessary libraries and modules, sets up a proxy client, and retrieves or deploys an orchestration configuration. It defines a function, "send_request", which configures and sends prompts to an AI model, then formats and prints the response. The function can access all the configured models in your generative AI hub instance using orchestration deployment. This setup aims to streamline AI-driven tasks using a consistent and reusable approach.

We will now find the urgency and sentiment in an incoming mail.

Use the following code to read a random mail with the ID assigned earlier.

1
mail = dev_set[EXAMPLE_MESSAGE_IDX]

As we saw earlier, developing a prompt to solve a business problem is an iterative process. This process is explained in the following steps:

1. Develop a basic prompt: We'll start with a basic prompt and then develop the prompt, finding an output that can be used by applications within the company. The first prompt is:
   Python

   Copy codeSwitch to dark mode

   12345678910111213
   prompt_1 = """Giving the following message:
   ---
   {{?input}}
   ---
   Your task is to extract
   - urgency
   - sentiment
   """
   
   f_1 = partial(send_request, prompt=prompt_1)
   
   response = f_1(input=mail["message"])
   

   We start with using an open-source model like meta--llama3-70b-instruct using the ‘send_request’ function. The default model in this function is meta--llama3-70b-instruct.

   Note that we don’t need to copy the entire mail or model details all the time because we are using generative-ai-hub-sdk.

   You can see the following output:

   

   You will see the output prompt. You have used SDK to generate a prompt and response.

   You can see prompt part of the output.

   

   You can see response part of the output.

   You see the result upon execution. You can utilize different models using the same function. This is a benefit of the orchestration feature of generative AI hub.

   The current lengthy response is not useful for our objective of assigning urgency and sentiment to emails for software input, so we should continue to develop it.

   Note

   You may get slightly different responses to the one shown here and in all the remaining responses of models shown in this learning journey.

   When you execute the same prompt in your machine, an LLM produces varying outputs due to its probabilistic nature, temperature setting, and non-deterministic architecture, leading to different responses even with slight setting changes or internal state shifts.

2. Assign values to urgency and sentiment: We'll now assign some basic urgency and sentiment values and execute the prompt.
   Python

   Copy codeSwitch to dark mode

   12345678910111213
   prompt_2 = """Giving the following message:
   ---
   {{?input}}
   ---
   Your task is to extract:
   - "urgency" as one of {urgency}
   - "sentiment" as one of {sentiment}
   """
   
   f_2 = partial(send_request, prompt=prompt_2, **option_lists)
   
   response = f_2(input=mail["message"])
   

   The code defines a prompt for extracting "urgency" and "sentiment" from a given message. It uses the "partial" function to configure a request with predefined options. Finally, it processes the email message by passing it through the configured request to get the needed details. This ensures consistent and accurate data extraction from messages.

   You can see the following output:

   

   You can see the prompt generated in the output.

   

   You get a response. You can see that the urgency and sentiment of the mail is generated.

   You can see that the values for urgency and sentiment are based on assigned values.

3. Generate JSON output: A software requires input in a specific format, and JSON is chosen for this purpose. JSON is a suitable choice because it's a lightweight, language-independent, and easy-to-parse format that enables efficient data exchange and simplifies development.

   We use the following code:

   Python

   Copy codeSwitch to dark mode

   12345678910111213
   prompt_3 = """Giving the following message:
   ---
   {{?input}}
   ---
   Extract and return a json with the following keys and values:
   - "urgency" as one of {{?urgency}}
   - "sentiment" as one of {{?sentiment}}
   Your complete message should be a valid json string that can be read directly and only contain the keys mentioned in the list above.
   """
   f_3 = partial(send_request, prompt=prompt_3, **option_lists)
   
   response = f_3(input=mail["message"])
   

   This code prepares and sends a request to extract specific information from a message. It uses a predefined prompt template to instruct the system to extract "urgency" and "sentiment" and return them in JSON format. The "partial" function sets up this request with fixed options, and then it sends the request using the message content.

   You can see the following output:

   

   You can see a JSON output.

4. Ensure that the JSON formatting is correct: This step is needed when the JSON output format is not clear in the previous prompt. You can make it clearer using the following code:
   Python

   Copy codeSwitch to dark mode

   123456789101112
   prompt_4 = """Giving the following message:
   ---
   {{?input}}
   ---
   Extract and return a json with the follwoing keys and values:
   - "urgency" as one of {{?urgency}}
   - "sentiment" as one of {{?sentiment}}
   Your complete message should be a valid json string that can be read directly and only contain the keys mentioned in the list above. Never enclose it in ```json...```, no newlines, no unnessacary whitespaces."""
   f_4 = partial(send_request, prompt=prompt_4, **option_lists)
   
   response = f_4(input=mail["message"])
   

   This code defines a prompt template called "prompt_4" that specifies instructions for extracting "urgency" and "sentiment" from an input message and returning it as a JSON string. This prompt gives clear instruction to provide a clean format without any quotes or whitespaces.

   You can see the following output:

   

   You can see that the JSON output is clear now. This is ready for software consumption.

5. Simple categories based on business functions: We have associated urgency and sentiment to a mail. However, we also need more tags for each message to categorize them for business needs. We can start with a simple code.
   Python

   Copy codeSwitch to dark mode

   12345678910
   prompt_5 = """Giving the following message:
   ---
   {{?input}}
   ---
   Assign a list of matching support category to the message.
   """
   f_5 = partial(send_request, prompt=prompt_5)
   
   response = f_5(input=mail["message"])
   

   This code helps categorize customer support messages by matching them to relevant support categories. It creates a prompt template with the message text, then uses a function, "f_5", to send the formatted request to a categorization service.

   The following screenshot shows the output of this code.

   

   You can see that categories are assigned to the message. The response assigns the support categories to the original message, streamlining customer support processes.

6. Assigning values to categories from a list: The categories generated above have overlapping values, such as urgency, and should be aligned with company-defined values to address business needs effectively.
   Python

   Copy codeSwitch to dark mode

   1234567891011
   prompt_6 = """Giving the following message:
   ---
   {{?input}}
   ---
   Assign a list best matching support category tags to the message:
   {{?categories}}
   """
   f_6 = partial(send_request, prompt=prompt_6, **option_lists)
   
   response = f_6(input=mail["message"])
   

   This code assigns appropriate support category tags to the email message. When executed with the specific input message, it sends the request, automating the tagging process to improve support efficiency.

   The following screenshot shows the output of this code.

   

   You can see that categories are assigned to values from the list. These are streamlined for business processing.

7. Generate JSON output for categories values: Similar to step 3 earlier, we need JSON output for processing these values in a software. We also ensure that the format of JSON output is without any unnecessary symbols or space.

   We use the following code:

   Python

   Copy codeSwitch to dark mode

   123456789101112
   prompt_7 = """Giving the following message:
   ---
   {{?input}}
   ---
   Extract and return a json with the follwoing keys and values:
   - "categories" list of the best matching support category tags from: {{?categories}}
   Your complete message should be a valid json string that can be read directly and only contain the keys mentioned in the list above. Never enclose it in ```json...```, no newlines, no unnessacary whitespaces.
   """
   f_7 = partial(send_request, prompt=prompt_7, **option_lists)
   
   response = f_7(input=mail["message"])
   

   This code creates a precise prompt template to extract specific category tags from an input message and return them in a valid JSON format. It then uses a predefined function, "send_request", with the constructed prompt and some options to generate the necessary response from the given input message. This ensures consistency and accuracy in data extraction.

   The following screenshot shows the output of this code.

   

   You can see categories in JSON output that can be processed in a software.

8. Combining all the steps: We have used generative-ai-hub-sdk to arrive at proper values of urgency, sentiment, and categories in JSON format step-by-step. Now, let's combine all these steps into a single prompt.
   Python

   Copy codeSwitch to dark mode

   1234567891011121314
   prompt_8 = """Giving the following message:
   ---
   {{?input}}
   ---
   Extract and return a json with the follwoing keys and values:
   - "urgency" as one of {{?urgency}}
   - "sentiment" as one of {{?sentiment}}
   - "categories" list of the best matching support category tags from: {{?categories}}
   Your complete message should be a valid json string that can be read directly and only contain the keys mentioned in the list above. Never enclose it in ```json...```, no newlines, no unnessacary whitespaces.
   """
   f_8 = partial(send_request, prompt=prompt_8, **option_lists)
   
   response = f_8(input=mail["message"])
   

   The code combines all the previous steps in one prompt. It helps processing an email message to identify and return key information. By using a predefined template, it extracts details like urgency, sentiment, and categories, then formats them as a JSON string. It ensures that the output is clean and directly usable by invoking a function with the given parameters.

   The following screenshot shows the output of this code.

   

   You can see the consolidated output that assigns urgency, sentiment, and categories to customer messages that can be used in software.

You have successfully developed a prompt to move towards a solution of the problem using generative-ai-hub-sdk.

In the next lesson, we'll learn to use generative-ai-hub-sdk to evaluate the response from this prompt.