Applying A/B Testing in Intelligent Selling Services for SAP Commerce Cloud

In e-Commerce, A/B testing serves as a strategy to compare two variations of an interactive asset. You can compare webpages or user experiences, determining which one performs better.

Essentially, it's an experiment where you can arbitrarily present two versions of a given asset to users. You can use statistical analysis to identify the version that achieves higher performance aligned with a specified conversion objective. Such objectives can include choosing a button, submitting a form, completing a purchase, or any other user action. Thus, A/B testing offers a way of evaluating alterations to a webpage or user interface against the existing design, determining which of them yields superior results.

In intelligent selling services, A/B testing is for evaluating two different product mixes associated with the same merchandising strategy. The evaluation is of both mixes, determining the better one.

As shown in the previous figure:

• ProductMix 1 recommends products related to the current product based on the prebuilt AI algorithm.
• ProductMix 2 leverages a personalized product recommendation, relying on an AI algorithm that analyzes each user's personal click history.

You can position both product mixes strategically on the product detail page, capturing customers' attention. This helps enhance cross-selling and up-selling opportunities. Nevertheless, the key question is: Which method produces better results on the product detail page?

Determining the answer using A/B testing is straightforward. By concurrently implementing both product mixes and observing their performance, you can identify the more effective method. Intelligent selling services can assist you in this procedure. Proceed further to learn more!

Conducting an A/B test requires two product mixes to be associated with a single strategy. You learn about this process in the previous unit.

As shown in the screenshot, the related product mix and personalized product mix are tied to the productDetailPageStrategy. You can distinguish these by the warning icon across the name of the first product mix. If you hover over it, a dialog box appears that states "This product mix has no conditions, so the product mixes below will be ignored".

To create an A/B test for comparing these two product mixes, you can simply choose the Create Test button.

You can enter the name and description for the test, and the following essential properties:

• Start: Specifies the date and time when the test becomes active within the strategy
• End: Specifies the date/time when the test must be deactivated or stopped within the strategy. This is optional. If the date/time is unspecified, the test runs indefinitely or until it identifies the winning product mix.
• Categories: If defined, the test only applies when the current category is one of the assigned values
• Confidence level: Is set to be a fixed 95%, it corresponds to the Bayesian probability that one variation is best
• Product Mixes: Specifies the two product mixes for testing
• Traffic Split: Is fixed at 50% and can’t be changed, due to the nature of A/B testing

To save the test and transition it to the 'scheduled' state, choose the Save button.

These settings ensure that half the customers landing on a page with a merchandising carousel referring to this strategy view content based on product mix variation 1 (relatedProductMix) and the other half view content based on product mix variation 2 (personalizedProductMix).

With this process, creating an A/B test is straightforward! The test automatically initiates at the specified start date and time. However, you can also start it manually by choosing the Start button.

Next, you learn how to track the test and verify the outcomes.

Once you initiate an A/B test, you can monitor it in the A/B-testing reporting workspace. You can monitor through two methods: first, by directly navigating to the A/B testing reporting workspace and, second, by choosing the View Report button on the specific A/B test itself.

In the A/B Test Reporting workspace, you can choose and analyze an ongoing or completed test for a comprehensive evaluation.

When you select a test, the time-series graph shows the metric being tested for the two product mixes, distinguished by unique colors. The solid line represents the actual metric observed for the given product mixes, while the dashed line represents the estimated metric.

The table in the Test Results screen area provides additional metric details.

• Impressions: Represents the unique user impressions, that is, the number of users who viewed the product recommendation.
• Click-through rate (CTR): This metric is calculated by dividing unique clicks by unique user impressions. It gives the proportion of users who clicked on the product after seeing it.
• Range: The range of the estimated CTR relevant to the selected confidence interval. This provides insight into the possibilities of the actual CTR.
• Probability to Be Best: This metric shows the probability that a given product mix is better than the other defined in the test.

In the previous screenshot, personalizedProductMix has a higher CTR than the relatedProductMix. As a result, the time-series graph also shows a higher estimated value for personalizedProductMix.

How do you determine if a test is completed?

The system identifies a product-mix variant as a winning mix if it achieves a probability of 95% or higher. After that, it transitions the test to the successful state, as shown in the following figure:

The test can be manually stopped or automatically stopped upon reaching the scheduled end date or time. Later, you can manually determine the winner based on the final outcomes.

It’s important to note that a successful test doesn't automatically apply the winning product mix in the strategy. In the strategy, you must still manually prioritize the product mix.