Analyzing the results of a classification model in SAP Analytics Cloud Smart Predict

• Predictive power measures the accuracy of the predictive model. It takes a value between 0% and 100%. This value should be as close as possible to 100%, without being equal to 100%.
• Predictive confidence indicates the capacity of your predictive model to achieve the same degree of accuracy when you apply it to a new dataset, which has the same characteristics as the training dataset. Prediction confidence takes a value between 0% and 100%. This value should be as close as possible to 100%.

Gives the frequency (%) of the two target categories. In this case 1 and 0 in the training and validation sub-sets.

Shows the relative importance of each variable used in the predictive model. It examines the influence on the target of the top 5 variables used in the predictive model. It is a useful report to spot if there are leaker variables in a model, as the leaker will have a suspiciously high contribution, overwhelming all of the other variables.

Compares the classification model performance (on the validation sub-sample of the data) to a random model and a hypothetically perfect (100% accurate) model.

The % detected target curve compares the model to the hypothetically perfect and random models. It shows the percentage of the total population (x-axis) that corresponds to the % of positive detected targets (y-axis) given by the classification model.

• If the model was perfectly accurate, then the blue model curve would overlap the green perfect model curve. Predictive power = 100%.
• If the model was perfectly inaccurate, then the blue model curve would overlap the red random curve. Predictive power = 0%.

The influencer contributions report shows the relative importance of each influencer used in the predictive model. It allows you to you to examine the influence on the target of each influencer used in the predictive model.

Examines the relative significance of all of the variables within the model. This is a useful report to spot if there are leaker variables in a model, as the leaker will have a suspiciously high contribution, overwhelming all of the other variables.

The influencers are sorted by decreasing importance. The most contributive ones are those that best explain the target. Only the contributive influencers are displayed in the reports, the influencers with no contribution are hidden.

The sum of their contributions equals 100%

Shows groupings of categories of a variable, where all the categories in a group share the same influence on the target variable. The report shows which category group has the most influence, however you can change the category group by using the input help in this chart to further examine the results by influencer.

In this example, for the influencer AGE the category 45-54 has the highest positive influence on the target.

Shows the details of how the grouped categories influence the target variable over the selected dataset.

• The x-axis displays the target mean. For a nominal target, the target mean is the frequency of positive cases for the target variable contained in the training dataset.
• The y-axis displays the frequency of the grouped category in the selected dataset.

Confusion matrix: Shows the performance of a classification algorithm by comparing the predicted value of the target variable with its actual value.

Also, known as the error matrix, each column represents the observations in a predicted category, while each row represents the observations in an actual class.

You can use the confusion matrix to compute metrics to associate with different needs.

Here's how to read the metrics:

• N: Number of observations
• TP (true positive): Number of correctly predicted positive targets
• FN (false negative): Number of actual positive targets that have been predicted negative
• FP (false positive): Number of actual negative targets that have been predicted positive
• TN (true negative): Number of correctly predicted negative targets

Associate a profit/cost with the positive categories (observations that belong to the population you want to target) of the confusion matrix. You can visualize your profit based on the selected threshold, or automatically select the threshold based on your profit parameters.

Set the threshold that determines which values are considered positive (see the relevant-related link) and provide the following:

• Cost per predicted positive: Is where you define a cost per observations classified as positive by the confusion matrix. This covers the costs both for true positive target (actual positive targets that have been predicted as positive) and false positive target (actual negative targets that have been predicted positive).
• Profit per actual positive: Is where you define a profit per true positive target (targets correctly predicted as positive) identified by the confusion matrix.

To see the threshold that will give you a maximum profit for the profit parameters you have set, you can select the maximize profit button in this report.

This table is updated accordingly to calculate your profit/cost. You obtain an estimation of the gap between the gain of the action based on a random selection (without any predictive model) and the gain based on the selection.

Evaluate the accuracy of your predictive model using the performance curves. Use the performance curves report to compare the performance of your predictive model to a random and a hypothetical perfect predictive model.

The % detected target curve is also displayed in the overview report. It compares the model to the perfect and random models.

It shows the percentage of the total population (on the x-axis) that corresponds to the % of positive detected targets (shown on the y-axis) given by the classification model.

The lift curve shows how much better the predictive model is than a random selection.

The x-axis shows the percentage of the population and is ordered from highest probability of positive detected target to lowest probability of positive detected target.

The y-axis shows how much better your model is than the random predictive model (lift).

The random selection (shown in red) is set where Lift = 1.00

Case study: In the example, a bank wants to create an advertising campaign. They have built a classification model to target which customers to send the campaign to and

The predictive model will classify the customers into two categories:

• Positive targets: the customers will response to the campaign.
• Negative targets: the customers will not response to the campaign.

The lift curve shows that by selecting 20% of the total population, the campaign would reach 3.09 times more positive cases using the predictive model than with a random customer selection (set at lift = 1.00). A perfect predictive model would reach 4.19 times more positive cases than the random selection.

The sensitivity curve, or ROC (Receiver Operating Characteristic) curve, shows how well a model discriminates in terms of the trade-off between sensitivity and specificity, or, in effect, between correct and mistaken detection, as the detection threshold is varied.

The curve is constructed by plotting the true positive rate (TPR) against the false positive rate (FPR).

The true positive rate is the proportion of observations that were correctly predicted to be positive out of all positive observations (TP/(TP + FN)). Similarly, the false positive rate is the proportion of observations that are incorrectly predicted to be positive out of all negative observations (FP/(TN + FP)). For example, in medical testing, the true positive rate is the rate in which people are correctly identified to test positive for the disease in question.

• Sensitivity: The probability that the model predicts a positive outcome for an observation when the outcome is actually positive. This is the TPR.
• Specificity: The probability that the model predicts a negative outcome for an observation when the outcome is actually negative. 1-Specificity is the FPR. A model with high sensitivity and high specificity will have a sensitivity curve that hugs the top left corner of the plot.

A model with low sensitivity and low specificity will have a curve that is close to the 45-degree diagonal line. As a baseline, a random classifier is expected to give points lying along the diagonal (FPR = TPR). The closer the curve comes to the 45-degree diagonal of the sensitivity space, the less accurate the test.

Lorenz (1-sensitivity) displays the cumulative proportion of missed signals (false negatives) accounted for by the records corresponding to the bottom x% of model scores.

The lorenz (specificity) curve, displaying the cumulative proportion of true negatives (specificity) accounted for by the bottom x% of model scores, is also available by using the drop down menu for this graph.

The density curves display the density function of the variable score for the target category (curve density "positive") and for the non-target category (curve density "negative").

The density curves display the density function of the score (probability that an observation belongs to each class) for positive and negative targets.

• The estimated density function in an interval is equal to the formula: (Number of observations in the interval/total number of observation)/length of the interval.
• The length of an interval is its upper bound minus its lower bound.
• The X axis shows the score and the Y axis shows the density.

As a default view, a line chart is displayed with the following density curves:

• Positives: This curve displays the distribution of population with positive target value per score value.
• Negatives: This curve displays the distribution of population with negative target value per score value.