Monitoring and Improving Detection Quality

Objective

After completing this lesson, you will be able to monitor and improve detection quality.

Calibration

Simulation of detection strategies in SAP BIS is essential for developing a robust and agile fraud detection and prevention framework that protects the organization's assets and reputation while maintaining operational efficiency.

Here are some reasons why the simulation is important:

Simulations allow organizations to test and refine detection rules and strategies in a controlled environment, reducing the risk of false positives and ensuring legitimate transactions aren't wrongly flagged.
By simulating detection strategies, businesses can identify and eliminate inefficiencies in their processes, ensuring that resources are focused on genuine risks rather than being wasted on ineffective measures.
Simulations help in assessing the potential operational and financial impact of detection strategies without the cost or risk of implementing them in a live environment.
Simulating detection strategies allows for the fine-tuning of algorithms and rules, improving the accuracy and reliability of fraud detection systems.
Simulations enable organizations to test against a variety of "what-if" scenarios, preparing them better for unexpected or rare fraudulent activities.

The SAP BIS Calibration tool offers the following functions:

Fine-tune the settings and compare with a reference strategy
You can compare the actual result of the strategy with the simulated results.
You can also define a selection period for the data that is relevant for the simulation.
You can change the threshold of the strategy.
You can also change the weighting factors, and the parameter values of the detection methods.
You can compare the actual and simulated results of the strategy with the results of another strategy that has the same detection object type (reference strategy).

Note

Once the detection strategy is adjusted, you can save the new set of weighting factors, parameters, and the threshold. By saving the results of the calibration, the system creates a new detection strategy version with status inactive. An existing inactive strategy version will be overwritten.

Which KPIs are displayed in the Calibration:

KPI	Explanation
Confirmed	Number of alert items that have been created by this strategy and are classified as Confirmed within a given time frame.
False Positives	Number of alert items that have been created by this strategy and are classified as False Alarm within a given time frame.
Confirmed	Number of alert items that would have been created and classified as Confirmed for detection objects that already have been identified as Confirmed in the actual data.
False Positives	Number of alert items that would have been created and classified as Confirmed for detection objects that already have been identified as False Alarm in the actual data.
New Alert Items	Number of alert items that will be created during simulation and that have not been created within the actual data.
Missed Alert Items	Number of alert items that were created by the strategy in the past, but were not found during the current simulation.
Found Alert Items	Number of alert items that were both created by the strategy in the past and were found during the current simulation.
Risk Value of New Alert Items	Sum of all the risk values of the new alert items found during the current simulation that were not found by the strategy in the past.
Risk Value of Found Alert Items	Sum of all the risk values of the alert items that were both created by the strategy in the past and were found during the current simulation.
Risk Value of Missed Alert Items	Sum of all the risk values of the alert items that were created by the strategy in the past, but were not found during the current simulation.
Efficiency (Actual)	Share of Confirmed alert items from the number of classified alert items, created by the strategy within a given time frame: Confirmed (actual) / (Confirmed (actual) + False Positive (actual))
Efficiency (Simulation)	Share of simulated Confirmed alert items from the number of classified alert items, created by the simulated strategy within a given time frame: Confirmed (sim) / (Confirmed (actual) + False Positive (sim))

Optimization

To improve quality of existing strategies you leverage the automated optimization functionality of the calibration tool.

This feature uses historical alert decision data (Confirmed Alert Items and Known False Positives) to optimize a detection strategy by applying mathematical modeling techniques to find a best fit for the weighting factors in a detection method.

Please note the following prerequisites to use this feature:

A set of at least 100 historical data findings – with both confirmed alerts and findings of no fraud – would be required to optimize a detection strategy that has 10 detection methods.
A detection strategy must contain at least two detection methods to be optimized automatically.

You can use the Profit Factor and Cost Factor input parameters to tell Find Best Values what goals to follow when it optimizes a detection strategy. How should the optimization balance the goals of finding as many confirmed cases as possible while minimizing the number of false positives for which alerts are raised?

With Profit Factor, you say how important it is to you to find as many real cases of fraud as possible. A high Profit Factor tells Find Best Values to optimize for finding a high proportion of the real cases of fraud in your historical data set.

Hint

The meaning behind it is: I expect to get this much value – Profit Factor value – from finding each case of real fraud.

With Cost Factor, you say how important it is to you to avoid false-positives, that is, alerts for data that has been shown not to be fraudulent.

Hint

The meaning behind it is: I expect it to cost me this much – Cost Factor value – to clear a false-positive alert and recognize that it is false.

You access this feature in the Find Best Values menu on the standard Calibration screen.

Simulation

Another option to simulate detection strategy in SAP BIS is to use the simulation option available in the mass detection run settings.

This option is used for detection strategies containing the address screening methods, since the calibration tool does not support the address screening functionality.

In this case the simulation is performed as a background job and the results are stored in the following back-end views:

FRA_V_MD_S2_OVW  Mass Detection Simulation Result Overview. In this view, you will see all detection objects and their score that were considered in this simulation run. In addition, you can display the detection strategy, strategy version, selection date, detection object type, threshold, and so on
FRA_V_MD_S2_RES Mass Detection Simulation Detail Results. In this view, you will see the details for each detection object and its applicable detection methods
FRA_V_MD_S2_TXT Mass Detection Simulation Details with Texts. In this view, you will see the text messages provided by the detection method for each detection object

Note

Use the external ID of the detection run to find the simulation results in the views.

Next lesson