Synthetic User Monitoring

SAP Cloud ALM Synthetic User Monitoring is a 24/7 monitoring solution to detect performance and availability issues of web applications from a client-side perspective​.

With Synthetic User Monitoring (SUM), you have the ability to define scenarios based on common user usage of your own or third-party web applications. This means you can simulate user interactions and test the performance and availability of your applications from a client-side perspective. One of the key features of Synthetic User Monitoring is the ability to choose a specific web browser to execute these scenarios. This allows you to accurately measure browser timings, such as UI rendering time, and gain insights into the user experience. Furthermore, Synthetic User Monitoring provides you with 24/7 monitoring of all your critical applications. You can fine-tune the monitoring frequency based on your specific needs, ensuring that you have real-time visibility into the performance and availability of your applications.

The graphic below provides a high-level overview of Synthetic User Monitoring:

The monitoring process includes recording metrics from different measurement locations and networks. This allows you to gather data from various sources and get a comprehensive view of your application's performance. In the event of an availability or performance issue, Synthetic User Monitoring can emit an event through Intelligent Event Processing. This early detection system enables you to identify and resolve problems before they become noticeable to your users. To help you keep track of scenario executions, you can check the immediate and time range-based statuses of these executions. This gives you a quick overview of the current state of your applications.

For a more detailed analysis, Synthetic User Monitoring provides a centralized visualization of all scenario executions. You can select a specific time range and easily compare the performance and availability of your applications over time. If you need to export execution statistics for further analysis, Synthetic User Monitoring offers an analytics API (Application Programming Interface). This allows you to extract data and integrate it into your existing analytics systems. Additionally, you can drill down into the details of each scenario execution, even down to the specific step that created an error. This level of granularity helps you pinpoint issues and troubleshoot them effectively. Finally, Synthetic User Monitoring allows you to compare metrics over time. By re-executing the same scenarios, you can monitor the evolution of performance and availability relative to a particular location. This helps you identify trends and make informed decisions to optimize your applications.

Synthetic User Monitoring simulates a user who accesses the application to monitor. To achieve this:

1. You define the scenarios, which represent sequences of actions to perform in the browser.
2. You define the runners, which are installed in the measurement locations.
3. Synthetic User Monitoring regularly executes the scenarios on the runners. From the executions, Synthetic User Monitoring determines the monitored application availability and performance.

Monitoring is based on annotated Selenium IDE scripts, which is a powerful open source tool that can be used to support most web applications. By leveraging Selenium IDE, developers can create scripts that mimic user interactions on a website. These scripts can then be annotated with monitoring instructions to record important metrics during execution. In addition to monitoring scripts, Synthetic User Monitoring provides a way to regularly execute scenarios on remote runners and record metrics. These runners are compatible with Selenium's remote WebDriver infrastructures, following the W3C standard. The runners act as virtual browsers, executing the scenario actions just like a real user would.

The following flowchart illustrates how this works:

During execution, availability and performance metrics are recorded and stored centrally. This allows for easy access and analysis of the collected data. The monitoring UI provides a comprehensive view of all scenario information, making it easy to track the performance and availability of web applications. Overall, monitoring based on annotated Selenium IDE scripts and Synthetic User Monitoring provide a robust solution for monitoring web applications. By simulating user interactions and recording important metrics, developers and system administrators can gain valuable insights into the performance and availability of their web applications.

Dynamic thresholds in Synthetic User Monitoring are used to evaluate the availability and performance of a services/system with the help of an algorithm enables by Artificial Intelligence (AI).

The following graphic illustrates the concept of Dynamic Thresholds:

In terms of performance evaluation, dynamic thresholds are computed automatically based on historical executions of a scenario. The execution duration of the scenario is compared against these dynamic thresholds. The historical executions serve as a learning window, allowing to predict future durations and determine thresholds for the next period of time. During subsequent executions, if the execution duration exceeds the thresholds, it is considered a performance issue. The purpose of dynamic thresholds is to detect abnormal durations that deviate from the usual duration, indicating a performance problem.

Starting from the Synthetic User Monitoring tile, the Overview screen displays scenarios running on different runners. It provides information about the last execution and statistics for the last hour. The history shown depends on the selected time range, with the default being one hour. The overview also indicates the current status of each scenario and highlights any recent failures. Choosing a scenario in the overview reveals more details.

The following screenshots show the Overview, Scenario Executions and the Execution Details views:

The Executions view shows the specific scenario and the runners it is deployed on. Each block represents one execution, but the size of the block does not correspond to the execution duration. The validity of an execution lasts until the next planned execution. Additional information, such as availability, distribution and performance, is displayed below the execution details.

By choosing a specific execution it provides more detailed information. This includes the actual duration of the execution and the dynamic thresholds at the time of execution. The Execution Details section allows users to analyze the specific execution in depth and understand its performance and status.

The execution expanded feature allows users to view detailed information about a specific execution. This includes the actual duration of the execution and the corresponding dynamic thresholds at the time of execution. Users can also see the availability and performance information for each step of the execution. For example, they can identify poor performance in a specific step taking longer than expected. The expanded view also allows users to remove filters and see all executions in parallel.

The next screenshots show all Executions expanded, Captured Screenshots and the Metric Report views:

When drilling down into a specific execution, you have the option to access additional details. If enabled, you can also view screenshots captured during the execution.

Synthetic User Monitoring is providing users even more options for analysis and reporting and offers the ability to export the execution details to external tools.

In the following screenshots you find a few examples of how you can use Grafana to analyze various metrics in the Analytic Dashboard:

By utilizing Grafana, you can extract valuable insights and gain a better understanding of the data. This will enable you to monitor and track execution details, allowing you to create more comprehensive graphs and cross-use case analytics.

Let's start with a definition of terms:

Selenium IDE scripts

A Selenium IDE (SIDE) script is a set of instructions, or commands, that dictate the actions to be performed on a web browser. This script is created using Selenium IDE, which is an open source tool that can be installed as a plugin in a browser. Selenium IDE provides convenient features such as recording and replaying functions, which allow users to easily capture and repeat a series of actions performed on a browser. Additionally, Selenium IDE includes an editor that enables users to create and modify Selenium IDE scripts. To enhance the functionality of Selenium IDE scripts, users have the ability to annotate them. These annotations can be used to define specific steps, create variables, set secure content, and more. By leveraging these annotations, users can introduce Synthetic User Monitoring functionalities, which can help simulate real user behavior and monitor the performance of web applications

Resources

Selenium IDE scripts are represented as Resources when uploaded in Synthetic User Monitoring. One resource may contain several scripts. Synthetic User Monitoring will interpret those Selenium IDE scripts and users can create Scenarios out of it.

Scenarios

A Scenario is a combination of a resource script (content) and runners (runtime infrastructures). Synthetic User Monitoring measures the availability and performance of a scenario (global or at step level). Performance is evaluated using dynamic thresholds defined either at scenario or step levels.

Runners

A Runner can be any infrastructure/component compatible with Selenium WebDriver (W3C standard). You need to be aware that SAP Cloud ALM does not provide this service.

Every runner has a runner endpoint, which is the selenium remote webdriver infrastructure that Synthetic User Monitoring accesses to pass the scenario commands to execute. It also hosts the web browser used to access the monitored applications. To access the endpoint, Synthetic User Monitoring uses the connection information defined in the runner configuration. Two types of configuration are possible:

Next, you will see the Deployment and Connection options for the runner based on the specific use case:

• On-premise: The endpoint is accessed via the Cloud Connector information specified in the configuration.
• Cloud: The endpoint is access via the HTTPS URL specified in the configuration.

To get a Selenium remote webdriver infrastructure, you have several possibilities (you can find more information about Selenium remote webdriver on the following Selenium Page).

The figure illustrates the usage of a Remote WebDriver Infrastructure:

Therefore, in general the following possibilities for remote WebDriver Infrastructure exist:

Deployed from custom-assembled elements: Selenium compatible Server, Web Browser, WebDriver

• You are in charge of the hosting.
• Web Browsers and WebDrivers are executables provided by the browser vendors.
• Ready to use server exists (Selenium standalone server).
• Servers also provide support for parallel executions (also known as Grid).

By deploying an "all-in-one" package (for example a docker image)

• You are in charge of the hosting (i.e. you decide where to deploy the package: cloud, on-premise).
• Some packages also support multiple browsers and parallel executions.

Via a 3rd party service provider.

• This often includes the hosting of the infrastructure.
• Generally provides:
  • Support for different browsers (Chrome, Edge, …)
  • Parallel executions
  • Multiple locations

The choice of which runner deployment you should use, depends on multiple factors like your monitored landscape, infrastructure or subscriptions you may already have in your company environment. In the table, you see some examples of preferred options for runners. The choice of deployment depends on where the monitored application users location, the runner location, and if a customer owned solution or a third-party provider is used, for example if your monitored application is an on-premise application accessed from a private network, you would prefer a local on-premise Selenium Docker or Grid.