Describing Observability in Kyma

After completing this lesson, you will be able to:

After completing this lesson, you will be able to:

  • Describe briefly observability core concepts
  • Illustrate the basic concepts of observability in Kyma
  • Identify the different components of observability in Kyma
  • Describe the strategic shift of the observability architecture in Kyma

Introduction to Observability

You are now operating some workloads on the SAP BTP, Kyma runtime and want to gain more insights into your running applications. For this purpose, you want to use the observability features of Kyma. In this lesson, you will learn about the observability features of Kyma and how to use them.

What is Observability?

Observability means the ability of a system to provide information about its internal state. This information can be used to monitor the system's behavior and diagnose problems. Observability is a key concept in microservice-based applications, where the number of components is large and the interactions between them are complex. Applications can expose insights about their internal state using logs, metrics and traces. Collectively, these insights are called "telemetry".

Observability in Kyma

Project "Kyma" comes with several open-source tools to collect, expose and analyze in-cluster telemetry data.

Data Collection

For data collection, Kyma uses the open-source project Prometheus. Prometheus is a time-series database that collects metrics from instrumented applications and stores them in a time-series database. Prometheus is a pull-based system, which means that applications need to expose an endpoint from which Prometheus can scrape the metrics. Many programming languages provide a Prometheus client library that can be used to expose metrics in the required format. Prometheus can also instrument applications without a prometheus-scrable endpoint by using Prometheus exporters.

For log collection, Kyma uses Fluent Bit. Fluent Bit is a super fast and scalable logging and metric processor. It collects logs from your workloads and forwards them to destinations, such as Prometheus.

The open-source project "Kyma" also comes with a tracing solution based on Jaeger. Jaeger is a distributed tracing system that helps to monitor and troubleshoot transactions between distributed services. As mentioned at the beginning of the course, tracing is not part of the SAP BTP, Kyma runtime.

Data Analysis and Visualization

For data analysis and visualization, Kyma uses the open-source project Grafana. Grafana is a visualization and analysis tool to visualize and query time-series data. With Grafana, you can create dashboards that show the current state of your applications. And Grafana can easily visualize data from Prometheus and other sources such as Loki.

The Different Components of Observability in Kyma

Relationship Between the Different Components

Prometheus is centrally running in the kyma-sytem namespace. From there, it scrapes metrics from different targets, such as services. Prometheus stores them in a time-series database. This database can be queried with Prometheus' query language PromQL. The Prometheus Web UI or Grafana can be used to visualize the data. Both are running in the kyma-system namespace as well.

Accessing both GUIs is not directly possible from the public internet. To access them you can access them by port-forwarding the services to your local machine.

For the Prometheus Web UI, you can use the following command:

Code snippet
kubectl port-forward -n kyma-system svc/monitoring-prometheus 9090:9090
Copy code

This will forward the service monitoring-prometheus in the namespace kyma-system to your local machine on port 9090. You can then access the Prometheus Web UI at http://localhost:9090.

For Grafana, you can use the following command:

Code snippet
kubectl port-forward -n kyma-system svc/monitoring-grafana 3000:80
Copy code

This will forward the service monitoring-grafana in the namespace kyma-system to your local machine on port 3000. You can then access Grafana at http://localhost:3000.

Configuring Prometheus

Prometheus doesn't scrape metrics from all services by default. Instead, you have to configure it to scrape metrics from the services that you want to monitor. This is done by creating a ServiceMonitor CRD. The ServiceMonitor resource tells Prometheus which services to scrape and how to scrape them. The following example shows a ServiceMonitor resource that tells Prometheus to scrape metrics from the hello-kyma service.

Code snippet
kind: ServiceMonitor
  namespace: kyma-system # place the ServiceMonitor in the kyam-system namespace
  name: monitoring-hello-kyma
    app: hello-kyma
      app: hello-kyma
    - port: http
      any: true
Copy code

In this example, Prometheus will scrape metrics from services labeled with app: hello-kyma. The ServiceMonitor resource tells Prometheus to scrape metrics from the service on port http, which is defined in the service's manifest. By default, Prometheus scrapes at the /metrics endpoint. Learn more about the ServiceMonitor resource here.

In SAP BTP, Kyma runtime and thus also in project "Kyma", many ServiceMonitor resources are already created. You can find them in the kyma-system namespace.

Code snippet
kubectl get servicemonitors -n kyma-system
Copy code

The Strategic Shift of the Observability Architecture in Kyma

A Note on the Strategic Shift of Observability in Kyma

Based on experience and input from the community over the last years, SAP has noticed that the current out-of-the-box in-cluster observability stack in Kyma doesn't fully qualify for enterprise-grade qualities. Evolutionary changes in the observability stack are planned to be implemented in the near future. The new stack will leverage the OpenTelemetry standards, allowing a more flexible, open, and vendor-neutral approach to observability. As a result, you can then use the observability system that best fits your needs. This new architecture will not replace the existing stack but will complement it.

The new stack will introduce a new layer of collectors. One collector for each signal/telemetry type (logs, metrics, traces). These collectors will be responsible for collecting the telemetry data from the workloads and forwarding them through defined pipelines to the backends. The back ends will be responsible for storing and analyzing the telemetry data. The back ends will be pluggable, so that you can use the back end that best fits your needs. The configuration for each collector will be handled by a new telemetry operator, which can be configured via Kubernetes API and native tooling (for example, kubectl). The following diagram shows the new architecture:

To get more information about the new architecture, you can read this blog post about the upcoming architecture and the Observability Strategy Paper.


In this lesson, you have learned about the different components of the observability stack in Kyma. You have also discovered how to access the different components and how to configure Prometheus to scrape metrics from your services. You also learned about the upcoming strategic shift of the observability stack in Kyma.

In the next exercise, you will learn how to define custom metrics and monitor them with built-in tools. To monitor out-of-the-box metrics, you can look at this blog post.

Further Reading

Log in to track your progress & complete quizzes