Describing Observability in Kyma

For data collection, Kyma uses the open-source project Prometheus. Prometheus is a time-series database that collects metrics from instrumented applications and stores them in a time-series database. Prometheus is a pull-based system, which means that applications need to expose an endpoint from which Prometheus can scrape the metrics. Many programming languages provide a Prometheus client library that can be used to expose metrics in the required format. Prometheus can also instrument applications without a prometheus-scrable endpoint by using Prometheus exporters.

For log collection, Kyma uses Fluent Bit. Fluent Bit is a super fast and scalable logging and metric processor. It collects logs from your workloads and forwards them to destinations, such as Prometheus.

The open-source project "Kyma" also comes with a tracing solution based on Jaeger. Jaeger is a distributed tracing system that helps to monitor and troubleshoot transactions between distributed services. As mentioned at the beginning of the course, tracing is not part of the SAP BTP, Kyma runtime.

For data analysis and visualization, Kyma uses the open-source project Grafana. Grafana is a visualization and analysis tool to visualize and query time-series data. With Grafana, you can create dashboards that show the current state of your applications. And Grafana can easily visualize data from Prometheus and other sources such as Loki.

Prometheus is centrally running in the kyma-sytem namespace. From there, it scrapes metrics from different targets, such as services. Prometheus stores them in a time-series database. This database can be queried with Prometheus' query language PromQL. The Prometheus Web UI or Grafana can be used to visualize the data. Both are running in the kyma-system namespace as well.

Accessing both GUIs is not directly possible from the public internet. To access them you can access them by port-forwarding the services to your local machine.

For the Prometheus Web UI, you can use the following command:

1
kubectl port-forward -n kyma-system svc/monitoring-prometheus 9090:9090

This will forward the service monitoring-prometheus in the namespace kyma-system to your local machine on port 9090. You can then access the Prometheus Web UI at http://localhost:9090.

For Grafana, you can use the following command:

1
kubectl port-forward -n kyma-system svc/monitoring-grafana 3000:80

This will forward the service monitoring-grafana in the namespace kyma-system to your local machine on port 3000. You can then access Grafana at http://localhost:3000.

Prometheus doesn't scrape metrics from all services by default. Instead, you have to configure it to scrape metrics from the services that you want to monitor. This is done by creating a ServiceMonitor CRD. The ServiceMonitor resource tells Prometheus which services to scrape and how to scrape them. The following example shows a ServiceMonitor resource that tells Prometheus to scrape metrics from the hello-kyma service.

123456789101112131415
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  namespace: kyma-system # place the ServiceMonitor in the kyam-system namespace
  name: monitoring-hello-kyma
  labels:
    app: hello-kyma
spec:
  selector:
    matchLabels:
      app: hello-kyma
  endpoints:
    - port: http
  namespaceSelector:
      any: true

In this example, Prometheus will scrape metrics from services labeled with app: hello-kyma. The ServiceMonitor resource tells Prometheus to scrape metrics from the service on port http, which is defined in the service's manifest. By default, Prometheus scrapes at the /metrics endpoint. Learn more about the ServiceMonitor resource here.

In SAP BTP, Kyma runtime and thus also in project "Kyma", many ServiceMonitor resources are already created. You can find them in the kyma-system namespace.

1
kubectl get servicemonitors -n kyma-system

Based on experience and input from the community over the last years, SAP has noticed that the current out-of-the-box in-cluster observability stack in Kyma doesn't fully qualify for enterprise-grade qualities. Evolutionary changes in the observability stack are planned to be implemented in the near future. The new stack will leverage the OpenTelemetry standards, allowing a more flexible, open, and vendor-neutral approach to observability. As a result, you can then use the observability system that best fits your needs. This new architecture will not replace the existing stack but will complement it.

The new stack will introduce a new layer of collectors. One collector for each signal/telemetry type (logs, metrics, traces). These collectors will be responsible for collecting the telemetry data from the workloads and forwarding them through defined pipelines to the backends. The back ends will be responsible for storing and analyzing the telemetry data. The back ends will be pluggable, so that you can use the back end that best fits your needs. The configuration for each collector will be handled by a new telemetry operator, which can be configured via Kubernetes API and native tooling (for example, kubectl). The following diagram shows the new architecture:

To get more information about the new architecture, you can read this blog post about the upcoming architecture and the Observability Strategy Paper.

In this lesson, you have learned about the different components of the observability stack in Kyma. You have also discovered how to access the different components and how to configure Prometheus to scrape metrics from your services. You also learned about the upcoming strategic shift of the observability stack in Kyma.

In the next exercise, you will learn how to define custom metrics and monitor them with built-in tools. To monitor out-of-the-box metrics, you can look at this blog post.