Discovering Observability and Monitoring in SAP BTP, Cloud Foundry Runtime

As your company's flagship application gains traction and user traffic spikes, performance issues and unexpected errors are beginning to surface. To maintain a seamless user experience and ensure system reliability, you need to implement robust observability and monitoring strategies. In this lesson, you'll discover how to utilize tools within the SAP BTP, Cloud Foundry runtime to gain deep insights into your application's performance and rapidly address any issues.

Observability and monitoring are your windows into the inner workings of your software and systems. Observability enables you to understand what's happening inside your applications and infrastructure by analyzing the data they produce. Meanwhile, monitoring leverages this observability data to actively check and alert on the system's status, ensuring that performance remains optimal and issues are quickly addressed.

SAP BTP, Cloud Foundry runtime provides following observability indicators:

Logs in Cloud Foundry originate from two main sources: your applications and the Cloud Foundry system components.

• Application logs provide insights into the operations and issues within your applications. They can be defined by developers in terms of content and format. See App logging in Cloud Foundry for more details.
• System logs offer a glimpse into how the platform manages and interacts with your applications. They're generated by Cloud Foundry’s internal components, such as Cloud Controller.

The logging architecture of SAP BTP, Cloud Foundry runtime leverages Firehose architecture. At its core, Loggregator collects and combines logs into a standardized format, then pushes them into the Firehose data stream.

You can access real-time logs using the cf log command in the Cloud Foundry CLI. SAP BTP, Cloud Foundry runtime also provides a built-in Log Cache for short-term storage, facilitating real-time troubleshooting and analysis.

For long-term data retention and in-depth analysis, forwarding logs to external systems becomes necessary. You can configure Syslog Drains to selectively route specific types of logs to a syslog server for centralized storage. You can set up nozzles, customizable plugins that integrate with the Firehose, to channel the entire Firehose stream to an external log management system, such as Splunk or Elasticsearch.

Once your logs are in an external storage system, you can employ different visualization tools to create dynamic and insightful dashboards. For example, Kibana is often employed with Elasticsearch to explore logs interactively.

We can separate Metrics in Cloud Foundry categories: container metrics and custom metrics, each serving distinct monitoring needs:

• Container metrics reveal how your applications utilize the resources allocated to their containers, including CPU usage, memory consumption and disk space. Diego Cell collects these metrics approximately every 15 seconds and forwards them to Loggregator for aggregation. By default, container metrics are stored temporarily in Log Cache for quick retrieval, and accessible using Cloud Foundry CLI commands, such as cf app <app-name>. For long-term storage and historical analysis, metrics are usually forwarded to external systems like Prometheus through nozzles. Learn more at Container metrics.
• Custom metrics are user-defined, which can measure transaction rates, error counts, user engagement metrics, or any other data relevant to your application's performance and business goals. This often requires instrumentation within your code, either by language-specific libraries (for example, expvar for Golang), third-party monitoring systems (for example, Prometheus), or SAP's Java library. Most metrics libraries allow metrics to be exposed via an HTTP endpoint. Alternatively, you can run an agent within the application’s container. These methods handle metrics collection independently from the application logic, offering flexibility in gathering and exporting custom metrics with minimal changes to the code.

Both types of metrics are usually stored directly in monitoring tools like Prometheus, enabling detailed analysis, alerts, and integration with other systems. For visualization, Grafana is a popular choice in the Cloud Foundry community.

In Cloud Foundry, audit events provide insights into resource interactions by documenting actions such as application deployments, service bindings, user management, and so on. Unlike security events, which focus on access and authorization, or usage events, which are geared towards billing, audit events offer a more granular view into the activities that impact resources, recording details like the actor (the entity initiating the action), the target (the affected resource), and a timestamp.

Audit events are generated by applications and Cloud Foundry components, then captured by Loggregator and streamed through Firehose to various endpoints for access and analysis.

Watch the following video to discover how to monitor events for individual applications or across entire spaces.

Remember: as SAP BTP retains Cloud Foundry events for 14 days, you should forward these data to an external storage solution for long-term persistence and advanced analytics.

To better understand audit events, check out the following documentation: Audit events overview

Health checks are automated tests that Cloud Foundry performs on your running application instances to determine their status, ensuring reliability and availability. Cloud Foundry employs liveness and readiness check, serving distinct functions:

• Readiness checks ensure that an application instance is prepared to handle requests. If a readiness check fails, the instance is temporarily removed from the load balancer, preventing it from receiving traffic until it stabilizes. This is essential during updates or scaling operations to ensure graceful deployments.
• Liveness checks verify that an application instance is active and running. If a liveness check fails, the instance is automatically restarted. This selfrecovery mechanism helps mitigate system failures efficiently.

There are three types of health checks:

• Port checks (default): Check if a TCP connection can be established on the application's port.
• Process checks: Verify if the application's main process is running. It's useful for worker processes or applications that don't use standard ports.
• HTTP checks (recommended): Perform a HTTP GET request to a specified endpoint, and check for a 200 status code. It provides more accurate feedback for web applications.

The life cycle of health checks can be summarized as the followings:

• Deployment: Health check configurations are defined in the application's manifest file (manifest.yml) or through the Cloud Foundry CLI during deployment. For details, please check Configuring health checks.
• Initialization: Diego initiates and schedules the application instance along with the specified health checks.
• Startup: Upon startup, health checks begin immediately every two seconds. This continues until either the application passes the checks or a specified timeout is reached (default: 60 seconds, configurable up to 10 minutes on SAP BTP as instructed in the Deploying large applications guideline.
• Ongoing monitoring: Once the application starts successfully, regular liveness and readiness checks are performed to continuously monitor its health.
• Failure handling: Failing a liveness check leads to a restart of the app instance. Failing a readiness check results in the instance being removed from routing, though it continues running for potential recovery. When health checks fail, analyze your application logs for errors or warnings. If you're using a custom HTTP endpoint, ensure it's returning the correct response. Additionally, consider resource constraints (memory, CPU) and network connectivity as potential culprits. Eventually, if an instance repeatedly fails health checks, Cloud Foundry will give up on restarting it and mark it as crashed.

To learn more about health checks, refer to the following documentation: Health checks in Cloud Foundry

SAP empowers you to effortlessly establish comprehensive observability and monitoring strategies for applications running on SAP BTP, Cloud Foundry runtime. This is achieved through a suite of robust services, including SAP Cloud Logging, SAP Alert Notification, and SAP Cloud Application Lifecycle Management (SAP Cloud ALM).

This lesson has equipped you with the knowledge to set up and manage observability and monitoring within the SAP BTP, Cloud Foundry runtime. By understanding and applying these tools, you can ensure that your applications are reliable, performant, and secure.

Mission: Implement observability in a full-stack CAP Application