Collecting Application Logs and Shipping them to SAP Cloud Logging

Objective

After completing this lesson, you will be able to collect and ship logs

Collecting Application Logs in Kyma and Shipping to SAP Cloud Logging

Usage Scenario

After deploying your microservice-based application to Kyma, you want to monitor it continuously with Kyma's telemetry capabilities and the integration to SAP Cloud Logging.

Task Flow

In this exercise, you will perform the following tasks:

  1. Deploy a sample application that produces application logs.
  2. Provision an SAP Cloud Logging service instance and create a service binding with credential rotation.
  3. Configure a Kyma Telemetry LogPipeline resource that references the binding.
  4. Add deep linking to Kyma dashboard.
  5. Use SAP Cloud Logging to explore logs.

Prerequisites

Before you begin, ensure that the following prerequisites are met:

Task 1: Deploy a Sample Application that Produces Application Logs

Steps

    In this task, you will create a new hello-kyma Deployment and Service in a new namespace called telemetry-exercise. The deployment uses the image ghcr.io/sap-samples/kyma-runtime-learning-journey/hello-kyma:1.0.0, which writes application logs for incoming requests to stdout in JSON format.

  1. Create a new namespace calledtelemetry-exercise:

    Code Snippet
    1
    kubectl create namespace telemetry-exercise

  2. In the telemetry-exercise namespace, create a new Deployment called hello-kyma:

    Code Snippet
    1
    kubectl apply -n telemetry-exercise -f https://raw.githubusercontent.com/SAP-samples/kyma-runtime-learning-journey/main/unit_9/hello-kyma-deployment-svc.yaml

  3. Verify if the Deployment is created:

    Code Snippet
    1
    kubectl get deployments -n telemetry-exercise

    If successful, you get the following output:

    Code Snippet
    12
    NAME         READY   UP-TO-DATE   AVAILABLE   AGE
hello-kyma   1/1     1            1           2m

  4. Port-forward the `hello-kyma` Service to your local machine:

    Code Snippet
    1
    kubectl port-forward -n telemetry-exercise svc/hello-kyma 8080:8080

    Leave this terminal window open, and open http://localhost:8080 in a new browser tab. You get the message Hello Kyma! (Version 1.0.0).

  5. Stop the port-forwarding by pressing Ctrl+C in the terminal window.

  6. Verify that logs are written for every request to stdout:

    Code Snippet
    1
    kubectl logs -n telemetry-exercise -l app=hello-kyma

    The logs retrieved with kubectl are stored temporarily on the Node file system. These logs are rotated frequently and are lost when the Pod or the Node is rescheduled. The following setup collects the logs from the Node file system, stores them long-term in SAP Cloud Logging, and makes them searchable across Pods and namespaces.

Task 2: Provision an SAP Cloud Logging Instance and Create a Binding with Credential Rotation

Steps

  1. Using the SAP BTP Operator module (which is in your Kyma cluster by default), define a new instance for SAP Cloud Logging:

    Code Snippet
    12345678910111213
    cat <<EOF | kubectl -n telemetry-exercise apply -f -
apiVersion: services.cloud.sap.com/v1
kind: ServiceInstance
metadata:
    name: cloud-logging
spec:
    serviceOfferingName: cloud-logging
    servicePlanName: standard
    externalName: Cloud Logging
    parameters:
      ingest_otlp: # must be enabled for trace and metric data ingestion
          enabled: true
EOF

  2. For the instance you just created, define a new Service Binding that has credential rotation enabled:

    Code Snippet
    1234567891011121314
    cat <<EOF | kubectl -n telemetry-exercise apply -f -
apiVersion: services.cloud.sap.com/v1
kind: ServiceBinding
metadata:
    name: cloud-logging
spec:
    serviceInstanceName: cloud-logging
    externalName: cloud-logging
    secretName: cloud-logging
    credentialsRotationPolicy:
      enabled: true
      rotationFrequency: 720h
      rotatedBindingTTL: 24h
EOF

    The credential rotation rotates the credentials in the generated Secret periodically, keeping the old credentials valid in parallel for the specified TTL. Whenever the content of the Secret changes, the LogPipeline resource defined in the following step reloads the Secret value dynamically. This assures that the credentials do not expire and are rotated frequently.

  3. To verify the setup, check the state of the service instance and binding:

    Code Snippet
    1
    kubectl -n telemetry-exercise get serviceinstance
    Code Snippet
    1
    kubectl -n telemetry-exercise get servicebinding

    It takes several minutes till the instance is provisioned and the binding and the related Kubernetes Secret are created. However, you can continue with the next steps already because the LogPipeline picks up the Secret value dynamically as soon as it is ready.

Task 3: Configure a Kyma Telemetry LogPipeline Referencing the Service Binding

Steps

  1. Create a LogPipeline:

    Code Snippet
    123456789101112131415161718192021222324252627282930
    cat <<EOF | kubectl apply -f -
apiVersion: telemetry.kyma-project.io/v1alpha1
kind: LogPipeline
metadata:
  name: cloud-logging
spec:
  output:
      http:
        dedot: true
        host:
            valueFrom:
              secretKeyRef:
                  name: cloud-logging
                  namespace: telemetry-exercise
                  key: ingest-mtls-endpoint
        tls:
            cert:
              valueFrom:
                  secretKeyRef:
                    name: cloud-logging
                    namespace: telemetry-exercise
                    key: ingest-mtls-cert
            key:
              valueFrom:
                  secretKeyRef:
                    name: cloud-logging
                    namespace: telemetry-exercise
                    key: ingest-mtls-key
        uri: /customindex/kyma
EOF

    Note

    This example uses a LogPipeline, but the Telemetry module is not limited to logs. You can integrate traces and metrics in a similar way.

Task 4: Use SAP Cloud Logging to Explore Logs

Steps

    Setting up SAP Cloud Logging also resulted in the creation of a Secret. This Secret contains details for pushing data to SAP Cloud Logging and for accessing the SAP Cloud Logging Dashboard.

    Note

    This example simplifies the setup by using Basic Authentication with a shared secret for the access to the SAP Cloud Logging Dashboard. However, when you're using Kyma in live environments, it's recommended to use single-sign-on (SSO), for example with a SAML configuration. If you use SSO, no credentials for the SAP Cloud Logging Dashboard access are provided with the generated Secret.

  1. To see your credentials, inspect the Secret:

    Code Snippet
    1
    kubectl -n telemetry-exercise get secret cloud-logging -o go-template='{{range $key, $value := .data}}{{$key}}={{$value|base64decode}}{{"\n"}}{{end}}'

    The attribute dashboards-endpoint contains the URL to the SAP Cloud Logging Dashboard. The attributes dashboards-username and dashboards-password contain the credentials for authentication.

  2. Open the URL taken from dashboards-endpoint and authenticate yourself with the credentials.

  3. In the Discover section, inspect the logs-json-kyma-* indexes. All logs are instantly available for query with the used log attributes.

Task 5: Add Deep Linking to Kyma Dashboard

Steps

  1. To add navigation and deep links to Kyma dashboard, follow Use SAP Cloud Logging Dashboards.

  2. To try out the new links, go to Kyma dashboard, choose a namespace, and select Discover Logs in SAP Cloud Logging.

Result

Bravo! You have completed this exercise. You have used the Telemetry module to collect and visualize the logs of your extension with SAP Cloud Logging.

Log in to track your progress & complete quizzes

Learning

SAP FacebookSAP YoutubeSAP LinkedIn