Monitoring and Observability in Kubernetes

Kubernetes has become the go-to solution for managing containerized applications, making it easier for teams to scale and manage their workloads. But as powerful as Kubernetes is, it also introduces complexity, especially when it comes to keeping an eye on everything that’s going on. That’s where monitoring and observability come into play. In this blog, we’ll break down these concepts, explain why they matter, and introduce you to the tools that can help you stay on top of your Kubernetes environment.

What Do We Mean by Monitoring?

Think of monitoring as keeping an eye on the health of your system. It’s about tracking key metrics—like how much CPU your application is using or how much memory it’s consuming—to make sure everything is running smoothly. Monitoring is like the dashboard in your car; it tells you if something needs attention, like if your engine is overheating or if you’re running low on gas.

And What About Observability?

Observability takes things a step further. While monitoring tells you what is happening, observability helps you figure out why it’s happening. It’s like having a detailed map of your car’s internals so that when something goes wrong, you can diagnose the problem more effectively. Observability involves looking at logs, metrics, and traces to get a complete picture of what’s going on inside your system.

Why Should You Care About Monitoring and Observability in Kubernetes?

Kubernetes is great because it lets you break down your application into smaller pieces, or microservices, that can run across many servers. But this also means that when something goes wrong, it can be tricky to figure out where the problem is. Monitoring and observability help you:

1. Catch Problems Early: Spot issues before they affect your users.
2. Understand What Went Wrong: Get to the root of the problem so you can fix it and prevent it from happening again.
3. Keep Things Running Smoothly: Make sure your applications are performing well and scaling as needed.

The Building Blocks of Monitoring and Observability in Kubernetes

1. Metrics

Metrics are like the vital signs of your system—they give you a snapshot of how things are going at any given moment. In Kubernetes, this might include how much CPU or memory a particular pod is using. Tools like Prometheus are great for collecting these metrics. With Prometheus, you can gather data from your applications and Kubernetes itself, and then use Grafana to create dashboards that visualize this data in a way that’s easy to understand.

2. Logs

Logs are the story of what’s happening in your system, told in a series of events. They can help you understand what was going on at a specific point in time. In Kubernetes, logs can come from different places—like your application containers or the Kubernetes components themselves. The Elasticsearch, Fluentd, and Kibana (EFK) stack is a popular setup for gathering, processing, and visualizing these logs.

3. Traces

Traces follow a request as it travels through your system, helping you see how long each part of the journey takes. This is especially useful in microservices architectures where a single request might pass through several different services. Tools like Jaeger and Zipkin are commonly used in Kubernetes to help you trace these requests and identify any slowdowns or bottlenecks.

Practical Example: Spring Boot with Actuator in Kubernetes

To make this more concrete, let’s look at how you can use Spring Boot with Actuator, combined with Kubernetes liveness and readiness probes, to achieve effective monitoring and observability.

Step 1: Add Actuator to Your Spring Boot Application

Spring Boot Actuator provides a set of built-in endpoints that give you metrics and insights into your application. To get started, add the following dependency to your pom.xml:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

This will enable various endpoints such as /actuator/health, /actuator/metrics, and more.

Step 2: Expose Actuator Endpoints

In your application.properties or application.yml, you can configure which endpoints are exposed:

properties

management.endpoints.web.exposure.include=health,info,metrics
management.endpoint.health.show-details=always

This setup ensures that health, info, and metrics endpoints are available, which you can later use in your Kubernetes probes.

Step 3: Set Up Liveness and Readiness Probes in Kubernetes

Kubernetes liveness and readiness probes help Kubernetes understand when your application is healthy and ready to serve traffic.

• Liveness Probe: This tells Kubernetes if your application is running. If the liveness probe fails, Kubernetes will restart the pod.
• Readiness Probe: This checks if your application is ready to handle requests. If the readiness probe fails, the pod will not receive traffic. Here’s how you can configure these probes using the Actuator’s /health endpoint in your Kubernetes deployment YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-spring-boot-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-spring-boot-app
  template:
    metadata:
      labels:
        app: my-spring-boot-app
    spec:
      containers:
        - name: my-spring-boot-container
          image: my-spring-boot-app:latest
          ports:
            - containerPort: 8080
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            initialDelaySeconds: 60
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 10

Step 4: Monitor with Prometheus and Grafana

You can configure Prometheus to scrape the metrics exposed by Spring Boot Actuator. First, ensure that Prometheus is set up to discover your Spring Boot application’s pods. Here’s an example configuration:

scrape_configs:
  - job_name: 'spring-boot'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [ __meta_kubernetes_pod_label_app ]
        action: keep
        regex: my-spring-boot-app
    metrics_path: /actuator/prometheus
    scheme: http

This setup tells Prometheus to scrape metrics from the /actuator/prometheus endpoint on all pods labeled with app: my-spring-boot-app.

Once Prometheus is collecting these metrics, you can use Grafana to create dashboards that provide insights into your application’s performance and health.

Tools to Help You Monitor and Observe Your Kubernetes Environment

Prometheus What It Does: Collects and stores metrics. Why It’s Useful: Prometheus integrates seamlessly with Kubernetes, making it easy to track key performance indicators across your environment.

Grafana What It Does: Visualizes metrics data.

Why It’s Useful: With Grafana, you can create custom dashboards that let you see exactly what you need to monitor at a glance.

EFK Stack (Elasticsearch, Fluentd, Kibana) What It Does: Aggregates, processes, and visualizes logs.

Why It’s Useful: The EFK stack helps you make sense of your logs, making it easier to search and analyze them when you need to troubleshoot an issue.

Jaeger/Zipkin What They Do: Track the flow of requests through your system.

Why They’re Useful: These tools help you understand the performance of your services and identify where improvements are needed.

Kubernetes-Native Tools What They Do: Provide insights specific to Kubernetes.

Why They’re Useful: Tools like Kube-State-Metrics and cAdvisor give you detailed metrics that are tailored to Kubernetes, helping you monitor things like resource usage and the health of your pods.

Best Practices for Keeping an Eye on Your Kubernetes Cluster

Instrument Everything:: Make sure all parts of your system—applications, infrastructure, and the Kubernetes control plane—are set up for monitoring and observability.

Use Labels and Annotations: Kubernetes labels and annotations are handy for organizing and filtering your data. They make it easier to find exactly what you’re looking for.

Centralize Your Data: Bringing all your metrics, logs, and traces together in one place makes it easier to see the big picture and spot correlations between different data points.

Set Up Alerts: Don’t just monitor—set up alerts so that you’re notified as soon as something goes wrong. This way, you can respond quickly and minimize the impact.

Keep Improving: Your monitoring and observability setup should evolve as your system grows. Regularly review and tweak your approach to ensure it continues to meet your needs.

Wrapping Up

Monitoring and observability are essential for running a smooth and reliable Kubernetes environment. By focusing on these aspects and incorporating practical tools like Spring Boot Actuator with Kubernetes probes, you can catch issues early, understand what’s happening in your system, and keep everything running efficiently. Whether you’re just starting with Kubernetes or looking to refine your setup, investing in the right tools and practices will pay off in the long run.