Basics of Container Logging in Kubernetes

In Kubernetes, logs help you understand what’s happening inside your application and the underlying infrastructure. Logs for a specific container are usually accessible through the standard output (stdout) and standard error (stderr) streams.

Accessing and Reading Logs

You can access logs using the kubectl logs command. Here are some useful variations:

# For current logs
kubectl logs <pod-name> -c <container-name>

# For previous terminated container logs
kubectl logs -p <pod-name> -c <container-name>

# To stream logs in real-time
kubectl logs -f <pod-name> -c <container-name>

Centralized Logging Solutions: Fluentd, ELK stack

  • Fluentd: An open-source data collector that integrates with Kubernetes to collect logs and forward them to various backends like Elasticsearch.
  • ELK stack: Stands for Elasticsearch, Logstash, and Kibana. Elasticsearch stores logs, Logstash processes them, and Kibana provides a web-based interface for querying the logs.

Troubleshooting Common Issues using Logs

  • Application Errors: Look for exceptions or error messages in your application logs to diagnose issues.
  • Resource Limitations: Check for out-of-memory errors or CPU limitations that could be affecting the container.
  • Connectivity Issues: Logs can help you identify if a container is unable to reach a specific service or endpoint.

Best Practices for Log Management

  • Structured Logging: Use JSON or another structured format for easier parsing and querying.
  • Log Rotation and Retention: Configure log rotation and set appropriate retention policies.
  • Sensitive Data: Never log sensitive information like passwords or API keys.
  • Log Levels: Use different log levels (INFO, WARN, ERROR, etc.) for easier filtering.

Exercise: Troubleshooting Using Logs

Setup

First, create a simple Pod with a container that writes some text to stdout and stderr.

kubectl run log-generator --image=busybox -- /bin/sh -c "echo 'Hello, stdout'; echo 'Hello, stderr' 1>&2; sleep 3600"

Steps for Troubleshooting

  1. Check the running Pods to ensure your Pod is running.

    kubectl get pods

  2. View the logs for the log-generator pod.

    kubectl logs log-generator

  3. If you find an issue, for instance, you’re not seeing the text “Hello, stdout”, troubleshoot by describing the Pod to check for events and status.

    kubectl describe pod log-generator

Solution

If everything is set up correctly, you should see “Hello, stdout” and “Hello, stderr” in the logs. If not, the kubectl describe command will help you uncover issues such as resource limitations, ImagePullBackOff, etc. Solve these issues accordingly.