Kubernetes Secrets are used to store sensitive information such as passwords, API keys, and tokens. Unlike ConfigMaps, Secrets are specifically designed to handle confidential data and are stored more securely by default.

Creating and Managing Secrets

To create a Secret, you can use the kubectl create secret command. For example:

kubectl create secret generic my-secret --from-literal=password=myPassword

You can also create a Secret from a YAML file:

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
type: Opaque
data:
  password: bXlQYXNzd29yZA==

To apply it, run:

kubectl apply -f my-secret.yaml

To use the Secret in a Pod, you can mount it as a volume or inject it as an environment variable.

Encrypting Secrets at Rest and in Transit

  • At Rest: By default, Secrets are stored in base64 encoding, which is not encrypted. Enable Encryption at Rest by configuring the Kubernetes API server with an encryption configuration file.

  • In Transit: Always make sure to enable TLS for API Server to etcd communication. This ensures that Secrets are encrypted while traveling over the network.

Rotating Secrets and Best Practices

  • Rotating Secrets: Periodically update Secrets and reflect the changes in your Pods. Use the following command to update the Pods using the Secrets:
kubectl rollout restart deployment/my-deployment
  • Best Practices:
    • Limit access using RBAC.
    • Never store Secrets in source code repositories.
    • Use namespace segregation.
    • Leverage Kubernetes audit logs to monitor access to Secrets.

Third-Party Tools for Enhanced Secret Security

  • Sealed Secrets: Encrypts Secrets so they are safe to store, even to a public repository.

  • Vault by HashiCorp: An external tool that integrates well with Kubernetes for storing, accessing, and managing Secrets.

  • AWS KMS, Azure Key Vault, Google Cloud KMS: Cloud-specific secret management services.

Troubleshooting Exercise

  1. Setup Commands:
kubectl create namespace secret-test
kubectl config set-context --current --namespace=secret-test
kubectl create secret generic test-secret --from-literal=password=oldPassword

  1. Exercise: You changed the Secret, but the Pod using the Secret is not picking up the new value.

  2. Debugging:

  • Check if the Secret was updated:
kubectl get secret test-secret -o jsonpath='{.data.password}' | base64 --decode
  • Check if the Pod is using the updated Secret:
kubectl describe pod my-pod
  1. Solution:

If the Pod is not picking up the new value, you will need to trigger a rollout to update the Pod:

kubectl rollout restart deployment/my-deployment