Home

Logging And Monitoring

  • get errors from the command line
  • get pod logs
  • get the status of pods
  • troubleshoot networking bugs
  • validate resource allocation

Logging For Docker Containers

# docker logs -f container-id
docker logs -f api-box

Logging For Pods in K8s

# kubectl logs -f pod-name
kubectl logs -f api-pod

Logging Containers in Multi-Container Pods

# kubectl logs -f pod-name container-name
kubectl logs -f api-pod api-emailing-sidecar-box

System Logs

systemd matters here - with systemd, logs go to journalct, viewable through the cli tool jourcnalctl -a.
Without systemd, logs go to /var/log/<agent-name>.log.
Things that log:

  • kube-scheduler
  • kube-proxy
  • kubelet
  • Docker

Fluentd for cluster-wide details

fluentd can help aggregate logs across a cluster:

  • fluentd agents run on each node via a DaemonSetTroubleshooting
  • they agg logs
  • they feed them to an elasticsearch instane
  • can be viz'd in a kibana dashboard!

Monitoring

Metric are collected with the kubelet:

  • kubelet is the agent on each node - it writes logs to the local fs via the docker logging driver
  • gets directions from master server
  • contains cAdvisor, getting performance metrics from pods
  • Get this up & running with
  • kubectl logs GETS the logs that the kubelet wrote

Several options are out there:

  • Metrics Server
    • an in-memory option && cannot see historical performance data
  • Prometheus
  • Elastic Stack
  • DataDog
  • Dynatrace

Monitoring Resource Consumption

  • cpu
  • network

Monitoring Objects And Object Health

  • number of nodes
  • health of nodes
  • pod count
  • pod cpu & memory usage

Other CNFC monitoring and tracing tools

The Cloud Native Computing Foundation has a few open-source tools for monitoring: Prometheus, Fluentd, OpenTracint, and Jaeger.
A bunch of these tools have batteries-included examples to spin up with docker, an express api, etc.

Prometheus

Prometheus

  • focuses on alerting and metrics
  • provides a db that is time-series oriented, query-able
  • cluster-wide alerts
  • integrates with grafana
  • grafana

Fluentd

Fluentd:

  • an open-source data collector
  • unifies data collection & consumption
  • JSON formatted logs
  • plugin-architecture: plugins galore
  • low hardware reqs:
    • "30-40MB of memory and can process 13,000 events/second/core" (from docs)

OpenTracing

NOTE: it looks like the openTracing has "moved" to OpenTelemetry.

OpenTracing

  • "distributed tracing"
  • maybe optimal for microservice archutecture
  • an API spec, has frameworks and libs that implement the spec
  • here is an example of the nodejs example running a simple express app

Jaeger

Jaeger. Developed by uber! Uber's blogpost on their evolution of distributed tracing.
Jaeger is used to "Monitor and troubleshoot transactions in complex distributed systems".

  • transaction monitoring
  • RCA
  • service dependency analysis
  • perf/latency optimization

Kubectl Debug

k8s docs:

  • can "debug" an object (i.e. pod)
  • can create an ephemeral container to an already running process!
  • access a nodes FS by creating a pod in a nodes host namespace
kubectl debug -h
# was kubectl alpha debug

# debug the master node
#   create a "busybox" image to run on the node
kubectl debug node master -it --image=busybox

# in the new container, view processes on the host node
ps
# in the container, print logs of a different pod
kubectl logs the-other-pod

References for Logging, Debugging and Troubleshooting

Things to be able to do

Tags:
Kubernetes
k8s
pods
containers
logs