Troubleshooting seems like a regular rhytm in K8s development - pods, nodes, service connectivity, ingress config, deployment "style" config, etc.
An Approach
get pod info
kkg pods
find a pod of specific interest
say apipod
.
- get details about the pod
kkdesc pod apipod
notices some specifics of pod description
Conditions
- best case: review that the pod is
Initialized
,Ready
ndPodScheduled
- best case: review that the pod is
Events
- best case: review that the pod events look something like
pulling <your-image>
Successfully pulled image <your-image>
Created container
Started container
- best case: review that the pod events look something like
look into container logs
kk logs apipod <api-container-name>
look into container-to-network connectivity
- best case: the container can access the internet
- shell into a container && nslookup a url
kubectl exec -it apipod -c <container-name> -- sh
nslookup www.google.com
- one permissioning issue could be that the container uid does not have permissions to access specific web resources like an html doc
look into service-to-pod connectivity
- an example: assuming a service is connecting to a pod and making the pod/container available through port publishing
- review available & running services
kkg svc
- best case: a service that you know is up & running, say
api-svc
- get config info on that service of interest
kkg svc api-svc -o yaml
- review that an ENDPOINT! is available for the svc
kkg ep
- best case: the svc port and name are connected to an available endpoint- get config deets of the endpoint
kkg ep apiapp -o yaml
- if the above are all working as expected (containers, pods, services, endpoints) check out more deets
- Check the status of the kube-proxy:
ps | grep kube-proxy
(maybe add flags-elf
)- to stdout, in a list
- or perhaps
journalctl -a | grep proxy
- check out the logs of the kube-proxy
kubectl -n kube-system logs kube-proxy-randoCharsHere
- Check that the expected "rules" are being created for a servuce
sudo iptables-save | grep <a-service-here>
- Check the status of the kube-proxy: