Awesome
A Docker image with Kubernetes manifests for investigation and troubleshooting your cluster.
Purpose
The DOKS team provides this image for use as-is and for transparency as the image used when a request to "deploy a debug pod" is made to our customers, which may occur when deeper investigation is needed with direct access to a cluster.
Usage
kubectl apply -f k8s/daemonset.yaml
This DaemonSet manifest will:
- Ensure a pod with our Docker image is running indefinitely on every node.
- Use
hostPID
,hostIPC
, andhostNetwork
. - Mount the entire host filesystem to
/host
in the containers. - Mount the
containerd
socket at/run/containerd/containerd.sock
from the host into the container.
In order to make use of these workloads, you can exec into a pod of choice by name:
kubectl -n kube-system exec -it my-pod-name bash
If you know the specific node name that you're interested in, you can exec into the debug pod on that node with:
NODE_NAME="my-node-name"
POD_NAME=$(kubectl -n kube-system get pods --field-selector spec.nodeName=${NODE_NAME} -ojsonpath='{.items[0].metadata.name}')
kubectl -n kube-system exec -it ${POD_NAME} bash
Once you're in, you have access to the set of tools listed in the Dockerfile
. This includes:
vim
- is a greatly improved version of the good old UNIX editor Vi.screen
- is a full-screen window manager that multiplexes a physical terminal between several processes, typically interactive shells.curl
- is a command-line tool for transferring data specified with URL syntax.jq
- is a lightweight and flexible command-line JSON processor.dnsutils
- includes various client programs related to DNS that are derived from the BIND source tree, specificallydig
,nslookup
, andnsupdate
.iputils-ping
- includes theping
tool that sends ICMPECHO_REQUEST
packets to a host in order to test if the host is reachable via the network.tcpdump
- a powerful command-line packet analyzer; and libpcap, a portable C/C++ library for network traffic capture.traceroute
- tracks the route packets taken from an IP network on their way to a given host.net-tools
- includes the important tools for controlling the network subsystem of the Linux kernel, specificallyarp
,ifconfig
, andnetstat
.netcat
- is a multi-tool for interacting with TCP and UDP; it can open TCP connections, send UDP packets, listen on arbitrary TCP and UDP ports, do port scanning, and deal with both IPv4 and IPv6.iproute2
- is a collection of utilities for controlling TCP / IP networking and traffic control in Linux.strace
- is a diagnostic, debugging and instructional userspace utility with a traditional command-line interface for Linux. It is used to monitor and tamper with interactions between processes and the Linux kernel, which include system calls, signal deliveries, and changes of process state.dstat
- is a versatile replacement for vmstat, iostat, netstat and ifstat. Dstat overcomes some of their limitations and adds some extra features, more counters and flexibility. Dstat is handy for monitoring systems during performance tuning tests, benchmarks or troubleshooting.htop
- is interactive process viewer for Unix systems.atop
- is an advanced interactive monitor for Linux-systems to view the load on system-level and process-level.wget
- for retrieving files using HTTP, HTTPS, FTP and FTPS.crictl
- A CLI for CRI endpoints. Configured to use/run/containerd/containerd.sock
as a default endpoint.
Tips and Tricks
chroot + systemctl
chroot /host /bin/bash
systemctl status kubelet
journalctl -xe
journalctl -u kubelet
Contributing
At DigitalOcean we value and love our community! If you have any issues or would like to contribute, feel free to open an issue or PR and cc any of the maintainers.