Debugging a Kubernetes Pod from within it

Introduction

As a rule, images with applications should not have anything installed beyond the tools and libraries necessary for their operation. This affects the image size as well as security considerations (fewer installed elements in the "system" mean fewer potential vulnerabilities).

However, there are situations where it is necessary to conduct analysis, verification, etc., within a pod. An example task is to verify traffic using the tcpdump tool, which is not installed by default in the images. What to do in such a case?

The best and fastest way to utilize missing tools in an application image is to launch a new pod with an additional container containing an image with the necessary tools for analysis. An example image is jrecord/nettools with ready-to-use network tools (such as curl, telnet, tcpdump).

Instructions

Below is an example of using tcpdump by launching an additional container within the Pod.

1. Apply the following YAML manifest to the cluster:

   apiVersion: v1 kind: Deployment metadata: name: nginx-deployment namespace: example spec: replicas: 1 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: # Main application container - name: nginx image: nginx:latest ports: - containerPort: 80 # Additional container with tcpdump - name: tcpdump-container image: jrecord/nettools command: ["sleep", "infinity"]

2. Add a Service to be able to connect to your pod inside the cluster:

   apiVersion: v1 kind: Service metadata: name: nginx-service namespace: example spec: selector: app: nginx ports: - protocol: TCP port: 80 targetPort: 80

3. Run a console in the tcpdump-container within the Pod with the application.

4. Execute the command tcpdump -i any port 80 -n.

5. (Optional) If needed, launch an additional pod with curl installed (in new terminal):

   kubectl run -n example nettools -it --image jrecord/nettools --rm

6. Send a request to the created Service (port 80).

   curl -I -L nginx-service.example.svc.cluster.local:80

   Response:

   HTTP/1.1 200 OK Server: nginx/1.25.3 Date: Mon, 08 Jan 2024 07:47:28 GMT Content-Type: text/html Content-Length: 615 Last-Modified: Tue, 24 Oct 2023 13:46:47 GMT Connection: keep-alive ETag: "6537cac7-267" Accept-Ranges: bytes

7. Check what the running tcpdump displays, it should capture request:

   # tcpdump -i any port 80 -n tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes 07:47:28.671309 IP 10.136.2.189.49612 > 10.136.2.208.http: Flags [S], seq 1465380728, win 62727, options [mss 8961,sackOK,TS val 2760271970 ecr 0,nop,wscale 7], length 0 07:47:28.671320 IP 10.136.2.208.http > 10.136.2.189.49612: Flags [S.], seq 2355749117, ack 1465380729, win 62643, options [mss 8961,sackOK,TS val 3411559377 ecr 2760271970,nop,wscale 7], length 0 07:47:28.671330 IP 10.136.2.189.49612 > 10.136.2.208.http: Flags [.], ack 1, win 491, options [nop,nop,TS val 2760271970 ecr 3411559377], length 0 07:47:28.671356 IP 10.136.2.189.49612 > 10.136.2.208.http: Flags [P.], seq 1:110, ack 1, win 491, options [nop,nop,TS val 2760271970 ecr 3411559377], length 109: HTTP: HEAD / HTTP/1.1 07:47:28.671358 IP 10.136.2.208.http > 10.136.2.189.49612: Flags [.], ack 110, win 489, options [nop,nop,TS val 3411559377 ecr 2760271970], length 0 07:47:28.671464 IP 10.136.2.208.http > 10.136.2.189.49612: Flags [P.], seq 1:239, ack 110, win 489, options [nop,nop,TS val 3411559377 ecr 2760271970], length 238: HTTP: HTTP/1.1 200 OK 07:47:28.671488 IP 10.136.2.189.49612 > 10.136.2.208.http: Flags [.], ack 239, win 490, options [nop,nop,TS val 2760271970 ecr 3411559377], length 0 07:47:28.671587 IP 10.136.2.189.49612 > 10.136.2.208.http: Flags [F.], seq 110, ack 239, win 490, options [nop,nop,TS val 2760271970 ecr 3411559377], length 0 07:47:28.671615 IP 10.136.2.208.http > 10.136.2.189.49612: Flags [F.], seq 239, ack 111, win 489, options [nop,nop,TS val 3411559377 ecr 2760271970], length 0 07:47:28.671646 IP 10.136.2.189.49612 > 10.136.2.208.http: Flags [.], ack 240, win 490, options [nop,nop,TS val 2760271971 ecr 3411559377], length 0

8. Clean up.