Kubernetes troubleshooting labs that build real operator judgment

Why most Kubernetes labs do not create troubleshooting skill

Many tutorials optimize for completion. They provide the command, the expected output, and the correction in one continuous sequence. That is useful when learning object syntax, but it removes the uncertainty that defines real operations work. In production, the ticket rarely says “the Service selector is wrong.” It says “the application is unavailable,” “the rollout is stuck,” or “the Pods are Running but customers still receive errors.”

A production-style lab must therefore separate symptom from cause. The learner should receive a user-visible symptom, partial telemetry, and a time boundary. The failure should remain small enough to diagnose, but realistic enough that several hypotheses are initially plausible.

The anatomy of a high-value troubleshooting lab

A reusable lab cockpit

Start broad enough to establish scope, then move toward the object most likely to explain the symptom. The exact commands vary, but the reasoning sequence should remain stable.

kubectl config current-context
kubectl get events -A --sort-by=.lastTimestamp
kubectl get pods -A -o wide
kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --previous
kubectl get service,endpointslice -n <namespace>
kubectl get deployment,replicaset -n <namespace>

The command list is not a ritual. Each command must answer a question. Events answer what the control plane recently rejected or retried. Describe connects an object to conditions and events. Logs explain application behavior. EndpointSlices prove whether a Service currently has ready backends. Controller objects reveal desired state, rollout history, and ownership.

Five lab families to rotate through

1. Workload lifecycle

Examples include CrashLoopBackOff, an incorrect command, missing Secret keys, OOMKilled, and a readiness probe that never succeeds. Require learners to distinguish container creation, process startup, liveness, readiness, and application correctness.

2. Service and routing

Use selector mismatches, wrong target ports, missing EndpointSlices, DNS configuration problems, and Gateway API backend-reference errors. The learner should trace the path from client to DNS to Service to EndpointSlice to Pod.

3. Scheduling and capacity

Use impossible node selectors, untolerated taints, insufficient resources, topology constraints, and PodDisruptionBudgets. The learner must read scheduler events rather than changing constraints blindly.

4. Security and admission

Use RBAC denials, Pod Security Admission rejections, and over-restrictive NetworkPolicies. The repair should preserve least privilege rather than weakening the cluster until the error disappears.

5. Control-plane and recovery

On a disposable self-managed cluster, stop the scheduler or controller-manager, validate an etcd snapshot, and restore the component. These labs teach the difference between API persistence, scheduling, and controller reconciliation.

How to grade a troubleshooting lab fairly

Do not grade only against one expected command sequence. Two engineers may take different valid routes. Grade the quality of evidence, the safety of the intervention, the strength of recovery proof, and the clarity of the root-cause explanation.

Make the lab portfolio-worthy

Store the known-good manifests, the failure injection, the evidence note, and the repaired state in Git. Include a short incident record that explains symptom, scope, evidence, root cause, intervention, validation, and prevention. This demonstrates far more professional skill than a screenshot of kubectl get pods.