Known-good state
Record readiness, endpoint, rollout, and user-path evidence before injecting anything. Without a baseline, every later observation is ambiguous.
Kubernetes operations practice
Kubernetes troubleshooting labs that build real operator judgment
A good lab does not ask you to copy a manifest and celebrate a green Pod. It gives you a known-good system, introduces one controlled failure, withholds the answer long enough for you to reason, and requires objective recovery proof.
The core loop: establish known good → predict → inject one fault → collect evidence → narrow the failure surface → make the smallest safe repair → prove recovery → explain the root cause.
Why most Kubernetes labs do not create troubleshooting skill
Many tutorials optimize for completion. They provide the command, the expected output, and the correction in one continuous sequence. That is useful when learning object syntax, but it removes the uncertainty that defines real operations work. In production, the ticket rarely says “the Service selector is wrong.” It says “the application is unavailable,” “the rollout is stuck,” or “the Pods are Running but customers still receive errors.”
A production-style lab must therefore separate symptom from cause. The learner should receive a user-visible symptom, partial telemetry, and a time boundary. The failure should remain small enough to diagnose, but realistic enough that several hypotheses are initially plausible.
The anatomy of a high-value troubleshooting lab
One primary fault
Introduce one root cause and at most one contributing factor. Randomly breaking five objects teaches frustration, not diagnosis.
Progressive hints
Hints should move from scope to object to field. They should never reveal the repair before the learner has gathered evidence.
Recovery proof
“The command succeeded” is not proof. Validate controller state, Pod readiness, EndpointSlices, route conditions, and the user path.
A reusable lab cockpit
Start broad enough to establish scope, then move toward the object most likely to explain the symptom. The exact commands vary, but the reasoning sequence should remain stable.
kubectl config current-context
kubectl get events -A --sort-by=.lastTimestamp
kubectl get pods -A -o wide
kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --previous
kubectl get service,endpointslice -n <namespace>
kubectl get deployment,replicaset -n <namespace>The command list is not a ritual. Each command must answer a question. Events answer what the control plane recently rejected or retried. Describe connects an object to conditions and events. Logs explain application behavior. EndpointSlices prove whether a Service currently has ready backends. Controller objects reveal desired state, rollout history, and ownership.
Five lab families to rotate through
1. Workload lifecycle
Examples include CrashLoopBackOff, an incorrect command, missing Secret keys, OOMKilled, and a readiness probe that never succeeds. Require learners to distinguish container creation, process startup, liveness, readiness, and application correctness.
2. Service and routing
Use selector mismatches, wrong target ports, missing EndpointSlices, DNS configuration problems, and Gateway API backend-reference errors. The learner should trace the path from client to DNS to Service to EndpointSlice to Pod.
3. Scheduling and capacity
Use impossible node selectors, untolerated taints, insufficient resources, topology constraints, and PodDisruptionBudgets. The learner must read scheduler events rather than changing constraints blindly.
4. Security and admission
Use RBAC denials, Pod Security Admission rejections, and over-restrictive NetworkPolicies. The repair should preserve least privilege rather than weakening the cluster until the error disappears.
5. Control-plane and recovery
On a disposable self-managed cluster, stop the scheduler or controller-manager, validate an etcd snapshot, and restore the component. These labs teach the difference between API persistence, scheduling, and controller reconciliation.
How to grade a troubleshooting lab fairly
Do not grade only against one expected command sequence. Two engineers may take different valid routes. Grade the quality of evidence, the safety of the intervention, the strength of recovery proof, and the clarity of the root-cause explanation.
Strong submission: “The Pods were Running but not Ready. The readiness probe returned 404 because the path was `/ready` while the application exposed `/readyz`. I confirmed the Service had zero ready endpoints, corrected only the probe path, watched the rollout complete, and verified EndpointSlices and an in-cluster request.”
Make the lab portfolio-worthy
Store the known-good manifests, the failure injection, the evidence note, and the repaired state in Git. Include a short incident record that explains symptom, scope, evidence, root cause, intervention, validation, and prevention. This demonstrates far more professional skill than a screenshot of `kubectl get pods`.
Practice this instead of only reading it
Kubernetes Operator's Workbook, Second Edition turns these ideas into guided labs, 24 production incidents, control-plane recovery, and interview simulations.
Explore the workbook