Kubernetes incident response practice: from alert to recovery proof

Begin with impact, not YAML

Before editing an object, answer four questions: What is the user-visible symptom? Which service, namespace, region, or tenant is affected? When did the behavior begin? What changed near that time? This prevents a common failure mode: repairing an object that looks unhealthy but is not responsible for the incident.

The Kubernetes incident response sequence

1. Orient. Confirm cluster and namespace. Identify whether the problem is workload, platform, dependency, or access related.
2. Stabilize impact. Pause a bad rollout, remove a failing backend, shift traffic, or scale known-good capacity when evidence supports it.
3. Compare desired and actual state. Read conditions, events, ReplicaSets, EndpointSlices, route status, and node state.
4. Collect decisive evidence. Use the smallest set of commands that can falsify the leading hypothesis.
5. Choose rollback or forward-fix. Prefer the option with lower uncertainty and clearer recovery proof.
6. Verify recovery. Validate the user path and the platform state separately.
7. Preserve learning. Record root cause, contributing factors, detection gap, and prevention.

Rollback is a risk decision, not a reflex

Rollback is appropriate when the previous revision is known good, available, and compatible with current data and dependencies. Forward-fix is appropriate when there is no reliable previous state, the change is data-dependent, or reverting would create another failure. The operator should state why one path has lower operational risk.

kubectl rollout history deployment/portfolio -n portfolio
kubectl rollout status deployment/portfolio -n portfolio
kubectl get rs -n portfolio --sort-by=.metadata.creationTimestamp
kubectl rollout undo deployment/portfolio -n portfolio --to-revision=<n>

Do not stop at “rollout complete.” Confirm ready replicas, EndpointSlices, route conditions, error rate, latency, and a representative user request.

Evidence by failure surface

Practice communication as part of the lab

A technically correct repair can still be operationally poor when stakeholders receive no useful update. Use a simple format that separates facts from hypotheses and states the next decision point. Avoid declaring resolution until monitoring and user-path evidence remain healthy for a defined observation period.

Run incident drills with explicit constraints

A useful drill specifies a severity, time limit, evidence boundary, and change policy. Constraints prevent learners from escaping the diagnosis by destroying the environment and force them to preserve the failure long enough to understand it.

After recovery: convert the incident into a control

Every incident should produce at least one improvement: a validation rule, a safer rollout policy, an alert, a runbook change, a test, or a documentation correction. A missing Secret key might produce admission validation or CI schema checks. A Service selector mismatch might produce policy-as-code or manifest unit tests. An etcd recovery drill might produce a snapshot-validation job and a restore rehearsal calendar.

Practice the 90-second explanation

State the symptom, scope, evidence, root cause, intervention, and proof. Do not narrate every command. Interviewers and incident commanders want to hear how you reduced uncertainty and managed risk.