You can spin up a cluster with Argo CD in 15 minutes. But if you deploy a Kubernetes workload that vanishes at 2 AM, or a rogue dev spins up an unbounded PVC until the bill explodes, or a security audit turns up privileged: true on every namespace, your environment is not production — it’s chaos in a config file. This checklist evolved from real incidents at fintech and SaaS platforms. Lint your cluster against it before shipping your MVP to prod.
Immediate (Dev & Pr → Staging)
1 · Resource Requests ≥ Limits per container
Set CPU and memory requests equal to or slightly below the complementary limit to avoid "requests ceiling" throttling. Validate with CPU Throttling dashboards. How →
Set CPU and memory requests equal to or slightly below the complementary limit to avoid "requests ceiling" throttling. Validate with CPU Throttling dashboards. How →
2 · PodDisruptionBudget for every critical deployment
With a minAvailable ≥ 1 across 2+ AZ; test by draining a node to confirm eviction decisions.
With a minAvailable ≥ 1 across 2+ AZ; test by draining a node to confirm eviction decisions.
3 · RBAC per namespace; ClusterRole sweeping disabled
Each team gets a Role + RoleBinding tight to their namespace. Non-namespace cosmos use ClusterRole tied to labels.
Resources excluded: nodes, persistentvolumes, namespaces.
Each team gets a Role + RoleBinding tight to their namespace. Non-namespace cosmos use ClusterRole tied to labels.
Resources excluded: nodes, persistentvolumes, namespaces.
4 · NetworkPolicy deny-by-default; allow explicit egress to known CIDRs
Start with
Start with
0.0.0.0/0 egress allowed; tighten in staging UAT; then move to Cloud Network Firewall (CNF) trie at infra level.
5 · PodSecurity admission enforced on restricted profile
Pod Security Admission with enforce=restricted. Warn on baseline + non-root FSGroup. Map checks: no hostPath, no hostNetwork, readOnlyRootFilesystem=true.
Pod Security Admission with enforce=restricted. Warn on baseline + non-root FSGroup. Map checks: no hostPath, no hostNetwork, readOnlyRootFilesystem=true.
6 · Automated CSI snapshot/replication for stateful workloads
Validate PVC dynamic provisioner + StorageClass mapping. Verify restore from snapshot in dev. Coverage: EBS, Azure Disk, GCE PD.
Validate PVC dynamic provisioner + StorageClass mapping. Verify restore from snapshot in dev. Coverage: EBS, Azure Disk, GCE PD.
7 · Liveness/Readiness + startupProbe defined; starts after 5 min peak
Use average 95th percentile pre-prod latency as probe initialDelaySeconds ± 3 seconds per deployment stage.
Use average 95th percentile pre-prod latency as probe initialDelaySeconds ± 3 seconds per deployment stage.
8 · HPA + ClusterAutoscaler with binpacking scheduler
HPA min 1 → max 8 replicas; ClusterAutoscaler scale up delay 10s; Scale down stabilization window 300s; PodDisruptionBudget minAvailable=ceil(max(HPA-min,1)). Scheduler config: select
HPA min 1 → max 8 replicas; ClusterAutoscaler scale up delay 10s; Scale down stabilization window 300s; PodDisruptionBudget minAvailable=ceil(max(HPA-min,1)). Scheduler config: select
node.kubernetes.io/instance-type memory > 16GiB.
Production (Go-Live)
9 · NetworkPolicy cluster-wide enforced; allow prometheus-operator & cilium-agent pods
Audit mode > deny all; enforcement mode in staging; production blocklists: Scapy, Metasploit, Tor exit nodes, and common mining traffic ASNs.
Tips for cilium: use
Audit mode > deny all; enforcement mode in staging; production blocklists: Scapy, Metasploit, Tor exit nodes, and common mining traffic ASNs.
Tips for cilium: use
k8s:io.kubernetes.pod.namespace in egress rules to only whitelist namespaces that legitimately need internet.
10 · Cert-manager issuing production wildcard *.example.com
Issuer:
cluster-issuer -> certificate with solvers: http01 for DNS-01 prod staging gapfill; dns01 route53 recordsets; test issuance then roll.
11 · Prometheus-operator alert rules mirror the 11 golden signals
Set p99 latency in 5min windows, error budget burn rate > 1/12 in 5m triggers P1; same rules replicated into Opsgenie with auto-escalate.
12 · Resource quota + limit ranges scoped per namespace
limits.cpu, limits.memory, requests.cpu, requests.memory; quotas: CPU 64 cores, memory 256 GiB, pods 512; enforcement via ResourceQuota + LimitRange; OPA policy to veto speculative resource lines.
13 · Node drain safety harness: ASCS + Pod eviction
Before any infra update/maintenance, schedule drain verification:
kubectl drain --dry-run=server --selector=role=build, topspin rollback script in chat.
14 · Fire drill every sprint: chaotic chaos-engineering induced termination
Run one
Run one
kubectl delete or --force --grace-period=0 on a named deployment each sprint to ensure PDB + kube-controller-manager jointly do their jobs.
Monitoring & Runbooks
15 · Runbook per deployment + prometheus-operator alertsExample runbook: customer-facing rate-limit breach → flip circuit-breaker flag to true → roll back to last green image → notify Slack.
16 · Kubectl context frozen via kube-shell-envrc
Pin context to prod-default, cannot
Pin context to prod-default, cannot
kubectl apply -f in dev unless use kube-shell-envrc --context=prod-default.
17 · CRD + admission policies instantiated through kyverno-json or kube-ovn
All constraints come from Git takes over Argo CD syncs; no
All constraints come from Git takes over Argo CD syncs; no
kubectl apply config drift; examples: require label team, block top: pods access.
Security & Governance
18 · Named service accounts + least privilege RBAC App can use service accountsa-<app-name> with RoleBinding tight to the namespace only.
19 · Pod anti-affinity default: spread across zones
podAntiAffinity.preferredDuringSchedulingIgnoredDuringExecution: topologyKey=topology.kubernetes.io/zone
20 · Image provenance via Cosign + Sigstore
Attestations pushed to registry on build; Kyverno verifies signature in admission controller; reject unsigned.
Attestations pushed to registry on build; Kyverno verifies signature in admission controller; reject unsigned.
Audit (Every quarter)
Automate this with kubectl get --raw /metrics | kube-score | kubescape in a CronJob and post results to a compliance dashboard.
21 · kube-bench passed; non-root user everywhere
Gate deployments from
Gate deployments from
.securityContext: runAsNonRoot=true runAsUser=1000 fsGroup=2000; kube-bench run weekly + diff.
Resource Heat Map
22 · Cluster-wide resource heat map dashboardUse bin-packing metrics: resource_debt = ceiling((sum(item.requests) - node.allocatable)/node.capacity); daily dashboards; P1 alert when >30% for 1h.
23 · Ingress TLS 1.2+ enforced; redirect HTTP -> HTTPS
ingress.spec.tls.hosts[].secretName valid; nginx.ingress.kubernetes.io/ssl-redirect: "true"; modern cipher suites enabled (TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256).
24 · Stateful workloads: PVC with daily snapshots
VolumeSnapshotClass standard mapped to StorageClass; snapshot every 6h; retain 14 days; restore into -clone namespace for validation.
25 · Node OS weekly update rollout; drain one AZ at a time
CAN/UAT sets → prod staging → prod A → prod B → prod C; one AZ at a time; monitor NodeCon; rollback via Azure/GCP snapshot if degradation >1%. Roll out rollback in 5 min cycle per AZ max latency.
CAN/UAT sets → prod staging → prod A → prod B → prod C; one AZ at a time; monitor NodeCon; rollback via Azure/GCP snapshot if degradation >1%. Roll out rollback in 5 min cycle per AZ max latency.
Operational Excellence (One-pager per service)
26 · SLA dashboard: overall, per-service, per-AZInherits from ServiceLevelObjective
availability=0.9995; alert SLO burn SLO < 0.95; Ticket ≥ 15 min total incident duration triggers P1.
27 · Platform regression bucket: 3 staging environments; Git → Build
S1: PR -> preview (Pull request environment) S2: main -> dev (CI environment) S3: promotion to stage (performance regression gate) S4: promotion to prod (staging root certs).
Gate on every stage: kube-score, trivy, kubescape, Argo CD health.
S1: PR -> preview (Pull request environment) S2: main -> dev (CI environment) S3: promotion to stage (performance regression gate) S4: promotion to prod (staging root certs).
Gate on every stage: kube-score, trivy, kubescape, Argo CD health.
28 · Pod template hash mis-sets caught with imagePullPolicy=Always
Use semantic-versioning image tags (
Use semantic-versioning image tags (
v2.1.3) with imagePullPolicy=Always so Argo CD always syncs fresh hash even when the YML hasn’t changed.
29 · Service mesh (Istio/Cilium): telemetry + zero-downtime rolling updates
mTLS strict; rate-limiting 1000 RPS per upstream host; Cilium Hubble for per-service latency; zero-downtime upgrade gate: 10% traffic → 50% → 100%.
mTLS strict; rate-limiting 1000 RPS per upstream host; Cilium Hubble for per-service latency; zero-downtime upgrade gate: 10% traffic → 50% → 100%.
30 · cost-monitor daemonset + cluster cost APIs turned on
kyverno enforces cost cap quotas; Kubecost or VMware Velero cost-exporter for CSI / Pod cost attribution.
Cordon/punt pods exceeding €50/month projected cost within next 7 days.
kyverno enforces cost cap quotas; Kubecost or VMware Velero cost-exporter for CSI / Pod cost attribution.
Cordon/punt pods exceeding €50/month projected cost within next 7 days.
Your Turn: The 80-minute turborepo
Pick one environment (dev → staging OR staging → prod).
1) Onboard a new namespace: namespace: demo-app
2) Run the linter: kube-score score ./k8s
3) Fix the top-5 red items from the dashboard
4) Apply one NetworkPolicy with namespaceSelector to demo-app
5) Rerun kube-score -> green
6) Ship with Argo CD.
Checkout the companion repo github.com/tayoca/k8s-production-checklist-sample with everything baked into a GitHub Action that fails your PR on kube-score reds and applies sample NetworkPolicies.
TL;DR – Use the List, Fix the Cluster
Print this checklist, schedule a 50-minute brown-bag with your platform team, and walk through each item. If you hit a red item, you just found your production up grade work bench.