GitOps beyond hello world

Git is not the pipeline. Git is the declared desired state and review history; the pipeline exists to produce and promote trustworthy changes into that state.

1. Validate Helm before reconciliation

A large chart can change in ways that are syntactically valid but operationally surprising. Render the chart in CI with the same values sources the deployment will use. Validate schema, diff the rendered output, and make material changes reviewable before the GitOps controller sees them.

helm lint ./chart
helm template app ./chart -f values/staging.yaml > rendered.yaml
kubectl apply --dry-run=server -f rendered.yaml

Where server-side validation is not available in CI, use schema validation and policy tests against the rendered output. Avoid treating a simple `helm template` exit code as complete production safety.

Keep secret material outside chart values

Reference externally managed Kubernetes Secrets rather than committing plaintext secrets to `values.yaml`. External Secrets Operator, SOPS, Vault integration, or another approved secret-delivery mechanism can work; the important boundary is that the Git repository and rendered manifest review do not expose material that should remain secret.

2. Gate image provenance separately from Git approval

A merged manifest can still reference the wrong artifact. Prefer immutable digests for production references where practical, sign build outputs in CI, generate SBOM/provenance attestations when required, and verify those artifacts through admission policy.

cosign sign --yes registry.example.com/app@sha256:<digest>
cosign attest --yes --predicate sbom.spdx.json \
  --type spdxjson registry.example.com/app@sha256:<digest>

Keyless signing with workload identity can reduce long-lived signing-key management, but the identity, issuer, repository, workflow, and trust policy still need explicit verification. Do not use a permissive wildcard policy simply because the signature cryptographically verifies.

Admission belongs at the cluster boundary

Kyverno, Gatekeeper plus verification integrations, or another admission layer can block unsigned or otherwise non-compliant artifacts regardless of whether the request came from Argo CD or a manual API call. That makes the safety control independent of one deployment path.

3. Promote declarations; let Argo CD reconcile

Use environment-specific directories, values, or overlays with clear ownership. A build pipeline can open or update a promotion change that advances an immutable image reference. Review and merge that declaration, then let Argo CD reconcile the target cluster.

services/
  payments-api/
    dev/
    staging/
    prod/

ApplicationSets are useful when the set of applications follows a stable directory or cluster pattern. Keep the generator comprehensible; an ApplicationSet that operators cannot mentally expand during an incident is an operational liability.

4. Treat auto-sync as a policy decision

Automated sync, pruning, and self-healing are separate controls. Enable them only when the resource lifecycle and recovery behavior are understood. Production systems often need explicit protection for namespaces, stateful resources, CRDs, or shared infrastructure even when ordinary workloads can reconcile automatically.

• Prune: understand what deletion will cascade to before enabling it broadly.
• Self-heal: useful for enforcing desired state, but document the incident-response override process.
• Sync windows: useful where changes must obey operational or business windows.
• Health checks: define what “healthy” means for custom resources instead of relying only on sync completion.

5. Recovery is part of the GitOps design

A Git history is useful only when you know which state was actually running and how to safely return to a prior declaration. Record the promoted artifact digest, environment revision, migration compatibility, and any external state changes that make rollback unsafe. Practice recovery before you need it.

6. Keep production claims evidence-safe

Patterns in this article are implementation approaches, not a claim that every control is appropriate for every cluster or that Tayoca has delivered a specific quantified outcome with them. Production adoption should be validated against the target environment, Kubernetes and controller versions, security policy, and organizational change process.