GitOps Beyond Hello World

Most GitOps tutorials stop at argocd app create against a single-app repo. That demo runs clean because nothing is fighting you yet — no chart with 600 lines of values, no security team demanding signed images, no team shipping to prod while another ships hotfixes into staging at 2 AM. This post is the field manual for what comes next.

We have hardened GitOps for fintech and SaaS platforms running dozens of services across multiple clusters. The patterns below are configurations we use to reason about production GitOps controls. Skip to the one that hurts:

The fundamental error teams make is treating GitOps like CI/CD with a Git remote. Git is not the pipeline — it is the declarative source of truth. The pipeline exists to reach Git correctly.

1 · Surviving Helm Upgrade Corruption

helm upgrade --install is the most dangerous command in most platform stacks. The corruption modes we see in incident reviews fall into three buckets:

• State drift between releases. Someone edits a Kubernetes resource with kubectl (a manual HPA bump, a securityContext tweak) and the next helm upgrade either silently overwrites it or fails with a terrifying three-way patch error.
• Map merging clobbering team overrides. A chart's default values.yaml defines resources.requests.cpu: 100m, the team sets resources.limits.memory: 512Mi in their override, and Helm's recursive merge swallows requests entirely. The pod now runs with no CPU request.
• Secrets in values.yaml. The chart wants secrets.apiKey, so the team commits it encrypted with helm secrets — except half the team is on a stale age key and the decrypt step fails silently in CI.

The configuration we ship

Three settings, always together, in the Argo CD Application:

# gitops/apps/<svc>/argocd-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payments-api
  finalizers: [ resources-finalizer.argocd.argoproj.io ]
spec:
  syncPolicy:
    syncOptions:
      - CreateNamespace=false      # namespace pre-provisioned, with RBAC
      - ServerSideApply=true          # 3-way merge, no ownership flip-flops
      - PrunePropagationPolicy=foreground
    managedNamespaceMetadata:
      labels: { pod-security.kubernetes.io/enforce: restricted }
  source:
    chart: payments-api
    helm:
      valueFiles: [ ../../values/staging.yaml, values.yaml ]
      ignoreMissingValueFiles: true
      skipSchemaValidation: false     # JSON schema catches map-merge regressions
      passCredentials: false            # no secrets via helm registry

ServerSideApply=true turns Helm's three-way patch into Kubernetes SSA — ownership is tracked per-field, so a manual kubectl scale during an incident is no longer silently undone on the next sync. Combined with PrunePropagationPolicy=foreground, deletion cascades on the first sync, not the second.

Externally validating the chart before Argo CD touches it

The most reliable corruption check is one Argo CD never runs. We add a CI step that renders the chart and diffs it against the last-known-good release:

# .github/workflows/chart-validate.yml (excerpt)
- name: Render & diff against last-good release
  run: |
    helm template payments-api ./chart -f values/staging.yaml \
      > rendered-new.yaml
    helm pull oci://registry/chart:$(yq .appVersion Chart.yaml) \
      --version $(yq .version Chart.yaml) \
      -f values/staging.yaml --untar --untardir /tmp/good \
      && helm template payments-api /tmp/good -f values/staging.yaml \
      > rendered-good.yaml
    diff -u rendered-good.yaml rendered-new.yaml > chart-drift.diff \
      || (echo "::warning::Chart render drift detected" && exit 1)

If this step fails, the PR does not merge and Argo CD never observes a change. We catch the resources.requests.cpu vanishing here, in CI, before it lands in staging.

2 · Image Validation Gates Before Sync

A Git repo can be merged, but the image it references may not be admissible. The naive GitOps setup syncs any image tag the moment Git changes. That is how a developer's locally-pushed api:feature-branch-x ends up in production — the tag was reachable from the chart, so Argo CD applied it.

The fix is two checkpoints: provenance (was this image built by us, from this commit?) and admission (does this cluster accept this provenance?). Provenance lives in the registry as signatures; admission lives in the cluster as policy.

Signing the image in CI

# CI pipeline, after docker build/push
cosign sign --yes \
  --identity-token "$OIDC_TOKEN" \
  --certificate-identity "$REGISTRY/$REPO@$DIGEST" \
  --certificate-oidc-issuer "$OIDC_ISSUER" \
  "$REGISTRY/$REPO@$DIGEST"

# Bundle the SBOM and an in-toto attestation:
cosign attest --yes --predicate sbom.spdx.json \
  --type spdxjson "$REGISTRY/$REPO@$DIGEST"
cosign attest --yes --predicate provenance.json \
  --type slsaprovenance "$REGISTRY/$REPO@$DIGEST"

Keyless signing with OIDC is the right default — no long-lived COSIGN_PRIVATE_KEY sitting in a CI secret, no key rotation project. Each build signs with an ephemeral credentials identity tied to the pipeline run.

Admission policy in the cluster (Kyverno)

Argo CD will still happily apply a deployment referencing an unsigned image. We stop it with an admission policy:

# gitops/policies/verify-image.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: require-signed-images }
spec:
  validationFailureAction: Enforce
  rules:
  - name: verify-signature
    match: { resources: { kinds: [Pod] } }
    verifyImages:
    - imageReferences:
      - "registry.example.com/*"
      attestors:
      - count: 1
        entries:
        - keyless:
            subject: "$REGISTRY/$REPO@$DIGEST"
            issuer: "https://token.actions.githubusercontent.com"
            rekor: { url: https://rekor.sigstore.dev }
      verifyDigest: true
      required: true

Now a sync that references api:latest, or a digest with no signature, is rejected at the API server. Argo CD's sync shows Failed · admission denied and the bad image never schedules. This is the gate that lets us run Argo CD with auto-sync: true in production without losing sleep.

3 · Per-Namespace, CI-Driven Deployments

The temptation as a platform grows is to give every team their own Application object and let them file JSON patches. This scales to ~12 services before the review burden collapses. We instead use a single ApplicationSet templated off namespace directories, where CI does the promotion by writing to Git.

The ApplicationSet

# gitops/appset/services.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata: { name: services }
spec:
  generators:
  - git:
      repoURL: https://github.com/org/gitops
      revision: HEAD
      directories:
      - path: services/*/{staging,prod}   # one row per service-env
  template:
    metadata:
      name: '{{path.basename}}-{{path[2]}}'   # e.g. payments-api-staging
    spec:
      project: '{{path[2]}}'                       # Argo CD AppProject per env
      destination:
        namespace: '{{path[1]}}-{{path[2]}}'              # e.g. payments-api-staging
      syncPolicy:
        automated: { prune: true, selfHeal: true }
      source:
        repoURL: https://github.com/org/gitops
        targetRevision: HEAD
        path: '{{path}}'

CI promotes changes by updating the namespace-scoped declaration in Git. Argo CD remains the reconciler, so deployment state stays reviewable and recoverable from version control.