n8n + MCP on Kubernetes: production architecture patterns

1. Give each layer one clear job

Keeping those roles explicit prevents a useful automation framework from quietly becoming an unrestricted control plane.

2. Choose execution topology from workload behavior

A small internal workflow service may need only one n8n instance and durable database storage. Higher concurrency, long-running jobs or availability requirements can justify queue-backed workers and separate execution capacity. Choose the topology from measured workload needs instead of copying a “production” diagram.

3. Treat MCP servers as trust boundaries

An MCP server that can read a repository is different from one that can merge code, mutate infrastructure or send customer messages. Deploy and authorize them accordingly.

• Separate read-only and mutating tools when useful.
• Grant each server only the upstream credentials it requires.
• Prefer short-lived workload identity over long-lived tokens where supported.
• Apply namespace, network and admission controls according to the tool’s risk.
• Log material tool calls with enough context to investigate later.
• Keep human approval before irreversible or high-impact actions unless the workflow has an explicitly approved automation boundary.

4. Keep secrets out of workflow exports and manifests

Credentials should come from an approved secret-management path, not Git, hard-coded workflow JSON or copied environment files. Kubernetes Secrets may be the delivery object, while an external secret manager, encrypted Git workflow or platform-native identity handles the source of truth.

Rotation matters as much as storage. Test whether a credential can be changed or revoked without manually rebuilding every workflow.

5. Network the system by capability

Use ClusterIP-style internal services for components that do not need public exposure. Put external ingress only in front of the endpoints that require it. NetworkPolicy is useful when the CNI enforces it, but policy should reflect actual dependencies: DNS, database, queue, selected MCP services and required external APIs.

A workflow runner that can reach every internal service and every external API by default has a larger blast radius than most automations need.

6. Build observability around the workflow, not only Pods

Pod CPU and restart counts are necessary but insufficient. Instrument the automation outcome.

7. Design retries and idempotency before scaling

Distributed automation fails partially. A webhook may arrive twice. A worker can crash after an external API accepts the request but before the workflow records success. Build idempotency keys, deduplication, state transitions and compensation where the business action requires them.

8. Use GitOps for deployment, not secret or workflow theatre

Helm, Kustomize or another declarative packaging approach can version the platform configuration. CI can render and validate manifests; a GitOps controller can reconcile reviewed desired state. Keep runtime credentials outside plaintext Git and document how emergency operational changes interact with reconciliation.

git change
  → render / validate
  → review
  → merge desired state
  → GitOps reconciliation
  → health and workflow-level verification

9. Preserve a human boundary for material actions

AI-enabled automation is most useful when routine work can move quickly while consequential actions remain accountable. Examples that often justify approval or stronger policy include production infrastructure mutation, financial transactions, destructive data changes, external publication, customer-impacting account actions and security-sensitive credential changes.

The boundary should be encoded in the workflow or authorization layer rather than left as a tribal expectation.

10. Model cost as a workload equation

There is no universal “self-hosted wins” number. Compare the costs that apply to your workload: compute, database, queue/cache, storage, ingress/egress, observability, backups, engineering time and operational risk. Then divide by a useful business unit such as successful workflow executions or automated cases handled.

A managed service can be economically better at one scale and self-hosting can be better at another. The decision changes with workload, labor, compliance, control and reliability requirements.

Production validation checklist

• Workflow state and credentials survive Pod replacement as intended.
• Database backup and restore are tested.
• Retries do not duplicate material actions.
• MCP servers have bounded identities and network access.
• Mutating tools require the intended authorization or approval.
• Ingress, TLS and webhook authentication are validated.
• Queues and workers expose actionable saturation signals if used.
• Deployment reconciliation and rollback behavior are understood.
• A representative end-to-end workflow is tested after deployment.
• Operator runbooks state what to do when external APIs, models or tools are unavailable.