DocumentationLicense
Operations
Operations

Day-2 operations and upgrades

Keep a running MCPG fleet healthy — roll out new versions without dropping traffic, push config changes in place, back up cluster state, and upgrade the operator and its CRDs safely.

Getting MCPG installed is day 1. This page is day 2: how to change a running fleet — new binaries, new config, operator upgrades — without dropping requests, and what to back up so you can recover.

Two kinds of change

Not every change needs a restart. Separate them:

  • Config changes (a new tool, a policy tweak, a sink) can be applied to a live gateway in place — no process restart, no dropped sessions.
  • Binary/image changes (a new gateway version, a new plugin build) require the process to be replaced, so they go through a rolling deploy.

Reload config in place

When a control plane is attached, publishing a new config version pushes it to each running gateway, which validates and swaps it live. The listener never goes down. If you self-manage config files, the Kubernetes operator is the equivalent: edit an MCPGGateway's spec.config, and the operator re-renders the ConfigMap and triggers a controlled rollout.

Always validate before you ship a config change — the same check CI runs:

bash
mcpg config check new-config.yaml

Roll out a new version without dropping traffic

Version bumps are per project — the gateway, operator, and each plugin carry independent versions — so you upgrade the piece that changed, not the whole world. On Kubernetes, a new gateway image is a standard Deployment rollout; the readiness gate is what makes it safe:

  • gateway.server.health_path (default /health) is the liveness/readiness endpoint Kubernetes probes. A new pod only receives traffic once it answers, so a bad config or a failed plugin load holds the rollout instead of taking the fleet down.
  • Keep maxUnavailable low and maxSurge at least one so there is always a ready replica serving while the next one comes up.

For the modern stateless wire (2026-07-28), requests round-robin with no session affinity, so rolling pods is transparent. Sessions on the 2025-11-25 wire are pinned, so drain gracefully (below) to avoid cutting an in-flight session mid-call.

Drain gracefully

On shutdown the gateway stops accepting new work and lets in-flight calls finish before exiting. Give Kubernetes a terminationGracePeriodSeconds comfortably longer than your slowest tool call so a rolling replacement never severs an active request. In a clustered deployment, a leaving replica releases its coordinator leases so the rest of the fleet takes over cleanly — see Clustering for the state-sharing model.

Back up what matters

The gateway process itself is stateless-to-disk — its source of truth is the config. What you back up is:

  • The config (and any IaC that renders it) — version-controlled, so a rollback is a re-apply of a known-good version. The managed cloud keeps versioned config artifacts for exactly this.
  • Cluster coordinator state — if you run Redis or NATS JetStream for shared primitives (idempotency keys, suspended-call handles, rate-limit counters), back that store up on its own cadence. Losing it degrades in-flight suspended calls, not your served surface.
  • The audit ledger — ship each replica's audit trail to your SIEM or object store; the local file is per-replica and ephemeral.

Upgrade the operator and CRDs

When you upgrade the Kubernetes operator, apply the matching CRD version first (the operator reconciles resources it understands), then roll the operator Deployment. The admission webhook validates resources on the way in, so a schema-incompatible change is rejected at kubectl apply rather than half-reconciled. The operator's own metrics and probes on :8443 tell you when it is healthy again.

Where to go next