Clustering

A multi-replica MCPG fleet needs shared state: which replica owns a session, where to fan a notifications/cancelled, who holds a lease. The top-level cluster block selects one coordinator backend that provides all of it. This page documents the real per-backend config keys — the cluster block flows verbatim to the coordinator plugin, which validates it at boot.

The coordinator model

The cluster coordinator is the unified backbone for multi-instance state and coordination. A single backend exposes two orthogonal primitives, plus a coordinator-level surface:

• KeyValueStore — session / task / subscription / pipeline state, plus the atomic put_if_absent (single-winner reservation) and incr (delivery sequencing, quota counters).
• PubSub — delivery + cancellation fan-out across replicas.
• Coordinator surface — leader election and fenced distributed locks (with fencing tokens), and peer presence/health (watch_peers). These are methods on the coordinator itself, not separate primitives.

Capabilities (sessions, pipelines, tasks, subscriptions, delivery, cancellation) inherit those primitives by default. That inheritance is the whole trick: because every capability defaults to "use the cluster coordinator," a single-node deployment works with no extra config — cluster.kind: single_node installs an in-process coordinator and the capabilities transparently use its in-memory primitives.

Selecting a backend

cluster:
  kind: redis              # single_node | redis | nats
  # ...kind-specific keys below

single_node is the default when cluster is omitted — it ignores every other field in the block. Any other kind maps to a dev.mcpg.cluster.<kind> cdylib plugin that must be declared under the top-level plugins[] array. The inline cluster.* fields are the single source of truth for the coordinator's runtime config — they replace any config: block on the matching plugins[] entry, so you keep the cdylib location (source.oci / source.path) in plugins[] and the operational knobs in cluster.

If cluster.kind selects a plugin id with no matching plugins[] entry, the gateway fails fast at boot.

Try it locally in 30 seconds

mcpg dev cluster runs a real multi-node cluster on your machine — one NATS JetStream container (docker or podman required) plus N gateway nodes coordinating through it:

mcpg dev cluster up -n 3    # NATS container + 3 nodes on 127.0.0.1:8788-8790
mcpg dev cluster status     # node liveness + container state
mcpg dev cluster down       # stop everything (--purge also deletes the state dir)

up prints copy-paste curl examples that make the clustering visible: an MCP session minted on node-1 answers tools/call on node-2 and node-3 (the session lives in NATS, not in any node), and the cluster.ask demo tool suspends a call on one node and resumes it on another. The command pulls the dev.mcpg.cluster.nats coordinator plugin from the public registry on first boot (or uses a local build via --cluster-plugin / MCPG_PLUGIN_DIR), generates a throwaway config + state key under ~/.mcpg/dev-cluster, and doubles as a local harness for cluster development. It runs with allow_insecure_transport: true against a loopback broker under the license's non-production grant — a demo topology, never a production one.

A note on validation layers

mcpg config check validates the gateway's AppConfig shape. The cluster block's kind-specific fields are passed through as an opaque JSON map and are parsed by the coordinator plugin at boot, where each plugin enforces deny_unknown_fields. The practical consequence: a key that mcpg config check accepts can still be rejected by the plugin at startup. Use the exact key names below — they are taken directly from each plugin's config schema.

Transport security

The cluster coordinator carries all shared multi-instance state — sessions (including SSE replay windows with tool results), credential-cache events, delivery payloads, tasks, pipelines, and lease/fence tokens. A plaintext coordinator link exposes them across the whole deployment, so the gateway is secure by default: it refuses to boot against a plaintext (non-TLS) coordinator unless you explicitly opt in.

cluster:
  kind: redis
  url: "rediss://redis.svc:6379"   # TLS scheme → boots
  allow_insecure_transport: false  # default; set true ONLY for local/dev

What counts as plaintext, per coordinator:

Set allow_insecure_transport: true to accept a plaintext transport (local/dev/CI on a trusted network). The Helm chart exposes this as cluster.insecureTransport; the bundled redis/nats subcharts run plaintext in-namespace and the chart sets the opt-out automatically for that quick-start path only — point at an external, TLS-terminated coordinator for production.

Note. The boot guard inspects the configured value, which is evaluated before ${env.X} substitution. If you supply the coordinator URL via an env placeholder (url: "${env.MCPG_REDIS_URL}"), the guard can't see the resolved scheme — set a TLS scheme in the env value itself (rediss:// / tls:// / https://). The Helm chart renders literal URLs, so it is always covered.

Coordinator health + readiness

For any non-single_node coordinator (all of them — redis / nats — expose a KV), the gateway runs a periodic KV-ping probe and publishes a mcpg_cluster_backend_up gauge (1 reachable / 0 down) — independent of any lease consumer, so the McpgClusterCoordinatorDown alert fires even on a deployment running neither cedar nor identity.workload. cluster.readiness_gate controls whether that feeds /ready:

cluster:
  kind: redis
  url: "rediss://redis.svc:6379"
  readiness_gate: fail   # off | degrade | fail

Both external coordinators expose a KV, so both are probed — the gate applies uniformly.

The gateway also runs the probe once at boot (fail-closed): a non-single_node coordinator that is unreachable, or that advertises a role whose primitive is missing, stops the gateway from starting. Set cluster.allow_degraded_boot: true to boot anyway (per-replica state, with a loud error) — otherwise a replica can never come up silently de-clustered.

State encryption

A non-single_node coordinator requires cluster.state_encryption_key_env — the gateway refuses to boot without it. It names an env var holding a URL-safe base64 32-byte key that seals all coordinator-backed state (sessions including SSE replay, pipelines, tasks, delivery, idempotency, quota, request-state) with XChaCha20-Poly1305, and also derives the deterministic modern-session and request-state keys that make cross-replica resume work by default. The escape hatch is cluster.allow_plaintext_state: true (loud warning, cleartext state, cross-replica modern-session resume off) — for migration only.

cluster:
  kind: nats
  servers: ["tls://nats.svc:4222"]
  state_encryption_key_env: MCPG_CLUSTER_STATE_KEY   # required when clustered

Redis (dev.mcpg.cluster.redis)

cluster:
  kind: redis
  url: "${env.MCPG_REDIS_URL}"     # required — rediss:// (TLS) for production
  password: "${env.REDIS_PASSWORD}" # optional — keep credentials OUT of the url
  key_prefix: "mcpg:cluster:"      # namespace per deployment
  lease_ttl_ms: 30000              # default lock / leadership TTL
  peer_ttl_ms: 60000               # per-instance peer-presence TTL

There is no pool_size key on the Redis coordinator — connection pooling is internal: operations run concurrently over one multiplexed connection. Every write keeps a key index (<key_prefix>#kv-index), so a prefix listing costs what it matches; while a replica of an older plugin build is registered, listings scan the keyspace instead, as before.

The Redis coordinator needs one Redis primary: standalone, Sentinel, or a managed service in non-cluster mode. Every write is one Lua script over the data key and the two key indexes, so Redis Cluster mode, and proxies that route keys by hash slot, are not supported. Each live key takes about 750 bytes of Redis memory with 256-byte values.

NATS JetStream (dev.mcpg.cluster.nats)

NATS uses a list of servers and a structured jetstream block — not a single url / bucket.

cluster:
  kind: nats
  servers:
    - "${env.MCPG_NATS_URL}"       # nats:// | tls:// | nats+tls:// | ws:// | wss://
  node:
    id: "${env.HOSTNAME}"          # required — unique per replica
  jetstream:
    replicas: 3                    # 1 for dev, 3 for a real NATS cluster
    storage: file                  # file (durable) | memory

Size the state bucket for the live state. Each live key holds two records (its value and its key-index record): about 560 bytes per key with 256-byte values, so the default 256 MiB holds about 480 000 such keys. Every replica warns above 80 % (mcpg_cluster_kv_bucket_bytes against mcpg_cluster_kv_bucket_max_bytes). The sweep scans at a bounded rate: a bucket of a million keys is swept over about two minutes.

NATS auth is tagged on method:

cluster:
  kind: nats
  servers: ["tls://nats.svc:4222"]
  node:
    id: "${env.HOSTNAME}"
  auth:
    method: credentials_file
    path: "/etc/mcpg/nats.creds"
  tls:
    ca_cert: "/etc/mcpg/certs/nats-ca.pem"
    require_tls: true   # default — set false only for a plaintext dev link

Per-capability overrides

By default every capability inherits the cluster coordinator's primitives — that is what you want for HA, and what makes single-node work with zero config. You can override an individual capability's store or bus, but only to in-process kinds:

mcp:
  configurations:
    sessions:
      store:
        kind: cluster     # default — delegate to the coordinator (same as omitting)
    delivery:
      bus:
        kind: memory      # pin to in-process — single-replica only

redis and nats are not valid per-capability override kinds. To put capability state on Redis or NATS, set cluster.kind: redis | nats and let the capability inherit (via kind: cluster or by omitting the override). Pinning delivery.bus or cancellation.bus to memory in a multi-replica deployment silently breaks cross-replica delivery — leave them inherited.

Validating

mcpg config check ./config.yaml
# ✓ ./config.yaml: valid (1 bindings, audit on, observability on, cluster: nats)

Remember the two-layer model: mcpg config check confirms the AppConfig shape; the coordinator plugin validates the kind-specific keys (URL scheme, required fields, ranges) at boot. Use the key names above to avoid a boot-time rejection that the config check can't see.

What's next

• Deployment topologies — single-node vs HA shapes
• Kubernetes operator — Helm auto-wires cluster.kind
• Configuration reference — every config key