Clustering

A multi-replica MCPG fleet needs shared state: which replica owns a session, where to fan a notifications/cancelled, who holds a lease. The top-level cluster block selects one coordinator backend that provides all of it. This page documents the real per-backend config keys — the cluster block flows verbatim to the coordinator plugin, which validates it at boot.

The coordinator model

The cluster coordinator is the unified backbone for multi-instance state and coordination. A single backend internally provides four primitives:

• KeyValueStore — session / task / subscription / pipeline state.
• PubSub — delivery + cancellation fan-out across replicas.
• Lease — leader election and distributed locks (fencing tokens).
• Watch — peer presence and change notification.

Capabilities (sessions, pipelines, tasks, subscriptions, delivery, cancellation) inherit those primitives by default. That inheritance is the whole trick: because every capability defaults to "use the cluster coordinator," a single-node deployment works with no extra config — cluster.kind: single_node installs an in-process coordinator and the capabilities transparently use its in-memory primitives.

Selecting a backend

cluster:
  kind: redis              # single_node | redis | nats | consul | etcd
  # ...kind-specific keys below

single_node is the default when cluster is omitted — it ignores every other field in the block. Any other kind maps to a dev.mcpg.cluster.<kind> cdylib plugin that must be declared under the top-level plugins[] array. The inline cluster.* fields are the single source of truth for the coordinator's runtime config — they replace any config: block on the matching plugins[] entry, so you keep the cdylib location (source.oci / source.path) in plugins[] and the operational knobs in cluster.

If cluster.kind selects a plugin id with no matching plugins[] entry, the gateway fails fast at boot.

A note on validation layers

mcpg config check validates the gateway's AppConfig shape. The cluster block's kind-specific fields are passed through as an opaque JSON map and are parsed by the coordinator plugin at boot, where each plugin enforces deny_unknown_fields. The practical consequence: a key that mcpg config check accepts can still be rejected by the plugin at startup. Use the exact key names below — they are taken directly from each plugin's config schema.

Transport security

The cluster coordinator carries all shared multi-instance state — sessions (including SSE replay windows with tool results), credential-cache events, delivery payloads, tasks, pipelines, and lease/fence tokens. A plaintext coordinator link exposes them across the whole deployment, so the gateway is secure by default: it refuses to boot against a plaintext (non-TLS) coordinator unless you explicitly opt in.

cluster:
  kind: redis
  url: "rediss://redis.svc:6379"   # TLS scheme → boots
  allow_insecure_transport: false  # default; set true ONLY for local/dev

What counts as plaintext, per coordinator:

Set allow_insecure_transport: true to accept a plaintext transport (local/dev/CI on a trusted network). The Helm chart exposes this as cluster.insecureTransport; the bundled redis/nats subcharts run plaintext in-namespace and the chart sets the opt-out automatically for that quick-start path only — point at an external, TLS-terminated coordinator for production.

Note. The boot guard inspects the configured value, which is evaluated before ${env.X} substitution. If you supply the coordinator URL via an env placeholder (url: "${env.MCPG_REDIS_URL}"), the guard can't see the resolved scheme — set a TLS scheme in the env value itself (rediss:// / tls:// / https://). The Helm chart renders literal URLs, so it is always covered.

Coordinator health + readiness

For a non-single_node coordinator that exposes a KV (redis / nats), the gateway runs a periodic KV-ping probe and publishes a mcpg_cluster_backend_up gauge (1 reachable / 0 down) — independent of any lease consumer, so the McpgClusterCoordinatorDown alert fires even on a deployment running neither cedar nor identity.workload. cluster.readiness_gate controls whether that feeds /ready:

cluster:
  kind: redis
  url: "rediss://redis.svc:6379"
  readiness_gate: fail   # off | degrade | fail

Coordination-only kinds (consul / etcd, no KV accessor) can't be probed, so the gate is inert for them (the gateway logs a WARN if you set it).

Redis (dev.mcpg.cluster.redis)

cluster:
  kind: redis
  url: "${env.MCPG_REDIS_URL}"     # required — rediss:// (TLS) for production
  password: "${env.REDIS_PASSWORD}" # optional — keep credentials OUT of the url
  key_prefix: "mcpg:cluster:"      # namespace per deployment
  lease_ttl_ms: 30000              # default lock / leadership TTL
  peer_ttl_ms: 60000               # per-instance peer-presence TTL

There is no pool_size key on the Redis coordinator — connection pooling is internal.

NATS JetStream (dev.mcpg.cluster.nats)

NATS uses a list of servers and a structured jetstream block — not a single url / bucket.

cluster:
  kind: nats
  servers:
    - "${env.MCPG_NATS_URL}"       # nats:// | tls:// | nats+tls:// | ws:// | wss://
  node:
    id: "${env.HOSTNAME}"          # required — unique per replica
  jetstream:
    replicas: 3                    # 1 for dev, 3 for a real NATS cluster
    storage: file                  # file (durable) | memory

NATS auth is tagged on method:

cluster:
  kind: nats
  servers: ["tls://nats.svc:4222"]
  node:
    id: "${env.HOSTNAME}"
  auth:
    method: credentials_file
    path: "/etc/mcpg/nats.creds"
  tls:
    ca_cert: "/etc/mcpg/certs/nats-ca.pem"
    require_tls: true   # default — set false only for a plaintext dev link

Consul (dev.mcpg.cluster.consul)

Consul uses address (the HTTP API base URL), not url.

cluster:
  kind: consul
  address: "https://consul.svc:8500"  # required — https:// for production
  service_name: "mcpg"                # required
  kv_prefix: "mcpg/prod/"             # distinct per deployment

etcd (dev.mcpg.cluster.etcd)

etcd uses a list of endpoints, and key_prefix must end in /.

cluster:
  kind: etcd
  endpoints:
    - "https://etcd-0:2379"
    - "https://etcd-1:2379"
  key_prefix: "/mcpg/prod/"           # MUST end with '/'
  tls:
    ca_cert: "/etc/mcpg/certs/etcd-ca.pem"   # omit to use system roots
    # client_cert / client_key: "..."        # for mTLS
    # domain_name: "etcd.internal"            # SNI / cert-name override
  auth:
    username: "mcpg"
    password: "${env.ETCD_PASSWORD}"

Per-capability overrides

By default every capability inherits the cluster coordinator's primitives — that is what you want for HA, and what makes single-node work with zero config. You can override an individual capability's store or bus, but only to in-process kinds:

mcp:
  configurations:
    sessions:
      store:
        kind: cluster     # default — delegate to the coordinator (same as omitting)
    delivery:
      bus:
        kind: memory      # pin to in-process — single-replica only

redis and nats are not valid per-capability override kinds. To put capability state on Redis or NATS, set cluster.kind: redis | nats and let the capability inherit (via kind: cluster or by omitting the override). Pinning delivery.bus or cancellation.bus to memory in a multi-replica deployment silently breaks cross-replica delivery — leave them inherited.

Validating

mcpg config check ./config.yaml
# ✓ ./config.yaml: valid (1 bindings, audit on, observability on, cluster: nats)

Remember the two-layer model: mcpg config check confirms the AppConfig shape; the coordinator plugin validates the kind-specific keys (URL scheme, required fields, ranges) at boot. Use the key names above to avoid a boot-time rejection that the config check can't see.

What's next

• Deployment topologies — single-node vs HA shapes
• Kubernetes operator — Helm auto-wires cluster.kind
• Configuration reference — every config key