MCP Orchestrator — Helm Chart

Current release: the 4.1.x family — Helm chart on magertron.com/charts. For the exact version being served: helm repo update magertron && helm search repo magertron/mcp-orchestrator --versions | head -3

Install guide for the MCP Orchestrator Helm chart, focused on on-prem and private-cloud Kubernetes deployments. If you're on a managed cloud (EKS / GKE / AKS), the chart works there too — see the last appendix.

Sizing. The core platform requests just under 4 cores of CPU and about 4.5 GB of memory — orchestrator, three Envoy replicas, two PostgreSQL instances, the inventory service, and the two bundled MCP servers. Memory is not the constraint; CPU requests are, and a 4-core node has no room left for a server of your own. 8 cores is the practical floor unless you trim replica counts. Add roughly 5 GB and a GPU-free core if you enable the local LLM (Ollama), and more again for the embedded GRC service (Probo) — both are off by default.

Quickstart

The fastest path to a working install is the bundled script. It does everything below (preflight, license setup, node labeling, helm install, post-install verification) with one command:

git clone https://github.com/magertron/orchestrator.git
cd orchestrator
./install.sh --license /path/to/your-license.json

The script supports both fresh installs and upgrade-in-place. Pass --mode reinstall for a destructive fresh start (it requires typing 'destroy' to confirm); the default --mode upgrade preserves data.

The rest of this README walks through the same install manually — use it to understand exactly what the script is doing, or when your environment needs custom values the script doesn't expose.

Installing the CLI

mcpctl is the command-line client. It talks to a running orchestrator over HTTPS to deploy MCP servers, manage users and service accounts, and inspect cluster state. The CLI is Apache 2.0 licensed and free for all customers.

Pick your platform:

# Homebrew (macOS or Linux)
brew install magertron/tap/mcpctl

# curl one-liner (macOS or Linux)
curl -fsSL https://magertron.com/install-mcpctl.sh | sh

# Debian / Ubuntu (one-time setup, then apt install)
curl -fsSL https://magertron.com/apt/magertron-archive-keyring.gpg \
    | sudo tee /etc/apt/trusted.gpg.d/magertron-archive-keyring.gpg > /dev/null
echo "deb [signed-by=/etc/apt/trusted.gpg.d/magertron-archive-keyring.gpg] https://magertron.com/apt stable main" \
    | sudo tee /etc/apt/sources.list.d/magertron.list
sudo apt update && sudo apt install mcpctl

# RHEL / Fedora / Rocky
sudo curl -fsSL https://magertron.com/yum/magertron.repo -o /etc/yum.repos.d/magertron.repo
sudo dnf install mcpctl

Full CLI usage guide: see mcpctl/README.md.

What you're installing

The chart deploys the full MCP Orchestrator platform to your cluster:

Orchestrator (2 replicas by default) — the control plane, REST API, and admin UI
Envoy gateway — TLS termination and HTTP routing to MCP servers
PostgreSQL (×2) — one in-cluster database for orchestrator identity & audit, one for the mcp-inventory service that tracks deployed servers, service accounts, and grants
mcp-sync sidecar — watches MCP server CRDs and pushes Envoy routes
mcp-inventory — durable store for service accounts, orgs, and per-SA lifecycle state (expiry, rotation, last-used, owner email)
Namespaces for MCP servers (mcp-prod, mcp-staging, mcp-dev)
Network policies isolating MCP server namespaces
RBAC — a ServiceAccount + ClusterRole granting the orchestrator the k8s permissions it needs to deploy MCP servers on your behalf
CRDs — the mcp.io/McpRoute custom resource used internally

After install you retrieve the auto-generated admin password from the Kubernetes Secret (instructions printed in the post-install notes), log in, change the password, optionally apply an Enterprise license, and start deploying MCP servers.

Prerequisites

You need these BEFORE running helm install. The chart can auto-generate TLS certificates, JWT keys, and the database/admin passwords for you (the default behavior, suitable for dev/PoC), but for production deployments we recommend providing your own — see the relevant prerequisite sections below and the production-hardening guidance in the post-install notes.

1. A Kubernetes cluster

Version: 1.28 or newer
Who you talk to when you configure it: k8s admin (you)
Supported flavors for on-prem:
- k3s — recommended for single-box deployments
- kubeadm-installed — for multi-node production clusters
- RKE2 / OpenShift / Rancher — also fine, not explicitly tested

If using k3s, disable its bundled Traefik:
curl -sfL https://get.k3s.io | sh -s - --disable=traefik
Otherwise Traefik and Envoy both try to bind port 443 and one will fail.

2. Helm 3.x and kubectl

helm version    # want v3.x
kubectl version --client
kubectl get nodes     # confirms you can reach the cluster

3. A TLS certificate for your hostname

MCP Orchestrator terminates TLS at the Envoy gateway using a cert you provide. Pick the hostname customers will use to reach the orchestrator (e.g. mcp.yourcompany.com) and get a cert+key for it.

Options to get a cert:

Let's Encrypt via certbot (free, 90-day rotation)
Your company's internal CA
A commercial cert vendor

You need two files: tls.crt (full chain) and tls.key (private key).

The chart CAN generate a self-signed cert if you don't provide one, but self-signed certs only work for testing — browsers warn and IdPs will reject them.

4. JWT signing keypair

The orchestrator signs session tokens with RS256. Generate a keypair:

openssl genpkey -algorithm RSA -out jwt.key -pkeyopt rsa_keygen_bits:2048
openssl rsa -in jwt.key -pubout -out jwt.pub

Keep these files safe. Rotating them invalidates all active user sessions AND all service-account JWTs (existing SAs will need to be re-rotated through the API or UI to mint new tokens).

5. metrics-server (optional but recommended)

Needed for CPU/memory charts in the UI and for HorizontalPodAutoscaler on Envoy. Most production clusters have it already. If not:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

For non-cloud clusters you may need to add --kubelet-insecure-tls to the metrics-server deployment args.

6. License file (optional — Enterprise tier only)

Free tier covers core deployment, health monitoring, scaling, RBAC. Enterprise tier adds SSO, SCIM, governance, audit export, webhooks, deployment history, rollback.

If you have a license JSON file, you can apply it at install time or later. See "Apply a license" below.

Choose your ingress path

Your cluster needs to expose the Envoy gateway to the outside world somehow. This is the single biggest decision for on-prem installs.

Pick one of these based on your environment:

NodePort (recommended for single-box / small installs)

Envoy runs as a NodePort Service on port 30444. You configure your firewall, router or tunnel to forward external 443 to <cluster-node-ip>:30444. Works everywhere, no k8s add-ons required.

values.yaml:

loadBalancer:
  provider: nodeport

envoy:
  v3:
    nodePort: 30444   # can be 30000-32767

If you have read an older copy of this guide, it said 30443. That was the legacy Envoy data plane, retired once every server moved to routing_mode: v3. The port is pinned by the chart and needs no kubectl patch step.

k3s ships Traefik, which takes 443. Nothing routes through it to Magertron, so a request to https://<node-ip> returns Traefik's own 404 rather than the dashboard. Either disable it — --disable traefik, or disable: traefik in /etc/rancher/k3s/config.yaml — or reach the UI on 30444 directly.

MetalLB (multi-node bare-metal with LAN IP pool)

MetalLB assigns a "real" LoadBalancer IP from a pool you give it. Good when you have multiple cluster nodes and want a single stable IP that follows the Envoy pod.

Requires: a block of IPs on your LAN that aren't being used by DHCP.

values.yaml:

loadBalancer:
  provider: metallb
  metallb:
    install: true                # chart installs MetalLB
    ipRange: "10.0.1.240-10.0.1.250"   # adjust to YOUR LAN

Existing ingress controller (nginx-ingress, Traefik, etc.)

If your cluster already has an ingress controller handling TLS and HTTP routing for other apps, use it for the orchestrator too.

values.yaml:

loadBalancer:
  provider: existing
  existing:
    ingress:
      enabled: true
      className: nginx
      host: mcp.yourcompany.com
      tls: true
      tlsSecretName: mcp-envoy-tls

Cloud LoadBalancer (EKS / GKE / AKS)

Default mode. Your cloud provider's LoadBalancer controller assigns a public IP automatically. See the cloud appendix at the bottom.

Install (manual)

Most users should use the install.sh script above. This section documents the same steps the script performs, useful when you need to customize values the script doesn't expose, or when you want to read every step before running anything.

Add the helm repo (one time)

helm repo add magertron https://magertron.com/charts
helm repo update

Minimum viable install (Free tier, self-signed cert, NodePort)

Fine for kicking the tires. Not for production.

helm install mcp magertron/mcp-orchestrator \
  --namespace mcp-system --create-namespace \
  --set loadBalancer.provider=nodeport \
  --set-file secrets.jwtPrivateKey=jwt.key \
  --set-file secrets.jwtPublicKey=jwt.pub

Production install (Free tier)

helm install mcp magertron/mcp-orchestrator \
  --namespace mcp-system --create-namespace \
  --set loadBalancer.provider=nodeport \
  --set orchestrator.env.apiPublicUrl=https://mcp.yourcompany.com \
  --set secrets.dbPassword='<strong-password>' \
  --set-file secrets.jwtPrivateKey=jwt.key \
  --set-file secrets.jwtPublicKey=jwt.pub \
  --set-file tls.cert=tls.crt \
  --set-file tls.key=tls.key

Production install (Enterprise tier with license)

Same as above plus the license. The license JWT lives in a Secret:

kubectl create namespace mcp-system
kubectl create secret generic mcp-license \
  --from-file=license.json=/path/to/license.json \
  -n mcp-system

Then helm install ... as above. The orchestrator reads the license from this Secret at startup.

What the flags mean

Flag	Purpose
`--namespace mcp-system --create-namespace`	Install into a dedicated namespace
`loadBalancer.provider=...`	Pick your ingress path (see above)
`orchestrator.env.apiPublicUrl=...`	Public HTTPS URL for SCIM meta.location + OIDC redirects. Must match the URL customers actually use.
`secrets.dbPassword=...`	Set the PostgreSQL password. Default is `changeme` — don't ship it that way.
`--set-file secrets.jwtPrivateKey=...`	JWT signing key (RS256 PEM)
`--set-file secrets.jwtPublicKey=...`	JWT verification key (RS256 PEM)
`--set-file tls.cert=...`	Your TLS cert (full chain PEM)
`--set-file tls.key=...`	Your TLS private key (PEM)

After install

1. Watch the rollout

kubectl get pods -n mcp-system -w

Expected pods (give it 2-3 minutes on first install):

mcp-orchestrator-xxxxx                       2/2 Running
mcp-orchestrator-xxxxx                       2/2 Running
mcp-orchestrator-envoy-xxxxx                 1/1 Running
mcp-orchestrator-envoy-xxxxx                 1/1 Running
mcp-orchestrator-postgres-xxxxx              1/1 Running
mcp-orchestrator-inventory-xxxxx             1/1 Running
mcp-orchestrator-inventory-postgres-xxxxx    1/1 Running

If anything stays in Init or Pending for more than a minute, see Troubleshooting below.

Note: mcp-orchestrator-inventory typically restarts once or twice in the first 30 seconds while it waits for its Postgres to accept connections. This is normal and self-resolves.

2. Access the dashboard

For NodePort mode, if you haven't configured your firewall yet, quick test via port-forward:

kubectl port-forward svc/mcp-orchestrator-envoy \
  -n mcp-system 8443:443

Then open https://localhost:8443 in your browser. Self-signed cert warning expected if you didn't provide a real cert.

Log in as admin. Retrieve the auto-generated password from the Kubernetes Secret:

kubectl get secret -n mcp-system mcp-orchestrator-secrets \
  -o jsonpath='{.data.MCP_SEED_ADMIN_PASSWORD}' | base64 -d ; echo

(If you supplied your own password via --set seed.adminPassword=... at install, use that instead.)

Change the password immediately via Users tab → admin → Password. This is a bootstrap credential — treat it like an initial root password: use it once, change it, then rotate the Secret value to invalidate the bootstrap. See the post-install notes (helm get notes mcp -n mcp-system) for full rotation guidance.

Set the admin email while you're there — Users → admin → Email. As of v2.4.x, email is required for new users and is used for password reset and account event notifications. See Identity linking.

3. Configure SSO (Enterprise — optional)

Admin → "Identity Providers" tab → "+ Add provider"

Fill in the OIDC or SAML details from your IdP (Okta, Azure AD, Google Workspace, etc.). For OIDC you'll typically need:

Discovery URL (e.g. https://your-org.okta.com/.well-known/openid-configuration)
Client ID
Client Secret
Redirect URI: <apiPublicUrl>/api/v1/auth/sso/oidc/callback

If you want users to be auto-provisioned on first SSO login, set:

orchestrator:
  env:
    ssoAutoProvisionDomains: "yourcompany.com,partner.com"

Then helm upgrade to apply.

4. Configure SCIM (Enterprise — optional)

Admin → "Identity Providers" tab → "SCIM Tokens" sub-tab → "+ Mint token"

The modal shows the raw token exactly once. Copy it immediately — after dismissal, only the prefix is visible and you can't retrieve the raw value. If you lose the token, revoke and mint a new one.

In your IdP, create a SCIM 2.0 app pointing at:

Base URL: <apiPublicUrl>/scim/v2
Authentication: OAuth Bearer Token
Token: the raw token you just minted

Most IdPs have a "Test API Credentials" button. That's the fastest way to confirm the integration works.

5. Configure webhooks for expiry reminders (recommended)

The orchestrator fires HMAC-SHA256-signed webhooks 30, 10, and 5 days before any service account expires (see Service Account Lifecycle). Configure at least one webhook so credentials don't silently rot:

Admin → "Webhooks" tab → "+ Add webhook"

Provide a URL (Slack incoming webhook, generic HTTPS endpoint, etc.) and a shared secret. The orchestrator signs the payload with HMAC-SHA256; your endpoint should verify the X-Magertron-Signature header matches sha256=<hex> computed over the request body.

Service Account Lifecycle

Service accounts represent machine identities (AI agents, automation scripts, CI runners) that authenticate to MCP servers behind the orchestrator. Full lifecycle is built into the platform:

Create — Service Accounts tab → New. Set name, subject, role(s), scopes, TTL (default 90 days), and optional owner email.
List — see expiry status, last-used timestamp, owner email at a glance.
Rotate — generates a new JWT for an existing SA, soft-revokes the old credential. Used when the agent's environment has been compromised or as routine credential rotation. The owner email and audit history carry forward.
Revoke — soft-delete; preserves the audit chain. The JTI is added to the orchestrator's in-memory revocation set (refreshed every 30 seconds), so the old JWT stops authenticating across all replicas.
Owner email — optional contact field. Plumbs through to the expiry-reminder webhook payload so you can route reminders to a specific person.
Expiry reminders — 30 / 10 / 5 days before expiry, the orchestrator fires a webhook for each matching SA. Webhook payload includes SA id, subject, role IDs, owner email, days remaining, threshold matched.

All SA lifecycle events emit OCSF-aligned audit events (ServiceAccountCreated, Rotated, Revoked, ExpiringSoon) that appear in the Audit Log tab.

Identity linking (users + service accounts)

As of v2.4.x, both users and service accounts can be linked to a contact email:

users.email — required for new password-provisioned password users. Used for password reset and account event notifications. Set via the Create User modal at create time, or via Users → row → Email after. (SSO and SCIM users use their IdP-provided email; this field is separate and only applies to local password users.)
service_accounts.owner_email — optional contact for the human owner of a machine identity. Set via Create SA modal or PATCH after. Surfaced in the expiry-reminder webhook payload for routing.

Neither field is unique at the schema level — one human may have multiple usernames (e.g. alice / alice-admin) intentionally sharing an email, and one human may own many service accounts.

Audit log & retention

The Audit Log tab shows OCSF-aligned events across two planes:

Admin plane — user logins, service account lifecycle, policy changes, license events, retention events
Data plane — every request that traversed the Envoy gateway, with the principal, target, scopes consulted, and decision

A retention worker prunes old rows daily at 03:00 UTC. Default retention:

Table	Default retention
`audit_events`	90 days
`runtime_policy_audit`	30 days

Both thresholds are operator-configurable down to a 7-day safety floor (values below that are clamped to prevent accidentally wiping recent audit data). Configure via values:

retention:
  auditEventsDays: 90
  runtimePolicyAuditDays: 30
  cronExpression: "0 3 * * *"   # daily at 03:00 UTC

The worker is leader-elected — only one orchestrator pod runs the pruning even with multiple replicas — and emits a RetentionPruneCompleted audit event after each pass.

Production hardening

The chart auto-generates several credentials at install time for ease of use. These are appropriate for development, demos, and proofs of concept. For production, replace each with values sourced from your secret management system (Vault, AWS Secrets Manager, GCP Secret Manager, Sealed Secrets, etc).

This section mirrors the post-install notes (helm get notes mcp -n mcp-system) and is included here so customers evaluating the chart see the security story before installing.

TLS certificate

The chart auto-generates a self-signed TLS certificate if you don't provide one. Browsers will show a security warning and API clients will fail TLS verification by default. This is fine for dev but not for production.

The auto-generated cert has SANs only for mcp-orchestrator, localhost, and 127.0.0.1. Clients connecting via your public hostname (e.g. mcp.example.com) will fail verification regardless of whether they accept self-signed warnings.

Two production-grade alternatives:

Option A — Bring your own certificate at install time.

helm upgrade --install mcp magertron/mcp-orchestrator \
  --namespace mcp-system \
  --set-file tls.cert=/path/to/your-cert.pem \
  --set-file tls.key=/path/to/your-key.pem

Option B — Use cert-manager (recommended for cloud / Let's Encrypt).

Install cert-manager once per cluster:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml

Create a ClusterIssuer pointing at your CA (Let's Encrypt, internal PKI, AWS PCA, etc).

Pre-create a Certificate resource that writes to the mcp-envoy-tls Secret in the orchestrator's namespace:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: mcp-envoy-tls
  namespace: mcp-system
spec:
  secretName: mcp-envoy-tls
  dnsNames:
    - mcp.your-company.com
  issuerRef:
    name: your-issuer
    kind: ClusterIssuer

Run helm upgrade --install. The chart detects the existing TLS Secret (via Helm's lookup function) and reuses it. cert-manager handles renewal automatically.

JWT signing key

The JWT private key signs every authentication token issued by the orchestrator (user sessions AND service-account JWTs). If exposed, an attacker can forge tokens for any user or service account.

For production:

Option A — Generate keys on a trusted machine and pass at install:

openssl genpkey -algorithm RSA -out jwt.key -pkeyopt rsa_keygen_bits:2048
openssl rsa -in jwt.key -pubout -out jwt.pub

helm upgrade --install mcp magertron/mcp-orchestrator \
  --namespace mcp-system \
  --set-file secrets.jwtPrivateKey=jwt.key \
  --set-file secrets.jwtPublicKey=jwt.pub

shred -u jwt.key jwt.pub

Option B — Source from your KMS (Vault, AWS Secrets Manager, etc) and pre-create the Secret externally:

vault read -field=private secret/mcp-orch/jwt > /tmp/jwt.key
vault read -field=public  secret/mcp-orch/jwt > /tmp/jwt.pub

kubectl create secret generic mcp-orchestrator-secrets \
  --from-file=MCP_JWT_PRIVATE_KEY=/tmp/jwt.key \
  --from-file=MCP_JWT_PUBLIC_KEY=/tmp/jwt.pub \
  -n mcp-system

shred -u /tmp/jwt.key /tmp/jwt.pub

The chart's three-tier fallback in templates/secrets.yaml checks for an existing Secret first, so pre-creating the Secret externally takes precedence over auto-generation.

The auto-generated key lives in the Kubernetes Secret mcp-orchestrator-secrets. Anyone with get secrets permission in the orchestrator's namespace can retrieve it. RBAC discipline is the only enforcement boundary — see RBAC hardening below.

Important: the chart preserves the existing JWT keypair across helm upgrade (verified via Secret fingerprint comparison). Existing user sessions and service-account JWTs continue to work through chart upgrades. Manually rotating the keypair invalidates all of them and requires re-minting every service account through the API or UI.

PostgreSQL password

The chart auto-generates a 32-character random password for the internal PostgreSQL instance. PostgreSQL is internal-only (ClusterIP service, no external exposure) so this is appropriate for most installs without action.

Retrieve if needed (debugging, manual psql, backup tooling):

kubectl get secret -n mcp-system mcp-orchestrator-secrets \
  -o jsonpath='{.data.MCP_DB_PASSWORD}' | base64 -d ; echo

To pin a specific value (e.g. for off-cluster Postgres or shared credential management), pass --set secrets.dbPassword=<value> at install OR pre-create the Secret externally.

Admin bootstrap password

The chart auto-generates a 24-character random password for the seeded admin user on first install. Retrieve it from the Secret as documented above (with MCP_SEED_ADMIN_PASSWORD instead of MCP_DB_PASSWORD).

Treat this as a one-time bootstrap credential. Log in once, change the password via the UI, then rotate the Secret value to invalidate the bootstrap. The seed code only runs against an empty database, so changing the Secret value AFTER first install does not affect the running admin password — but it does invalidate any record of what the bootstrap value was, which is what you want.

RBAC hardening

All credentials above (TLS keys, JWT keys, DB password, admin password, SCIM tokens) live in Kubernetes Secrets in the orchestrator's namespace. Anyone with get secrets permission in this namespace can retrieve them in plaintext via kubectl. Restrict accordingly:

Audit who has Secret-read access in the orchestrator's namespace:

kubectl auth can-i --list --as=<user> -n mcp-system | grep secret

Enable encryption-at-rest in your cluster's etcd. This is a one-line cluster config; without it, anyone with etcd access reads Secrets in plaintext:

https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/
Audit cluster backups (Velero, etcd snapshots) for Secret content and ensure backup storage has equivalent or stricter access controls than the cluster itself.
For high-compliance environments (PCI, HIPAA, FedRAMP), consider:
- Sealed Secrets (encrypted Secret manifests in git)
- External Secrets Operator (Secrets fetched from external KMS at pod startup, never persisted to the cluster)
- Workload Identity (no Kubernetes Secret; the orchestrator gets its keys directly from cloud KMS via IAM)

These mitigations apply to all Secrets in the namespace, not just MCP Orchestrator's. They are standard Kubernetes hygiene.

Tracing (OpenTelemetry)

Off by default. Point the gateway at your OpenTelemetry collector and every call it handles becomes a span in your own tracing tool (Jaeger, Tempo, Datadog, Honeycomb, ...), tagged with Magertron's decision and the reason.

./install.sh --mode upgrade <your usual options> \
    --otel-endpoint otel-collector.observability.svc.cluster.local:4317 \
    --otel-sampling 10

install.sh option	Meaning	Default
`--otel-endpoint <host:port>`	Your collector's OTLP/gRPC address. Host and port only: no `http://`, port required. Port 4318 (OTLP/HTTP) will not work.	none (tracing off)
`--otel-sampling <percent>`	Share of calls traced, whole number 0-100	`100`

What you get:

Trace continuity. An incoming W3C traceparent is continued; the upstream server receives a traceparent naming the gateway's span.
Span names. The call: POST /mcp/mcp-prod/alpha-vantage/. The upstream call: upstream mcp-prod_alpha-vantage. Service name magertron-gateway.

Decision tags on the call's span:

Tag	Contents
`magertron.decision`	`allow` or `deny`
`magertron.stage`	What decided: `rbac`, `tool-approval`, `pdp`, `budget`, `mute`, `load-shed`, or `policy`
`magertron.reason`	Deny only: the reason the caller got with its 403 (max 256 chars)
`magertron.tool`	The MCP tool, REST operation or model
`magertron.subject`, `magertron.on_behalf_of`	Who called, and for whom (identity tags)
`magertron.plane`	`gateway`

Logs to traces. The gateway's JSON access log gains otel_trace_id.

Identity tags put user and service account names (often emails) into your tracing system. To leave them off, set otel.identityTags: false in your values file.

Tracing never changes a decision: spans are sent best-effort, off the request path, and a slow or missing collector never delays or fails a call. Only the gateway emits spans; the orchestrator's own work is not traced.

No collector yet? For a lab, Jaeger's all-in-one image receives OTLP/gRPC on 4317 and serves its UI on 16686 (in-memory; not for production).

Details: Operator's Guide Vol. 2, Chapter 37.

External policy (OPA)

Off by default. Magertron decides every call on identity, roles, tool approval and budgets. Business rules on the call's arguments ("no deposit over $2,000", "only operators may use topic network.commands") belong in a policy engine you own. After Magertron allows a call, it asks that engine, and a deny becomes a 403 with the engine's reason, written to the audit log. The policy can only deny: it never allows a call Magertron refused.

Bundled Open Policy Agent, with your policy and optional data file:

./install.sh --mode upgrade <your usual options> \
    --opa-policy ./magertron-policy.rego \
    --opa-data ./magertron-data.json \
    --pdp-namespaces "mcp-prod,netops"

Or your own PDP (OPA, Styra, ...), plain HTTP, in-cluster:

./install.sh --mode upgrade <your usual options> \
    --pdp-url http://opa.opa.svc.cluster.local:8181/v1/data/magertron/decision

install.sh option	Meaning	Default
`--enable-opa`	Deploy the bundled OPA with an allow-all policy	off
`--opa-policy <file.rego>`	Your policy for the bundled OPA (implies `--enable-opa`)	—
`--opa-data <file.json>`	Data for the policy; top-level key `allowances` is `data.allowances` (implies `--enable-opa`)	—
`--pdp-url <url>`	Use your own PDP instead; wins over `--enable-opa`	—
`--pdp-namespaces <list>`	`*`, or a quoted list with no spaces: `"mcp-prod,netops"`	`*`
`--pdp-fail <open\|closed>`	PDP unreachable: `closed` denies the call, `open` lets it through and audits `pdp:fail-open`	`closed`

Lists need 4.1.73 or later. Earlier installers passed the comma in --pdp-namespaces to Helm unescaped and the upgrade failed; use one namespace there.

Settings belong in the install command. kubectl set env MCP_PDP_* works for a test, but the next upgrade replaces it.

Magertron asks for data.magertron.decision and expects {"allow": false, "reason": "..."} (or "reasons": [...]) or true. Writing policies, the input document, verification and audit: Operator's Guide Vol. 2, Chapter 35.

Multi-replica HA

For deployments running more than one orchestrator replica — sizing guidance, configuration knobs, failover behavior, operational procedures, recovery from various failure modes — see the dedicated guide:

docs/HA-DEPLOYMENT.md

Quick start: set orchestrator.replicaCount=2 and envoy.replicaCount=2 in your values file, then helm upgrade --install. The chart enables leader election automatically when replicaCount > 1. The HA guide covers everything else.

Day-2 operations

Upgrade the chart

The bundled install.sh handles upgrades cleanly:

./install.sh --license /path/to/license.json --mode upgrade

--mode upgrade (default) preserves all data:

All Postgres data (users, service accounts, audit history)
All deployed MCP servers
The JWT signing keypair (so existing sessions / service-account JWTs survive)
The license secret

Manual upgrade equivalent:

helm upgrade mcp magertron/mcp-orchestrator \
  --namespace mcp-system \
  --reuse-values         # keeps your install-time --set values

CRDs are NOT upgraded by helm upgrade — that's intentional (helm treats CRDs as install-only because they can be destructive). If the chart ships new CRD versions, apply them manually:

kubectl apply -f https://magertron.com/charts/crds/

Schema migrations during upgrade

Some chart versions add new database columns. The inventory PostgreSQL runs its own migration scripts on pod boot; those are automatic.

The orchestrator PostgreSQL is initialized from a postgres-init ConfigMap that only runs on first PV creation. For existing installs, chart upgrades that add new columns to the orchestrator's tables (e.g. the users.email column added in v2.4.x) require a manual ALTER TABLE step. Per-version migration steps are documented in UPGRADING.md.

A SchemaGuard-style migration runner for the orchestrator (matching what inventory already does) is on the near-term roadmap; once it ships, manual ALTERs will no longer be needed for column additions.

Roll pods after a config change

kubectl rollout restart deployment/mcp-orchestrator -n mcp-system
kubectl rollout restart deployment/mcp-orchestrator-envoy -n mcp-system

Uninstall

helm uninstall mcp -n mcp-system

This removes Deployments, Services, Secrets, ConfigMaps. The PersistentVolumeClaims for both PostgreSQL instances are preserved (helm.sh/resource-policy: keep annotation) — your data survives uninstall. To also drop the databases:

kubectl delete pvc -l app=mcp-orchestrator-postgres -n mcp-system
kubectl delete pvc -l app=mcp-orchestrator-inventory-postgres -n mcp-system
kubectl delete namespace mcp-system

CRDs and the namespaces you created (mcp-prod, etc.) are also preserved. Delete them manually if you want a clean slate:

kubectl delete crd mcproutes.mcp.io
kubectl delete namespace mcp-prod mcp-staging mcp-dev

Troubleshooting

Orchestrator pods stuck in Init

Most common cause: PostgreSQL isn't ready yet. The orchestrator waits for Postgres readiness before starting. Check the Postgres pod:

kubectl logs mcp-orchestrator-postgres-xxxxx -n mcp-system

If you see initdb errors, the schema init scripts may have failed. For a fresh-slate fix (data loss acceptable on first install):

./install.sh --license /path/to/license.json --mode reinstall

You'll be prompted to type 'destroy' to confirm.

Envoy pod won't start — "cannot load TLS certificate"

Check that the mcp-envoy-tls Secret exists:

kubectl get secret mcp-envoy-tls -n mcp-system -o yaml

If it's missing, your install didn't pass --set-file tls.cert=... AND you also disabled auto-generation. Re-run install with a valid cert+key OR with tls.create=true (the default) to let the chart self-sign.

Inventory pod restarts a couple of times on first install

Expected. The inventory service waits for its Postgres to accept connections at startup. Two or three restarts in the first ~30 seconds is normal; it stabilizes after Postgres is ready. If it's still restarting after 2 minutes, check the inventory logs:

kubectl logs -n mcp-system -l app=mcp-orchestrator-inventory --tail=100

Cannot log in — "Invalid credentials"

The admin seed runs only against an empty database. If you've reinstalled the chart against an existing Postgres volume (PVC retained), the original admin user persists and the chart's auto-generated bootstrap password in the Secret is unused. Login with the original password (whatever was generated or set on first install) — that's the one in the database.

If you've genuinely lost it, the safe recovery path is to truncate the users table and let the orchestrator's seed code re-run against an empty database, picking up the current Secret value as the new admin password:

PG_POD=$(kubectl get pods -n mcp-system -l app=mcp-orchestrator-postgres \
  -o jsonpath='{.items[0].metadata.name}')

# WARNING: this removes ALL users — including any operators you've added,
# any SSO-provisioned users, and any custom RBAC. For a non-destructive
# reset, change the password via psql instead (you'd need to compute a
# bcrypt hash externally — out of scope here).
kubectl exec -n mcp-system $PG_POD -- psql -U mcp -d mcp_platform -c \
  "TRUNCATE users CASCADE;"

kubectl rollout restart deployment/mcp-orchestrator -n mcp-system

After restart, retrieve the admin password from the Secret as documented in the post-install notes. The seed code will have re-created admin with that value.

SCIM endpoints return HTML instead of JSON

This happens when your IdP hits a path the orchestrator doesn't handle — for v1, that's Groups (/scim/v2/Groups). The orchestrator returns a proper SCIM 404 with application/scim+json content-type. If you see HTML, you're on a version older than v1.2.0 — upgrade the chart.

Okta SCIM "Test API Credentials" fails with 401

Verify you picked the Bearer Token variant of Okta's SCIM test app, not the Header Auth variant. The Header Auth variant sends the raw token with no Bearer prefix, which our orchestrator rejects. The Bearer Token variant is the correct choice for our implementation.

SCIM meta.location points at localhost

You forgot to set orchestrator.env.apiPublicUrl at install time. Most SCIM operations still work (IdPs use the Base URL they were configured with for actual requests), but meta.location in response bodies will be cosmetically wrong. Fix with:

helm upgrade mcp magertron/mcp-orchestrator \
  --namespace mcp-system --reuse-values \
  --set orchestrator.env.apiPublicUrl=https://mcp.yourcompany.com

kubectl rollout restart deployment/mcp-orchestrator -n mcp-system

OIDC login redirects to localhost

Same root cause as above — apiPublicUrl not set. Fix the same way, then go to Identity Providers and click through the OIDC provider's details to re-trigger the redirect URL calculation. You may need to delete and re-create the provider if the cached URL persists.

Webhooks aren't firing for expiry reminders

Three things to check:

At least one webhook is configured. Admin → Webhooks tab. The expiry-reminder worker fires whichever webhooks are configured.
The orchestrator is the leader. Reminders are leader-only — only one of the two orchestrator replicas fires the cron. Check both pods' logs; one should show ExpiryReminderWorker: started, and that pod logs the actual fires.

Network reachability. Test the webhook URL from inside the cluster:

kubectl run -it --rm curl --image=curlimages/curl -- \
  curl -X POST https://your-webhook-url/test

Failed deliveries also emit audit events (WebhookDeliveryFailed), visible in the Audit Log tab.

Network policies block legit traffic

The chart creates NetworkPolicies on MCP server namespaces to restrict ingress to the mcp-system namespace only. If you're running a CNI that doesn't fully support NetworkPolicy (Flannel in default mode is a common offender), pods in MCP namespaces won't be reachable.

Options:

Switch to a CNI that supports NetworkPolicy (Calico, Cilium)
Disable network policies: --set networkPolicy.enabled=false

"Still showing Free tier after applying license"

Two things to check:

Did you restart the orchestrator pods after creating the Secret?

kubectl rollout restart deployment/mcp-orchestrator -n mcp-system

Is the Secret actually mounted? Exec into the orchestrator and check:
```
kubectl exec -it mcp-orchestrator-xxxxx -c orchestrator -n mcp-system -- \
  ls -la /etc/mcp-license/
```
You should see license.json. If not, the Secret name probably doesn't match license.secretName (default: mcp-license).

Full values reference

See values.yaml for the full configuration surface. Key knobs you'll commonly override:

Key	Default	Purpose
`orchestrator.replicaCount`	2	Orchestrator replicas. HA via leader election + LISTEN/NOTIFY fanout. See HA-DEPLOYMENT.md.
`orchestrator.env.apiPublicUrl`	`""`	Public URL for SCIM/OIDC
`orchestrator.env.ssoAutoProvisionDomains`	`""`	SSO JIT allowlist
`orchestrator.env.leaderElectionEnabled`	`""` (auto)	Auto-enabled when replicaCount > 1; force on/off with `"true"`/`"false"`
`orchestrator.env.useNotifyFanout`	`"true"`	Postgres LISTEN/NOTIFY for sub-millisecond xDS fanout. Default ON since v2.0; turn off only for debugging.
`envoy.replicaCount`	2	Envoy replicas (default; HPA scales to 10)
`loadBalancer.provider`	`cloud`	`cloud`/`metallb`/`nodeport`/`existing`
`postgresql.storage.size`	`10Gi`	Orchestrator DB volume size
`inventory.postgresql.storage.size`	`10Gi`	Inventory DB volume size
`postgresql.storage.storageClassName`	(default)	Pick a storage class if needed
`namespaces`	`[mcp-prod, mcp-staging, mcp-dev]`	MCP server namespaces to create
`networkPolicy.enabled`	`true`	NetworkPolicy creation
`secrets.dbPassword`	(auto)	PostgreSQL password
`license.secretName`	`mcp-license`	Name of the license Secret
`retention.auditEventsDays`	90	Daily prune threshold for `audit_events`
`retention.runtimePolicyAuditDays`	30	Daily prune threshold for `runtime_policy_audit`
`retention.cronExpression`	`"0 3 * * *"`	When the retention worker fires (UTC)
`otel.endpoint`	`""` (off)	OTLP/gRPC collector as `host:port`. Set by `--otel-endpoint`.
`otel.samplingPercent`	100	Share of calls traced, 0-100. Set by `--otel-sampling`.
`otel.serviceName`	`magertron-gateway`	`service.name` on the gateway's spans
`otel.identityTags`	`true`	Put `magertron.subject` / `magertron.on_behalf_of` on spans
`opa.enabled`	`false`	Deploy the bundled OPA. Set by `--enable-opa`.
`orchestrator.pdp.url`	`""` (off)	Policy decision point URL (plain HTTP). Set by `--pdp-url` or the bundled OPA.
`orchestrator.pdp.namespaces`	`*`	Namespaces sent to the PDP: `*` or `ns1,ns2`. Set by `--pdp-namespaces`.
`orchestrator.pdp.fail`	`closed`	PDP unreachable: `closed` denies, `open` allows and audits
`orchestrator.pdp.timeoutMs`	200	Wait for the PDP per call, 20-5000 ms

Appendix: Cloud Kubernetes (EKS / GKE / AKS)

Works with default settings. Your cloud provider's LoadBalancer controller handles the public IP automatically. You still bring your own cert and JWT keys:

helm install mcp magertron/mcp-orchestrator \
  --namespace mcp-system --create-namespace \
  --set orchestrator.env.apiPublicUrl=https://mcp.yourcompany.com \
  --set-file secrets.jwtPrivateKey=jwt.key \
  --set-file secrets.jwtPublicKey=jwt.pub \
  --set-file tls.cert=tls.crt \
  --set-file tls.key=tls.key

Then point mcp.yourcompany.com DNS at the LoadBalancer IP:

kubectl get svc mcp-orchestrator-envoy -n mcp-system
# Look at EXTERNAL-IP column

Cert-manager is a common choice for automating TLS on cloud clusters. Integration is straightforward but out of scope for this guide.

Support

For issues, check the orchestrator logs first:

kubectl logs -n mcp-system -l app=mcp-orchestrator -c orchestrator --tail=100

File bug reports with the log output attached.

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
.well-known		.well-known
blog		blog
docs		docs
helm/orchestrator		helm/orchestrator
images		images
mcpctl		mcpctl
scripts		scripts
vscode		vscode
.gitignore		.gitignore
LICENSE		LICENSE
Magertron-Getting-Started-Guide.pdf		Magertron-Getting-Started-Guide.pdf
README.md		README.md
agent-developers-guide.pdf		agent-developers-guide.pdf
install.sh		install.sh
magertron-allowances.json		magertron-allowances.json
magertron-allowances.rego		magertron-allowances.rego
region-bundle.sh		region-bundle.sh
restore.sh		restore.sh
schema-preflight.sh		schema-preflight.sh

Latest commit

History

Folders and files

Repository files navigation

MCP Orchestrator — Helm Chart

Quickstart

Installing the CLI

What you're installing

Prerequisites

1. A Kubernetes cluster

2. Helm 3.x and kubectl

3. A TLS certificate for your hostname

4. JWT signing keypair

5. metrics-server (optional but recommended)

6. License file (optional — Enterprise tier only)

Choose your ingress path

NodePort (recommended for single-box / small installs)

MetalLB (multi-node bare-metal with LAN IP pool)

Existing ingress controller (nginx-ingress, Traefik, etc.)

Cloud LoadBalancer (EKS / GKE / AKS)

Install (manual)

Add the helm repo (one time)

Minimum viable install (Free tier, self-signed cert, NodePort)

Production install (Free tier)

Production install (Enterprise tier with license)

What the flags mean

After install

1. Watch the rollout

2. Access the dashboard

3. Configure SSO (Enterprise — optional)

4. Configure SCIM (Enterprise — optional)

5. Configure webhooks for expiry reminders (recommended)

Service Account Lifecycle

Identity linking (users + service accounts)

Audit log & retention

Production hardening

TLS certificate

JWT signing key

PostgreSQL password

Admin bootstrap password

RBAC hardening

Tracing (OpenTelemetry)

External policy (OPA)

Multi-replica HA

Day-2 operations

Upgrade the chart

Schema migrations during upgrade

Roll pods after a config change

Uninstall

Troubleshooting

Orchestrator pods stuck in Init

Envoy pod won't start — "cannot load TLS certificate"

Inventory pod restarts a couple of times on first install

Cannot log in — "Invalid credentials"

SCIM endpoints return HTML instead of JSON

Okta SCIM "Test API Credentials" fails with 401

SCIM meta.location points at localhost

OIDC login redirects to localhost

Webhooks aren't firing for expiry reminders

Network policies block legit traffic

"Still showing Free tier after applying license"

Full values reference

Appendix: Cloud Kubernetes (EKS / GKE / AKS)

Support

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages