Step-ca-issued client certs + mTLS at a GCP HTTPS LB replace the
operator's hand-dropped vault.yaml with a brokered
Secret Manager fetch. Same primitive on macOS hardware (m4/r8) and
Linux hardware (talos/netperf).
Every worker needs a /var/root/vault.yaml (macOS) or
/root/vault.yaml (Linux) containing the per-role taskcluster
tokens, cltbld credentials, livelog secret, etc. Puppet reads it via
Hiera; without it, run-puppet.sh won't even start.
Today: an operator looks the role's entry up in 1Password, saves it locally, SCPs it onto the host. Every wipe, every replacement, every new role assignment — same dance.
Costs: operator must be online + have 1P + SSH + the right file. Easy to mistype a role and ship the wrong secret. Doesn't survive automation. Becomes the chokepoint when a fleet of M4s or r8s gets EACS'd in a batch.
The vault.yaml fetches itself. A host that's been MDM-enrolled (macOS) or operator-attested via SCEP enrollment (Linux) holds a step-ca-issued client cert; the cert's SPIFFE URI binds it to a single puppet role; the broker hands back the matching secret on demand.
Provisioning — EACS a Mac mini. SimpleMDM pushes profiles + the signed bootstrap PKG; the on-network runner auto-mints the SecureToken over an on-host ssh login (DEP skips Setup Assistant, and Apple only grants the first token interactively), then the bootstrap fetches vault.yaml, puppet provisions, and the worker registers in Taskcluster. Fully hands-off — one Hangar click.
Re-provisioning — Same flow. Worker dies, replace hardware,
EACS the new one (or run deliver_linux.sh for Linux), it
bootstraps into the same role automatically. No "find the right vault.yaml
again" step.
Operationally: the bottleneck moves from "a human with the right files" to "the device has a valid cert." That's a known-quantity infrastructure check, not a tribal-knowledge handoff.
forge.relops.mozilla.com validates every client cert chain via the same Trust ConfigSame SAN URI shape across all certs:
spiffe://relops.mozilla/host/<CN>/role/<puppet-role>
Broker URL-decodes (handles Mac%20mini), parses host + role, enforces role-match against URL path. Cert chain validation already happened at the LB.
| macOS | Linux | |
|---|---|---|
| Cert delivery | SimpleMDM pushes SCEP profile → mdmclient enrolls into System keychain | Operator's mint-scep-cert.sh → sscep → scp to worker |
| Where the key lives | macOS System keychain — ACL-restricted to OS network stack (configd, nehelper, NEIKEv2Provider) | /etc/relops-bootstrap/key.pem on disk — root:root 0600 |
| TLS client invocation | CURL_SSL_BACKEND=securetransport curl --cert "<CN>"(SecureTransport looks up identity by Subject CN) |
curl --cert cert.pem --key key.pem(plain PEM, LibreSSL backend) |
| vault.yaml destination | /var/root/vault.yaml |
/root/vault.yaml |
| Run-puppet trigger | signed bootstrap PKG installs com.mozilla.m4-bootstrap LaunchDaemon → loops run-puppet via ssh-to-localhost until safari semaphores fire |
bootstrap_linux.sh runs puppet apply directly in line |
| Operator gate | SimpleMDM ops + EACS click; one VNC/SSH login to seed BST if needed | Operator's gcloud auth (reads SCEP challenge from Secret Manager) + SSH to worker |
Time flows top to bottom. Same flow for macOS and Linux — the only difference is whether the cert is presented from the keychain (macOS via SecureTransport curl) or from a PEM file on disk (Linux via plain curl).
Auth happens in two stages: the LB does the cryptographic check (cert chain validation) and the broker does the authorization check (role-bound to SPIFFE URI). Splitting these lets the broker stay a thin Python service — it doesn't touch a cryptography library, parses headers + reads Secret Manager. Failure modes: missing/bad cert → 401 at the broker; cert valid but wrong role → 403 with the role mismatch in the body. Every fetch (success or denial) is written to Cloud Audit Logs with the cert serial.
The LB's Trust Config contains the step-ca root + intermediate PEMs. Any leaf
step-ca issues — to any platform — chains correctly. The LB validates the
client cert chain on every mTLS request to forge.relops.mozilla.com.
Single trust anchor for the whole fleet.
vault-* Secret Manager containers are empty (0 versions) — an enrollment would mint a cert but the fetch has nothing to return. Not usable until the secrets are populated. bootstrap_linux.sh + deliver_linux.sh in ronin_puppet PR #1253; pending a real-host test before merge. (macOS is the proven, live path — see the reprovision flow diagram.)Invoke-RestMethod -Certificate or curl.exe for the mTLS step. Need to find where the existing Windows bootstrap.ps1 lives (not in ronin_puppet).The vault.yaml itself didn't get any more or less sensitive. What changed is who's allowed to put one on a worker. Before: anyone with the 1Password vault entry and SSH access. After: only an authenticated MDM-enrolled (or operator-attested) host with a step-ca-issued, per-role-bound client cert. Every fetch is logged in Cloud Audit with the cert's serial number. The operator's job goes from "SCP this file" to "click EACS in SimpleMDM" (macOS) or "run one command" (Linux).
Everything is open source inside Mozilla — the broker code, the terraform, the operator-side helpers, the signed bootstrap PKG. Per-role rollout (r8, staging, catalina, the rest of the Linux roles) is mechanical follow-up work; happy to pair on it if you've got a role you want to migrate.