How relops-bootstrap works, and why we trust it

1The problem in plain terms

Every Mozilla RelOps worker — macOS, Linux, Windows — runs masterless Puppet at first boot to provision itself: install packages, set up users, register the Taskcluster generic-worker daemon, and so on. Puppet pulls its per-host configuration through Hiera, a data-lookup mechanism that reads YAML files. The most sensitive of those is data/secrets/vault.yaml, which contains the Taskcluster access_token for that role's worker pool, the cltbld user's password (hashed and unhashed), the Bugzilla API key, the livelog secret — all the stuff a worker needs to actually do work.

Historically the operator delivered this file by hand. They'd open 1Password, find the role's vault entry, save the YAML to their laptop, SCP it onto the worker at /var/root/vault.yaml (macOS) or /root/vault.yaml (Linux). The Puppet bootstrap script copied it into ./data/secrets/vault.yaml within the cloned ronin_puppet repo, and the run proceeded.

# SIP had to be disabled on the worker first via Recovery Mode
# (a separate, manual, hands-on hardware step we won't reproduce here).
# Then on the operator's laptop, with a checkout of ronin_puppet:

git clone ronin_puppet
# add the new host to the role's inventory (e.g. macmini-m4.yaml)

export VAULT_ADDR="https://vault.relops.mozops.net:8200"
vault login -method=oidc role="hiera_admin"
export VAULT_TOKEN=$(cat ~/.vault-token)

bolt plan run deploy::apply_no_verify \
    --targets macmini-m4-129.test.releng.mdc1.mozilla.com \
    noop=false --native-ssh

sudo mkdir /var/tmp/semaphore
sudo touch /var/tmp/semaphore/run-buildbot
reboot

# Then check Taskcluster's worker-pool UI for the new worker to appear.
# If it didn't, tail the worker stderr:
sudo tail -f /opt/worker/logs/stderr.log

This worked for ~10 hardware refreshes a year. It does not work for fleet-scale operations:

• It serializes on the operator. Batch-EACS'ing 20 M4s after a hardware refresh becomes a 20-step manual loop.
• It's mistake-prone. Easy to ship the wrong role's secrets to the wrong host. The error doesn't surface until the worker registers as the wrong pool.
• It's brittle. Operator must have 1Password, must have laptop SSH access to MDC1, must have this file ready at the moment the host needs it.
• It can't survive automation. Whatever tool drives EACS or kickstart needs the secret somewhere, and "the operator's laptop" is not somewhere.

So: we want a way for a worker, knowing only its own role, to fetch its vault.yaml automatically. The constraint that makes this interesting is that "a worker" is not a trustworthy actor — anyone can stand up a Mac mini and claim it's gecko_t_osx_1500_m4. The worker needs to prove it is that role.

2The mental model in three sentences

1. The host's MDM enrollment (macOS) or operator-attested SSH bootstrap (Linux) gets it a client certificate from our own Certificate Authority, with the host's puppet role bound into the cert's Subject Alternative Name as a SPIFFE URI.
2. The host uses that cert in a mutual-TLS request to a brokered service at forge.relops.mozilla.com; the Google-managed HTTPS Load Balancer validates the cert chain against our CA, and a Cloud Run broker parses the SAN and serves back the role-scoped vault.yaml from Secret Manager.
3. Every layer of this is designed so that compromising any single component does no worse damage than the operator-laptop model already does — and several layers do strictly better.

The remaining sections elaborate each of those. We'll work bottom-up: trust foundations first, then a single request walked through step by step, then platform-specific details, then where it can fail.

3Trust: what is trusted by whom, and why

Before talking about how the system works, it's worth being explicit about what we're trusting at each layer. "Trust" here means: if this thing goes bad, what's the consequence?

The certificate authority

We run our own step-ca instance on a GCE VM in the relops-bootstrap GCP project. It maintains a two-level hierarchy: a self-signed root CA (10-year validity) and an intermediate CA signed by the root (1-year validity, used as the actual signing CA for all worker leaf certs). This is the standard PKI pattern — keeping the root offline (or, in our case, behind a separate password-protected step-ca config) limits the blast radius of a compromise of the day-to-day-active intermediate.

step-ca's root and intermediate public PEMs are baked into:

• The repo at terraform/trust/{root,intermediate}.pem — these are public certs, no private key material.
• The Trust Config that the load balancer uses to validate incoming client certs.
• The system trust store of each worker, via the SCEP profile's PKCS#1 payload (macOS) or via curl's default CA bundle for the server-side check (Linux is already trusting forge.relops.mozilla.com's Google-managed cert via the public CA roots).

The private keys for the root and intermediate live on the step-ca VM, encrypted with a password loaded from /home/step/.step/secrets/password (which is itself only readable by the step user). The VM has no public-internet ingress beyond the HTTPS LB endpoint; admin access is via Google's Identity-Aware Proxy (IAP) tunneled SSH, which requires the operator's Google identity + a project-level IAM grant.

The SCEP challenge — the enrollment gate

Worker certs are issued via SCEP, the Simple Certificate Enrollment Protocol (RFC 8894). SCEP is what Apple's mdmclient already speaks natively when an MDM-pushed profile contains a com.apple.security.scep payload, which is why we use it instead of inventing something custom.

A SCEP enrollment request is gated on a shared secret called the "challenge." The challenge is set on each provisioner (one per puppet role in our setup) and is included in the CSR's challengePassword attribute. step-ca verifies the challenge matches before signing. Our challenge is stored in GCP Secret Manager (step-ca-scep-challenge) and the operator reads it via their gcloud authentication when delivering a Linux host (or it's embedded in the SimpleMDM SCEP profile for Mac, which SimpleMDM holds securely on its servers).

The Trust Config — the LB's job

The HTTPS Load Balancer at forge.relops.mozilla.com is configured with a Trust Config (a GCP Certificate Manager resource) that contains only our step-ca root cert as a trust anchor, plus the intermediate cert as a known signer. The LB's TLS policy is set to ALLOW_INVALID_OR_MISSING_CLIENT_CERT: meaning request a client cert during the TLS handshake, but don't require one to terminate. This lets us serve the /scep/* endpoints (which clients hit before they have a cert, to enroll) on the same hostname as /secret/*.

When a client does present a cert, the LB:

1. Asks for the cert during the TLS handshake's CertificateRequest message.
2. Receives the client's cert chain in the Certificate message.
3. Verifies the chain against the Trust Config: leaf signed by intermediate, intermediate signed by root, root matches one of our trust anchors. If verification fails, the LB does not reject the connection (because of the policy mode) — instead it forwards the request with an X-Client-Cert-Chain-Verified: false header.
4. Validates the cert hasn't expired (notBefore ≤ now ≤ notAfter).
5. Forwards the request to the broker, injecting headers: X-Client-Cert-Present, X-Client-Cert-Chain-Verified, X-Client-Cert-Leaf (base64-encoded PEM of the leaf), X-Client-Cert-Serial-Number, and several others.

The broker's role

vault-broker is a small FastAPI service running on Cloud Run. It reads the LB-forwarded headers, decodes the leaf cert PEM, parses its SubjectAltName extension for a URI of the form spiffe://relops.mozilla/host/<hostname>/role/<role>, and:

• Verifies X-Client-Cert-Present: true and X-Client-Cert-Chain-Verified: true (otherwise → 401).
• Verifies the cert has a SPIFFE URI in the expected shape (otherwise → 401).
• URL-decodes the URI components — important because Apple's SCEP enrollment URL-encodes spaces (a cert subject "Mac mini" produces SAN URI ".../host/Mac%20mini/..."), and the LB's built-in SPIFFE parser silently drops malformed-looking URIs. By parsing the leaf cert ourselves we sidestep that.
• Checks the role in the URI matches the URL path's role (otherwise → 403 with the role mismatch named in the body).
• Reads vault-<role> from Secret Manager via the broker's service account (which has per-secret IAM bindings — it can read only the vault-* secrets, nothing else in the project).
• Returns the secret content as text/yaml.

Every accept and reject decision lands in Cloud Audit Logs with the cert serial number, the requested role, the requesting IP, and the bytes returned. This is the audit trail.

4Anatomy of one vault.yaml fetch

Let's walk through a single request from the worker's perspective, in detail. Assume the worker has already been enrolled — if you want the prequel (what EACS and DEP re-enrollment actually do to get here), see what EACS actually does.

Step 0: the worker has a cert

On macOS, this means a SCEP-issued identity in the System keychain, with a private key whose ACL allows the OS network stack to sign with it. On Linux, this means a PEM cert + PEM key at /etc/relops-bootstrap/{cert,key}.pem owned by root with 0600 permissions. The cert's Subject Common Name is whatever the SCEP profile's subject renders to — on macOS that's the device %ComputerName% (frequently the default "Mac mini", so the CN is not a unique hostname), on Linux the host FQDN. The CN is only used to locate the identity in the keychain; the authoritative field is the SAN's SPIFFE URI, which binds the cert to exactly one puppet role.

The cert and key were minted by step-ca's SCEP provisioner at enrollment time. The role-binding is enforced by the provisioner's x509 template — we have one provisioner per role, and each provisioner's template hardcodes "spiffe://relops.mozilla/host/{{ .Subject.CommonName }}/role/<role>" into the SANs. There is no way for the worker to request a cert with a role it shouldn't have — the choice of role is determined by which provisioner it enrolls against, and the provisioner's URL path is delivered via MDM profile (macOS) or specified by the operator (Linux). The worker has no input on it.

Step 1: TLS ClientHello

The worker initiates a TLS 1.3 handshake with forge.relops.mozilla.com:443. The client sends a ClientHello advertising the cipher suites it supports and the SNI of the hostname.

The LB responds with the server's certificate (a Google-managed cert whose chain validates against the public CA roots already in every worker's system trust store — this is the same way HTTPS works everywhere on the internet) plus, critically, a CertificateRequest message telling the client "I want your cert too." The CertificateRequest also (sometimes) includes the list of acceptable CA distinguished names. In our case, GCP's LB sends an empty acceptable-CA list, which is a subtle point that bit us in development.

Step 2: client cert presentation + signature

The worker responds with its own Certificate message containing its leaf cert (and optionally the intermediate, though our workers send only the leaf — the LB has the intermediate in its Trust Config). It then signs a hash of the entire handshake transcript with its private key, and sends that signature in a CertificateVerify message. This proves the client actually possesses the private key matching the cert it just sent (otherwise it could just replay someone else's cert).

This signing step is where the macOS keychain ACL gets involved. The signing happens inside whichever process is doing the TLS handshake. On macOS, when we use CURL_SSL_BACKEND=securetransport curl --cert "<CN>", the handshake runs inside Apple's SecureTransport library, which is permitted by the keychain ACL to sign with our SCEP-installed key. (We explore why this matters in detail in section 5.) On Linux, the signing happens in LibreSSL / OpenSSL, reading the private key from the PEM file — straightforward, no OS gatekeeping.

Step 3: LB chain validation

The LB verifies:

1. The leaf cert is signed by a key whose corresponding cert is in our Trust Config (the step-ca intermediate).
2. The intermediate is signed by a trust anchor in our Trust Config (the step-ca root).
3. The signature in CertificateVerify validates against the leaf cert's public key over the handshake transcript — i.e. the client actually has the private key.
4. The cert's validity window contains "now."

If any of these fails, the LB does not tear down the connection — its policy is "allow invalid or missing." Instead, it records the failure and prepares to inject X-Client-Cert-Chain-Verified: false later. If validation succeeds, it prepares to inject the cert's metadata as request headers.

Step 4: TLS handshake completes, HTTP begins

Both sides exchange Finished messages. The TLS session is now established and encrypted. The worker sends an HTTP/2 request:

GET /secret/gecko_t_osx_1500_m4 HTTP/2
Host: forge.relops.mozilla.com
User-Agent: curl/8.7.1
Accept: */*

Step 5: LB → broker, with headers injected

The LB rewrites the request as it forwards to Cloud Run. The rewritten request looks roughly like:

GET /secret/gecko_t_osx_1500_m4 HTTP/1.1
Host: vault-broker-internal
X-Forwarded-For: 63.245.209.101
X-Client-Cert-Present: true
X-Client-Cert-Chain-Verified: true
X-Client-Cert-Serial-Number: 65:20:86:3c:23:5e:fa:0a:c8:5d:47:70:1a:74:43:75
X-Client-Cert-SHA256: TGt3g713vVfsEadMuNovHAe5460Y/py52mXy+6jkJxw
X-Client-Cert-Leaf: MIIDSDCC...<base64 PEM of leaf cert>...DEhQ==
X-Client-Cert-SPIFFE: <possibly empty if LB couldn't parse>
X-Client-Cert-Subject-DN: MDYxEDAOBgNV...<base64 DN>...

This is the load-bearing invariant of the whole design, so it's worth stating precisely. The broker does not re-verify the cert chain — it trusts the LB's X-Client-Cert-Chain-Verified header. Two facts make that safe, and a security review should confirm both:

1. The LB overwrites the header names it injects. The custom_request_headers on the backend service (terraform/lb.tf) set each X-Client-Cert-* value the broker reads, overwriting any inbound value for those names — so a client cannot present a forged X-Client-Cert-Chain-Verified: true.
2. Nothing can reach the broker except through the LB. Cloud Run ingress is INGRESS_TRAFFIC_INTERNAL_LOAD_BALANCER (live value internal-and-cloud-load-balancing), so a direct request to the Cloud Run URL is refused at the platform layer — even though allUsers holds roles/run.invoker (the standard grant for an LB-fronted service, gated entirely by that ingress setting).

Together these mean an attacker can't bypass the LB and forge the headers. The corollary a reviewer should note: if that ingress setting were ever relaxed, the header trust would become an authentication bypass. That single setting is what earns the broker the right to trust its headers.

Step 6: broker parses and authorizes

Inside the broker, in broker/app/auth.py:

# 1. The basic LB-said-yes checks
if cert_present != "true": raise AuthError("no client cert")
if chain_verified != "true": raise AuthError("chain not verified")

# 2. Decode the base64-PEM leaf into a cryptography.x509.Certificate
leaf = x509.load_pem_x509_certificate(base64.b64decode(cert_leaf))

# 3. Extract SAN, find a URI matching our trust domain prefix
san = leaf.extensions.get_extension_for_oid(SUBJECT_ALTERNATIVE_NAME)
for uri in san.value.get_values_for_type(x509.UniformResourceIdentifier):
    decoded = urllib.parse.unquote(uri)  # handle Mac%20mini → "Mac mini"
    if decoded.startswith("spiffe://relops.mozilla/host/"):
        hostname, _, role = decoded[len(prefix):].split("/", 2)
        break

# 4. Role-match against URL path
if role != url_path_role: raise AuthError("role mismatch")

If everything matches, the broker calls read_secret(f"vault-{role}") via the Google Secret Manager Python client, and returns the content as the response body with content-type text/yaml.

Step 7: response, write to disk, puppet runs

The broker writes a 200 OK back to the LB, the LB tunnels it back through the TLS session to the worker. The worker (in our bootstrap script) atomically writes the body to /var/root/vault.yaml (macOS) or /root/vault.yaml (Linux) at 0600 root:root. From here, the rest of the puppet bootstrap is unchanged from how it worked before — Hiera reads the file, populates the per-role lookups, puppet applies, worker is provisioned.

5macOS — the keychain ACL story

Of the two platforms we support, macOS is the harder one to make this work on, and the reasons are educational in their own right. The short version: macOS treats keychain-stored cryptographic keys as the OS's responsibility, not the application's, and the OS imposes restrictions that took us several days to fully understand. The mechanism we landed on works by going through the OS's own network stack rather than fighting it.

Where the key actually is

When SimpleMDM pushes a com.apple.security.scep profile to an M4 mini, mdmclient on the device:

1. Generates an RSA-2048 keypair, placing the private key in the System keychain at /Library/Keychains/System.keychain.
2. Builds a CSR with the subject and challenge from the profile.
3. Sends the CSR to the SCEP URL from the profile, signing with the SCEP encryption cert (provided in the SCEP server's GetCACert response).
4. Receives a signed cert back, places it in the System keychain, paired with the private key as a SecIdentity.

Importantly, the private key never touches the filesystem in plaintext. It's stored in the System keychain's encrypted database file, and reading or using it requires going through the macOS Security framework.

The ACL on the key

Every keychain key has an access-control list. Apple's SCEP-enrollment flow installs the key with three ACL entries (which we discovered by dumping the keychain on a real worker):

Read carefully: the only ACL entry that grants sign authorization is the second one, and it grants it only to a hardcoded list of Apple's own networking-stack helpers. Our custom code — a Swift binary we wrote, a Python script, even a shell command running as root from a LaunchDaemon — is not in this list and never can be. When we tried to call SecKeyCreateSignature on this key from any of those contexts, macOS returned errSecInteractionNotAllowed (-25308) with the underlying CSSM error being CSSMERR_CSP_NO_USER_INTERACTION — i.e., "I would need to show a UI prompt to authorize this, and I can't, so no."

Working with the OS instead of against it

Once we accepted that we cannot sign with this key from our own process, the question became: which process can? The ACL lists Apple's network-stack helpers. Anything that initiates a TLS connection through those helpers can effectively sign with the key during the handshake.

macOS's curl binary, since macOS 12 or so, ships with two TLS backends compiled in: LibreSSL (the default) and SecureTransport (Apple's own TLS implementation, deprecated for new code but still present and functional). When you invoke curl --cert "<Common Name>" with no key file, curl assumes the argument is a keychain identity reference — but only when SecureTransport is the active backend. The CURL_SSL_BACKEND=securetransport environment variable flips it.

With SecureTransport active, curl asks the macOS Security framework to find an identity with that Common Name, get a SecIdentityRef, and use it during the TLS handshake. The handshake — including the CertificateVerify signature — runs inside SecureTransport itself, which goes through the network-stack helpers that are in the keychain ACL. Result: the cert gets used, the key never leaves the keychain, no UI prompt, no PKCS#12 extraction.

This is, in retrospect, exactly the affordance Apple designed for. We were just slow to find it because we kept trying to make our own code sign things, and the documentation for the SecureTransport curl backend is sparse-bordering-on-undocumented. (We have notes on this for posterity in scripts/INSTALL-on-worker.md.)

6Linux — same primitive, different constraints

Linux has no MDM. There is no daemon that runs at first boot and enrolls the device into a management plane that holds an SCEP profile. We could install one — mosquitto-style or chef-client-style — but Mozilla RelOps doesn't, and shipping a new agent for one bootstrap problem is the wrong tradeoff.

Instead, on Linux, the operator is the trust anchor at enrollment time. The operator's laptop runs provisioners/linux/deliver_linux.sh, which:

1. Fetches the SCEP challenge from GCP Secret Manager via the operator's gcloud auth.
2. Invokes scripts/mint-scep-cert.sh (from a checkout of relops-bootstrap) which:
   • Generates an RSA keypair locally with openssl.
   • Builds a CSR including the challenge in the challengePassword attribute.
   • Runs sscep enroll against the appropriate per-role SCEP provisioner endpoint.
3. SCPs the resulting cert + key files to /etc/relops-bootstrap/ on the target worker, as cert.pem and key.pem at 0600 root:root.
4. SCPs the bootstrap script and invokes it. The bootstrap script's fetch_vault_yaml_via_mtls function detects the cert and key, runs curl --cert ... --key ... against forge.relops.mozilla.com, lands /root/vault.yaml, then proceeds with the existing puppet bootstrap flow.

The operator's gcloud authentication is the gate. To run this flow, you need (a) GCP IAM permission to read Secret Manager in the relops-bootstrap project, (b) SSH access to the target Linux worker (which the operator already needs for any provisioning work), and (c) the relops-bootstrap repo checked out locally with sscep installed.

Why we accept PEM-on-disk on Linux

The Linux worker's PEM cert + PEM key live on the filesystem at /etc/relops-bootstrap/{cert,key}.pem, root:root, mode 0600. An attacker with root on the worker can read both.

This is a step down from macOS's keychain protection, but it's not a step down from the baseline: the file the cert is fetching — /root/vault.yaml — also lives root-readable on the filesystem in the same way it always has. An attacker who can read the cert can also just read the vault.yaml directly. The cert's only purpose is bootstrap-time enrollment; it doesn't grant any additional privilege beyond what root already has on that host.

The choice we deliberately did not make: use TPM-backed key storage on Linux. The hardware exists (TPM 2.0 is in most modern servers and almost all modern Mozilla hardware), and tpm2-tss + openssl-tpm2-engine would let us store the key in TPM and sign via the TPM. We chose not to for two reasons: (1) it adds a significant dependency to the bootstrap script, increasing the surface area of "things that can break during enrollment," and (2) the threat model doesn't really call for it — see the previous paragraph. If the calculus changes (e.g. cert lifetimes become long enough that key compromise becomes meaningful), we'd revisit.

7The dead ends we hit (and what they teach)

An honest account of how we got here. Each of these is a path we pursued, sometimes deeply, before realizing it wouldn't work. Understanding why each failed informs what the current design is defending against.

Dead end 1: JWT signed by the cert's private key

Our first design was conceptually clean: the worker holds a SCEP cert; at request time it constructs a short-lived JWT with claims {sub, role, aud, exp, iat, jti}, signs it with the cert's private key, embeds the cert chain in the JWT header as x5c, and sends it to the broker as a Bearer token. The broker validates the JWT signature against the cert's public key, chain-verifies the cert to our root, parses claims, role- binds, returns the secret.

This is a very standard "workload identity" pattern (it's how Kubernetes service accounts work, broadly). It died on macOS because signing the JWT requires calling SecKeyCreateSignature from our own process, which the keychain ACL forbids (see section 5). We tried for two days to get our Swift bootstrap binary into the ACL — through profile flag changes, through entitlement requests, through a LaunchDaemon at first boot, through SCEP enrollment with various profile options. None of it worked. The conclusion was that our own code fundamentally cannot sign with this key.

We then briefly considered pivoting to "let curl do the JWT" but curl doesn't sign JWTs. We considered a small Apple-signed helper binary — we don't have an Apple Developer ID with the right entitlements, and getting one would be a 3-month process. mTLS — where the OS does the signing during the TLS handshake — became obviously better.

Dead end 2: ACME with Apple device attestation

Apple's macOS supports ACME enrollment with device attestation challenges, where the device proves it's a real Apple device via a signed statement from the Secure Enclave. This is the "most secure" modern enrollment option — much better than our shared SCEP challenge. We invested significant time to make it work: configured step-ca's ACME provisioner with Apple attestation, fixed multiple iterations of the mobileconfig payload, debugged the device-attest-01 challenge flow with step-ca's logs.

We got it working end-to-end: the device successfully enrolled, step-ca validated the Apple attestation, a cert was issued. But the cert landed in a different access group within the keychain (com.apple.identities) than the SCEP-issued one (com.apple.systemkeychain). The com.apple.identities group is gated by an Apple-only entitlement that no third-party developer can request. Even find-identity on a privileged process returned errSecMissingEntitlement (-34018). Neither our code nor even SecureTransport curl could use the cert. It was, by Apple's design, only usable by Apple's own MDM-aware services.

We documented this and reverted to SCEP. ACME-with-attestation might become viable on a future macOS version if Apple loosens com.apple.identities, but for now SCEP is what works.

Dead end 3: URLSession with explicit credential

Before discovering the SecureTransport curl backend, we attempted to do mTLS from a Swift binary using URLSession, which is Apple's standard high-level networking API. URLSession has a delegate callback for NSURLAuthenticationMethodClientCertificate where the application provides a URLCredential with a SecIdentityRef. We wrote this code, found the SCEP identity by Subject CN, constructed the credential, returned it.

The behavior we observed: the TLS handshake failed with error -9860 (errSSLClientCertRequested) and the diagnostic key _kCFStreamPropertySSLClientCertificateState=0 — meaning "no client cert was actually sent." After much debugging we concluded: URLSession silently drops the credential if the LB's CertificateRequest message contains an empty acceptable-CA list (which GCP's LB sends). URLSession's policy appears to be "if I don't know which CA the server wants, don't send anything." There is no API to override this behavior. SecureTransport curl, by contrast, sends the cert anyway and lets the server reject it if it doesn't like it.

This was a surprising finding — URLSession is the modern, recommended API, and SecureTransport is the deprecated one. But for our use case, the deprecated API is the one that actually works.

What these dead ends collectively teach

All three were attempts to use Apple's modern APIs (URLSession, JWT- signing-from-app-process, ACME with attestation). All three failed because Apple's modern APIs prioritize a different security model than the one we needed — they assume the app is the entity holding the cert, and Apple's frameworks gatekeep that. The working solution (SecureTransport curl + SCEP) treats the OS as the entity holding the cert and goes through Apple's own networking helpers. Once you frame it that way, our setup looks more like "use macOS as nature intended" than "fight macOS to get our way."

8Threat model — what we defended against, what we didn't

Walking through the realistic attackers, what they can do, and how the layered defenses respond. We've kept this honest — some threats are not fully mitigated, and we want that to be visible.

T1: Random internet attacker (off MDC1 network)

Mitigation: Cloud Armor source-CIDR allowlist on the LB. The LB has a security policy that denies all traffic except from 63.245.209.101/32 (Mozilla's MDC1 NAT egress IP, plus GCP's standard health-check ranges). An attacker from anywhere else on the internet sees a 403 from Cloud Armor before TLS even terminates. They can't even probe.

Residual risk: if MDC1's NAT egress IP changes and we forget to update the allowlist, legitimate traffic fails. Operational, not security, but worth knowing.

T2: Attacker on MDC1 network without a valid cert

Mitigation: mTLS at the LB rejects unauthenticated /secret/* requests at the broker. The attacker can reach the LB, can open a TLS connection, can send an HTTP request to /secret/<role> — but without a cert, X-Client-Cert-Present arrives as false and the broker returns 401. Every attempt is audit-logged.

Residual risk: the attempt itself reveals our endpoint structure. Mitigation: we don't consider forge.relops.mozilla.com itself secret; its existence is in DNS.

T3: Attacker who compromises a worker as root

Mitigation: the cert grants no privilege the attacker doesn't already have. Root on a worker can read /root/vault.yaml (Linux) or /var/root/vault.yaml (macOS) directly — that's how puppet reads it too. The cert lets the attacker re-fetch that same vault.yaml from the broker. There's no escalation.

On macOS specifically, the cert's private key is not exfiltrable — the keychain ACL prevents export. The attacker can use the cert on that host but can't take it elsewhere.

On Linux, the PEM key is on disk and the attacker can copy it. They can then use it from elsewhere to fetch the same role's vault.yaml. But they need to also be on the Cloud Armor allowlist (63.245.209.101/32), so this only works if the attack pivots to a system on MDC1 first.

Residual risk: Linux key portability. Could be addressed with TPM-backed key storage, deferred (see section 6).

T4: Attacker who steals the SCEP challenge

Mitigation: Cloud Armor + per-role-binding. Even with the challenge, an attacker needs to be on the MDC1 source-IP allowlist to talk to the SCEP enrollment endpoint. If they are on MDC1 and have the challenge, they can mint a cert claiming any role whose provisioner uses the same challenge — and then use that cert to fetch any role's vault.yaml.

Residual risk: this is the single biggest exposure in the current design. The challenge lives in (a) GCP Secret Manager (read by operators via gcloud auth), (b) SimpleMDM's profile delivery system (read by mdmclient on each device, embedded in the mobileconfig). A compromise of either Mozilla's GCP IAM or SimpleMDM's profile storage exposes this. Mitigations available but not yet implemented: per-role challenges (currently we share one), short-TTL challenges that rotate (step-ca supports this), moving Linux enrollment to a JWT-token-based provisioner where the token is signed by a key only the operator's gcloud auth can reach.

T5: Attacker who compromises the broker

Mitigation: the broker's IAM is scoped to read vault-* secrets only. The broker SA cannot read any other Secret Manager secret, cannot read Cloud Storage, cannot modify GCP resources, cannot reach the step-ca VM. Compromise gives the attacker the ability to read all vault-* secrets — every role's vault.yaml. That's significant: it includes every role's taskcluster access token, every cltbld password, etc.

Residual risk: "read all vault-* secrets" is real. We mitigate how easy a compromise is by keeping the broker surface area tiny (it's ~150 lines of FastAPI; no cryptography of its own beyond parsing the leaf cert to read its SAN; no user input beyond the URL path), but we do not mitigate the consequence. This is the same blast radius as compromising operator 1Password access, which RelOps has lived with for years.

T6: Attacker who compromises step-ca

This is the worst case. An attacker who runs arbitrary code on the step-ca VM can mint certs for any role, use them to read any vault.yaml, and arguably do worse — they could issue certs that are themselves trust anchors for downstream attacks.

Mitigations: step-ca VM has no public ingress except the LB. Admin SSH requires Google IAP + IAM grant in the relops-bootstrap project. The OS disk is encrypted at rest by GCP. The intermediate signing key is encrypted at rest within the VM behind a password loaded from /home/step/.step/secrets/password, which is only readable by the step service user.

Residual risk: if a Mozilla GCP admin's account is fully compromised, the attacker can SSH to step-ca and read the key. This is the same trust we already place in GCP admins across the entire Mozilla cloud footprint — we don't have a special defense against it here, and we don't pretend to.

T7: Attacker who compromises an operator's laptop

An operator with a compromised laptop is, today, also an operator with 1Password access (which is the baseline). Our system doesn't make this worse: the attacker gets the operator's gcloud auth, which lets them read GCP Secret Manager, which gives them the SCEP challenge and the role secrets. The same compromise gave them 1Password vault access anyway. No regression.

The summary table

Net assessment: the new system trades one fairly small new exposure (SCEP challenge as a single point of failure) for several substantial operational and security improvements (audit trail, automation, removal of the "operator with a YAML on their laptop" chokepoint). We think the tradeoff is favorable, especially given the challenge surface is small and operational mitigations exist.

9Operational reality: how operators actually use this

The macOS provisioning experience

An operator clicks Reprovision in Hangar (one host or a whole pool); the on-network runner drives the rest. It quarantines and drains the worker in Taskcluster, then triggers EACS. The device wipes and DEP re-enrollment kicks in. SimpleMDM pushes the assigned profiles, including Dev - SCEP. mdmclient enrolls a fresh cert into the System keychain. The runner then auto-mints the first SecureToken over an on-host ssh login (DEP skips Setup Assistant, so admin holds no token until a PAM login — and Apple only grants the first SecureToken through an interactive session, which is exactly why the driver must be on-network; a script/PKG can't), and the Bootstrap Token is escrowed alongside it. The bootstrap — a signed PKG (managed install) built from scripts/simplemdm-m4-no-sip-bootstrap.sh that lands during DEP convergence and runs as root — then runs CURL_SSL_BACKEND=securetransport curl --cert "<CN>" ... to fetch /var/root/vault.yaml. It clones the appropriate ronin_puppet branch, installs a reboot-survivable com.mozilla.m4-bootstrap LaunchDaemon that runs run-puppet.sh via ssh root@localhost until the safari semaphores fire. Once they do, the daemon writes a sentinel, kicks the worker-runner trigger, and removes itself.

Total operator touches: one click in Hangar — the runner handles quarantine, drain, EACS, mint, escrow, and un-quarantine, and streams every phase back to a live view. The entire flow from click to "worker re-registered in Taskcluster" is roughly 30-45 minutes wall-clock; most of that is puppet doing real installation work, not our bootstrap. (The individual steps are also runnable directly from the reprovision CLI for a single host without Hangar.)

The Linux provisioning experience

# operator terminal, MDC1-accessible laptop
export RELOPS_BOOTSTRAP_REPO=~/git/relops-bootstrap
cd ~/git/ronin_puppet/provisioners/linux
./deliver_linux.sh worker-04.mdc1.mozops.net gecko_t_linux_2404_talos

# script handles: SCEP enrollment locally, scp cert to host,
# scp bootstrap_linux.sh, kick the bootstrap. Operator watches.

Total operator touches: one command. The operator's laptop needs sscep installed (brew or apt) and the relops-bootstrap repo cloned somewhere.

What failure looks like

Some realistic failures and what they look like to an operator:

The Bootstrap Token thing (macOS specific)

Briefly worth knowing: Apple Silicon devices have a concept called the Bootstrap Token (BST), separate from SecureToken, separate from FileVault. The BST is generated by macOS the first time a SecureToken-eligible user logs in interactively, and is escrowed to the MDM so that MDM can subsequently grant SecureToken to other users programmatically.

Our SimpleMDM bootstrap script calls sysadminctl -secureTokenOn admin -password admin, which (on macOS 12.4+) automatically uses the escrowed BST to grant SecureToken to admin without requiring another SecureToken user's credentials. This is the standard way to get the admin user into a state where it can manage FileVault and other privileged things later on. Empirically: BST generation happens automatically on first console login or SSH login, but BST escrow to MDM can be in a separate state; the script explicitly calls profiles install -type bootstraptoken to force the escrow if it hasn't happened. This wasn't quite obvious from Apple's docs and bit us empirically — see the feedback_apple_silicon_bst_escrow memory if you've forgotten.

10Open questions and follow-ups

Things we know we should think about, in roughly priority order:

• Per-role SCEP challenges. Currently all provisioners share one challenge. Splitting them would limit blast radius of a leak. step-ca supports per-provisioner challenges natively; the lift is rotating the challenge in SimpleMDM profiles + Secret Manager.
• Challenge rotation. The challenge is currently static. Rotating it on a schedule (e.g. monthly) would meaningfully reduce the risk of an undetected long-term leak. SimpleMDM profile updates can re-deliver it.
• CRL / revocation. Not currently implemented. step-ca supports CRL endpoints; the broker would need to fetch + cache the CRL and check incoming cert serials. Cert lifetimes are 24 hours (step-ca's built-in default — not explicitly pinned in ca.json, so it should be set explicitly if we ever want to rely on it) so revocation gets you ~half a day of reduction in window of misuse — not zero but not nothing.
• Hardware-bound keys. macOS: blocked by Apple's com.apple.identities entitlement model; revisit if Apple loosens it. Linux: feasible via TPM 2.0; not currently a priority.
• Per-role expansion. Live today: the m4 family (gecko_t_osx_1500_m4, …_staging, …_no_sip) has both a provisioner and a populated secret. r8 and 1015 aren't started (empty secret containers, no provisioner); the six Linux roles have provisioners but empty secrets (see §6). Each remaining role is a step-ca SCEP provisioner + SimpleMDM SCEP profile + populated Secret Manager secret — mechanical, ~30 min per role.
• Dead ACME route. The CA/IaC drift is otherwise closed — scripts/bootstrap-step-ca.sh now codifies all eight live SCEP provisioners, the abandoned acme-no-sip provisioner and its mobileconfig template were removed (dead end 2). One tidy-up remains: the LB url-map still routes /acme/* to step-ca (harmless — step-ca no longer has an ACME provisioner, so it just 404s); drop that path rule from terraform/lb.tf when convenient.
• Windows hardware support. Same architecture would work natively on Windows (native cert store, Invoke-RestMethod -Certificate, or curl.exe --cert). Pending finding where the existing Windows bootstrap.ps1 lives — it's not in ronin_puppet.
• Audit dashboard. Cloud Audit Logs has everything but a per-role/per-host view requires querying. A small page in hangar would surface this nicely.

11Glossary

ACME

Automatic Certificate Management Environment, the protocol Let's Encrypt uses (RFC 8555). step-ca supports it; we briefly tried it for macOS enrollment with Apple device attestation before hitting the keychain access group wall.

BST (Bootstrap Token)

A per-Apple-Silicon-device token generated at first SecureToken-eligible login, escrowed to MDM, used to grant SecureToken to other users programmatically. Distinct from but related to FileVault setup.

Cloud Armor

GCP's WAF / source-CIDR allowlist service that sits in front of HTTPS LBs. We use it as our outer perimeter — only the MDC1 NAT egress IP is allowed.

cltbld

The CI build/test user on a worker. Created by puppet; gets the cltbld_user credentials from vault.yaml; runs the actual Taskcluster generic-worker process.

CSR (Certificate Signing Request)

The thing a client sends to a CA when it wants a cert. Contains the public key (the CA signs this), a proposed subject, and (in SCEP's case) a challengePassword attribute for authentication.

EACS

"Erase All Content and Settings" — Apple's mechanism to factory-reset a Mac while preserving DEP enrollment. Our primary tool for re-provisioning workers.

Hiera

Puppet's hierarchical data-lookup system. Reads YAML files in a configured hierarchy (e.g. roles/<role>.yaml then os/<os>.yaml then common.yaml). Our vault.yaml lives in this hierarchy at the highest priority.

IAP (Identity-Aware Proxy)

GCP's authenticated proxy for SSH access to VMs without exposing port 22 publicly. We use it for step-ca admin access.

mdmclient

Apple's MDM client daemon on macOS. Receives profiles from the MDM server (SimpleMDM in our case), installs them, runs the SCEP enrollment flow when a SCEP profile is delivered.

mTLS (mutual TLS)

TLS where both the server and the client present and verify certificates. Standard TLS authenticates only the server; mTLS authenticates the client too. We use it at the LB.

PEM

The base64-encoded-with-headers format for X.509 certificates and private keys. -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----. Linux workers store their cert + key as PEM files at /etc/relops-bootstrap/.

PKI (Public Key Infrastructure)

The umbrella term for the system of CAs, certs, signatures, and validation rules that lets parties trust each other's certificates. Our step-ca + Trust Config + per-host certs are the PKI here.

SCEP (Simple Certificate Enrollment Protocol)

RFC 8894. The protocol Apple's mdmclient natively speaks for cert enrollment when given a com.apple.security.scep profile. step-ca supports it.

SecureToken

An Apple Silicon attribute on a user account that grants them the ability to unlock FileVault and access certain hardware-backed crypto. SecureToken is granted at first interactive login or via BST.

SecureTransport

Apple's (deprecated but still functional) TLS library, used by macOS's curl as one of its backends. Has the unique property of being able to use keychain identities by Common Name. The load-bearing piece for our macOS mTLS flow.

SimpleMDM

The MDM SaaS Mozilla RelOps uses for macOS device management. Pushes profiles to devices, runs script-jobs at first boot, manages enrollment.

SPIFFE URI

A URI scheme for naming workloads (spiffe://<trust-domain>/<path>). We use it in the cert SAN to bind a cert to a single puppet role: spiffe://relops.mozilla/host/<CN>/role/<puppet-role>.

sscep

An open-source SCEP client CLI. Used on the operator's laptop to enroll a Linux worker's cert.

step-ca

An open-source ACME / SCEP / JWT-token-based certificate authority, by Smallstep. We run it on a GCE VM as our CA for relops-bootstrap.

Trust Config

A GCP Certificate Manager resource that lists the trust anchors (and optional intermediates) the HTTPS LB will accept for mTLS client cert validation. Ours contains the step-ca root + intermediate.

vault.yaml

The per-puppet-role secrets file. Hiera reads it. Contains taskcluster_access_token, cltbld_user.password (hashed and plaintext), bugzilla_api_key, livelog_secret, etc. The thing this entire system exists to deliver.