darwin CI agents

Provisioning for the macOS test agents on queue test-darwin.

Two modes:

tart: for Apple Silicon hosts with memory to spare. The host runs only buildkite-agent and Tart; every test job runs in a fresh macOS guest cloned from a baked image and deleted afterwards.
bare: for Intel hosts (Tart cannot virtualize macOS on Intel) and small Apple Silicon hosts. The bun toolchain and scripts/agent.mjs service run on the host itself.

Agents tag themselves os=darwin arch=... release=<macOS major> release-tier=... and .buildkite/ci.mjs selects on those. In tart mode release is the guest's macOS version, and a guest cannot be newer than its host.

Layout

host.sh            first contact: installs brew and bun, then runs `main.ts provision`
main.ts            provision | setup-user | bake | install-agent
lib/               host hardening, tailscale, the unprivileged CI user, tart, bake, agent config
hooks/             agent hooks for tart hosts (command, pre-exit, environment)
guest/bake.sh      runs inside the guest once, at bake time
guest/job.sh       runs inside the guest for every job

provision <hostname> tart disables remote management, makes sshd key-only, joins the tailnet, installs buildkite-agent and Tart, creates an unprivileged auto-login user (Virtualization.framework needs a console session), bakes the guest image from a public base image plus scripts/bootstrap.sh, and starts the agent as that user with hooks/ as its hooks path. It asks for one reboot the first time and is re-run after it.

provision <hostname> bare does the same host setup, then runs scripts/bootstrap.sh on the host and installs the scripts/agent.mjs service.

bake is safe on a live host: it builds a staging image and swaps it in only after the toolchain verifies. Re-run it when toolchain pins move.

Bringing up a host

Prerequisites on a freshly imaged host: an admin account you can ssh into with a key, passwordless sudo for it, the host's address on the agent token's IP allowlist, and the agent token written to the root-only file named in lib/config.ts (or DARWIN_CI_TOKEN_FILE).

scp -r scripts/darwin-ci <admin>@<host>:
ssh <admin>@<host> 'darwin-ci/host.sh <hostname> tart --tags <tailscale tags>'

Approve the Tailscale login it prints, reboot when it asks, run the same command again, and check the agent appears under queue=test-darwin.

Removing a host

Unload the agent (launchctl bootout the com.buildkite.buildkite-agent job), drop the host from the token allowlist and the tailnet, and release it. Nothing on a host needs preserving.