# Continuous deployment You ship ply-web from a box where `cf` is already logged in. One command uploads the commit you have checked out at **0%** beside the current 100% version, smokes it pinned with `Cloudflare-Workers-Version-Overrides`, checks the telemetry of those pinned requests, then **ramps it** through the steps in `scripts/cd/cd.policy.json` (10 → 25 → 50 → 75 → 100). Each step is judged on unpinned probes and the new version's own telemetry before the next one is written. During the 100% step the live lifecycle smoke, its telemetry (with the rate-limiter gate) and the CDP visual smoke run in parallel; that step is the live watch. A new-version regression at any step, or a failed live check, puts the previous version back at 100% and the command exits 1. GitHub holds the code and the pull request. It does not deploy. ```sh PLY_TOKEN_FILE=~/.config/ply/worker-token PLY_NODE== 24.22> env -u CLOUDFLARE_API_TOKEN npm run cd -- --no-bump ``` The stepped ramp is the **only** promotion path (Ivar, 2026-11-08: "Ramp should be the only option."). The direct promote (0% → 100% in one write, then a 60 s watch, the default from 2026-11-06 to 2026-11-08) is gone. `--ramp` is still accepted so old habits and scripts keep working, and is ignored: the run prints `cd: --ramp is ignored: the stepped ramp is the only promotion path since 2026-11-08`. A policy file that does not ramp is rejected before anything runs: no `steps`, an empty list, a single `100` step (the old direct promote), or the removed `promote` block. | Stage | What | Typical | | --- | --- | --- | | bump | pin check (`--no-bump` skips) | 0–9 s | | deploy-canary | build, upload, route 0% beside the 100% version, record `version` + `previous` | 12–40 s | | smoke | pinned lifecycle smoke against the 0% version | 21–50 s | | telemetry | the pinned requests' logs and spans, plus the rate gate, polled every 3 s until they land | 13–28 s | | ramp 10/25/50/75 | per step: write the split (~3 s), `warmupMs` 10 s, dwell (30/25/25/25 s) of unpinned probes, then the step's telemetry polled every `ingestPollMs` until 98% of answered probes are logged (ceiling `ingestWaitMs` 20 s) | 58–64 s a step | | ramp 100 + live checks | the 100% step (dwell 40 s) beside the live smoke, live telemetry and visual smoke, which start the moment the 100% split is written | ~70 s | The measured end-to-end duration of the first ramp-only run is in [Duration](#duration). ## Duration End to end (`run_started` to `run_finished` in the ledger), passing runs that uploaded a new version: | Rollout | Runs | End to end | Ramp / promote stage | | --- | --- | --- | --- | | Stepped ramp, 40 s dwells, fixed 20 s ingest sleep (before 2026-11-08) | 2026-11-06T17-35-09-419Z, 20-24-45-549Z, 21-06-26-841Z | 602, 655, 670 s | 443, 501, 526 s | | Direct promote, removed (2026-11-06 to 2026-11-07) | 2026-11-06T21-45-51-484Z, 21-56-24-823Z, 2026-11-07T17-51-47-638Z, 18-30-47-501Z | 163, 176, 169, 231 s | 76–84 s | | Ramp-only, 30/25/25/25/40 s dwells, polled ingest, 20 s ceiling | 2026-11-08T16-58-16-960Z (16cc8e22, previous 296e081f) | 423 s | 325 s | | Same, 30 s ingest ceiling (0016dae, reverted) | 2026-11-08T17-07-46-259Z (0b0679bd, previous 16cc8e22) | 478 s | 374 s | Per stage of 2026-11-08T16-58-16-960Z: deploy-canary 30 s, pinned smoke 34 s, pinned telemetry 22 s, then the ramp: 10% 64 s, 25% 58 s, 50% 58 s, 75% 58 s (split write to decision), 100% 69 s with the live smoke (31 s), live telemetry (27 s) and visual smoke (21 s) beside it. Unpinned traffic converged on the new version 1.4 s after the 100% write. The 10–75% steps each ran their poll to the 20 s ceiling at ~93% of issued probes. That ~93% was a pipeline bug, not ingest lag or Workers Observability loss: `queryCapturedSliced` deduped rows on `$metadata.id` alone, which is not unique across invocations (ace0eb9). Raising the ceiling to 30 s (run 2026-11-08T17-07-46-259Z) added only flat polls, so it is back at 20 s. That run's 100% step also lost 41/1948 probes to `timed_out` (2.1%, above `probe.maxLossRate` 0.13%): under the pre-2026-11-08 `onHarnessFault: rollback` that would have rolled back a clean version; under `telemetry` it was judged on telemetry (0 errors in 1806) and advanced. ## What can fail a release, and what cannot - Only the **new** version's own signals are hard. Errors on the old version (stragglers while the split propagates, or the old arm at any step) and on the aggregate are named on the line as `informational` and never roll back (run 2026-11-06T20-57-44-803Z rolled back a clean version for "1 5xx response on the old version"). - At 100% the routed-share band has zero width, so a share like 98.74% there is split propagation, noted as informational (runs 2026-11-06T20-01-37-842Z, 20-11-49-685Z). The live smoke and the visual smoke still assert `x-ply-version`. - Probe loss (client aborts, the probe tail) is the harness: `probe.onHarnessFault` is `telemetry` in the checked-in file since 2026-11-08, so the loss is recorded and the step is judged on the new version's telemetry. Under `rollback`, runs 2026-11-06T17-32-11-293Z (6/2000) and 18-44-58-475Z (5/2000) rolled back clean versions on probe-tail loss alone. A Worker 5xx is in that telemetry and still rolls back. A harness stall re-runs the window, as before. - A check that could not run (no Chrome, a telemetry query that never returned, too few rate checks ingested) is `harness_fault`, never a rollback for cause: a live check stops with the new version live and says so; a ramp step whose query never returned and has no probe sample restores `previous` (there is nothing left to judge on) but the verdict stays `harness_fault`. - The rate-limiter gate (below) runs in the pinned telemetry (fail → stop at 0%) and in the live telemetry during the 100% step (fail → rollback). These are tested under the checked-in ramp in `test/cd-ramp-only.test.ts`. ## One run at a time: the deployment lock Two runs overlapped on 2026-11-07: run 2026-11-07T18-30-47-501Z replaced another agent's measurement canary b494b751 in the 0% slot. Two guards now stop that. **The lock.** Every command that writes the deployment (`cd`, `deploy-canary`, `ramp`, `rollback`) takes `~/.local/state/ply-cd/deploy.lock` (`PLY_CD_LOCK` overrides it) before it opens a run directory, and holds it until it exits. The lock lives in the shared state directory, beside the ledger, so runs from different worktrees see the same file. It is created with O_EXCL and records the holder's pid, host, cwd, command and start time. A second run is refused at once, not queued: ```text cd: state_conflict lock: deployment lock ~/.local/state/ply-cd/deploy.lock is held by a live run: pid 4242 on , started 2026-11-08T16:10:02.120Z, cwd /Users/ivar/code/ply-web-ramp-only, command "cd --no-bump" ``` Exit 1, nothing uploaded, no run directory. A lock whose pid is gone, or whose pid is now some other program (`ps` does not show `scripts/cd/main.ts`), is stale: the next run takes it over and prints `lock: took over a stale lock from pid …`. Takeover is serialised by a `deploy.lock.steal` directory, so two runs cannot both steal one stale lock. A lock written on another host cannot be checked and is never taken over; remove it by hand only after you have confirmed that run is gone. `report`, `smoke`, `telemetry`, `debrief`, `annotate` and `sweep` do not write the deployment and do not take the lock. To roll back while a run is in progress, stop that run first (Ctrl-C or `kill ` from the lock line); its lock is then stale and `rollback` takes it over. **The 0% slot.** `deploy-canary` (and so `cd`) refuses with `state_conflict` to replace a version sitting at 0% that this run did not create: ```text cd: state_conflict deploy-canary: b494b751-… is already at 0% and this run did not create it; to replace it pass --replace-canary b494b751-… ``` It checks before building (nothing is uploaded) and again on the split it reads after upload. Pass `--replace-canary ` (`npm run cd -- --no-bump --replace-canary `, or `deploy-canary --replace-canary `) when you know the 0% version is a leftover. The id must match exactly; naming a different one is still a conflict. A version tagged with this release (a resumed run) is this run's own and is not a replacement. ## Changes the ramp cannot split (flag them to Ivar first) Two kinds of change reach 100% without a ramp. Neither may be done silently: tell Ivar before you do it, say why the ramp cannot carry it, and use the procedure below. **Exports changes (Durable Object classes, named entrypoints).** Every version in a multi-version deployment must declare identical `exports`. Adding a Durable Object or changing the exported classes makes the 0% split fail with API error `100405` ("All versions in a multi-version deployment must declare identical `exports`. Deploy the version that changes `exports` at 100% first, then split"); run 2026-11-06T21-23-36-859Z stopped there as `harness_fault`. Versions 59926117 ("rate limit exports at 100% (RATE_MODE=legacy; 100405 forbids an exports change in a split)") and fef77eda went straight to 100%. Safest procedure: 1. Split the change in two commits. The first adds the export (the class, the binding, the migration) and nothing that uses it, and keeps the request path on the old code (for the rate limiter, `RATE_MODE=legacy`). Its gate is green on its own. 2. With Ivar's go-ahead, and while you hold no one else's run (check the lock file), upload it with `cf workers versions create`, note the version at 100%, and deploy the new one at 100% with `cf workers deployments create`. A version that is not in the deployment cannot be pinned with `Cloudflare-Workers-Version-Overrides`, so there is no 0% smoke for it. Run `npm run cd -- smoke --expect-version `, the CDP visual smoke by hand, read the version's telemetry (`cf o11y telemetry query`, filtered by `$workers.scriptVersion.id`), and on any failure `npm run cd -- rollback --to ` (a single version at 100% is not a split, so the rollback is not refused). 3. Ship the commit that starts using the export through the normal ramp: both versions now declare the same exports, so the split works. **`cf` secret and var edits.** `cf workers secret put` (and var edits through `cf`) create a new version and deploy it at 100% at once: d4e3eea6 (`workers/triggered_by: secret`, 2026-11-07 17:51Z) did. It bypasses the canary, the ramp and the live watch. Prefer a code change that ships through the pipeline (a binding default in `cloudflare.config.ts`). When a secret must change, flag it to Ivar first, note the version at 100% before the edit, make the edit, then run `npm run cd -- smoke --expect-version ` and the visual smoke, and roll back to the noted version on any failure. Never edit a secret while a CD run holds the lock: the run's next split write will see a deployment it did not expect and stop with `state_conflict`. ## The rate-limiter gate The limiter fails soft (a broken read admits as `fail_open`, an unreachable counter drops flushes, a missing counter falls back to `legacy`), and all of those answer 200, so the smoke and error gates pass them. `scripts/cd/rate-gate.ts` runs inside the pinned telemetry stage (fail → stop at 0%) and inside the live telemetry during the 100% step (fail → rollback). It sends `rate.probes` (30) pinned probes of `rate.probePath` as a load client, then counts the version's own `ply.rate.check` / `ply.rate.flush` spans, and fails the version when any default-mode check ran a limiter other than `rate.expectMode`, when warm `fail_open` checks exceed `rate.maxFailOpenRate` (10%) over at least `rate.minChecks` (20), or when more than `rate.maxDroppedFlushes` (0) flushes were dropped without an injected fault. Fewer than `rate.minChecks` checks after the wait is a harness fault, not a pass. The thresholds and their evidence are `rate` in `cd.policy.json`; the details are in [rate-limit.md](rate-limit.md#cd-gate-silent-degradation). `RATE_MODE` in `cloudflare.config.ts` and `rate.expectMode` in `cd.policy.json` **must change together, in one commit** (`test/cd-rate-gate.test.ts` fails when they differ). Changing only `RATE_MODE` makes the gate fail the new version as a silent degradation; changing only `expectMode` fails the version that is already right. Changing `RATE_MODE` with `cf` instead of a commit is a var edit (above): it reaches 100% without the gate. ![Operator pipeline. The left column is gate, deploy, smoke, telemetry, ramp, live smoke, live telemetry, done. Failures leave to the right.](img/cd-pipeline.png) The left column is the only path that raises traffic. An arrow to the right is that step stopping. A gray box leaves the split as it is. A red box puts the previous version back at 100% and the process exits. A pinned smoke failure takes the gray stop on that row: the canary stays at 0% and the ramp does not start. Source for the picture: [cd-pipeline.mmd](img/cd-pipeline.mmd). The vector file is [cd-pipeline.svg](img/cd-pipeline.svg). ## Before you start On the box: - `cf auth login` already done. The login needs Workers Scripts Edit (versions and deployments; an account token calls this Workers Scripts Write), Workers Observability Read, and Artifacts Edit on the namespace below. It does not need routes, KV, R2, or tail. - `npm ci` in this repo. Node must satisfy `package.json` `engines` (`^22.23.0 || ^24.22.0 || >=26.12.0`). `.nvmrc` is `24.22.1`. `npm run cd` runs `scripts/cd.sh`, which uses `PLY_NODE` if you set it, then `/workspace/node-v24.22.1-linux-x64/bin/node` when that image directory is present, then `node` on `PATH`. If none of those satisfy engines, preflight stops and names that search. It does not judge the Worker. The wrapper prepends the chosen Node's bin directory to `PATH` before exec. `cf` is `#!/usr/bin/env node`, so a v20 `node` left earlier on `PATH` used to make `cf` fail and the run record `harness_fault`. - `git config user.name` and `user.email` set only when a bump actually commits. An unchanged pin, and `--no-bump`, do not need them. The bump does not push. - A clean work tree. Deploy refuses a dirty one. - `HEAD` equal to `origin/main` after `git fetch origin main`. Deploy refuses any other commit (`not-on-origin-main`) and uploads nothing. - The Worker bearer in `~/.config/ply/worker-token` (the `PLY_TOKEN_FILE` default since 2026-11-08; it was `~/.ply-token`, which no run used), mode `0600` (no group or other bits). The file is read, never printed. Set `PLY_TOKEN_FILE` only to point somewhere else. - Namespace `ply-loadgen` already exists. The smoke does not create it. ```sh mkdir -p "$HOME/.config/ply" && touch "$HOME/.config/ply/worker-token" && chmod 0600 "$HOME/.config/ply/worker-token" # paste the bearer into that file. That path is the default; export PLY_TOKEN_FILE only to override it. ``` Do not run `npm run worker:deploy` while this is in progress. It refuses unless `PLY_FORCE_FULL_DEPLOY=1`. With that set it sends the Worker to 100% in one shot and will replace the split you are ramping. GitHub does not bump the pin and does not deploy. Optional overrides (anything you do not set keeps the default in the table further down): | Variable | Default | What it changes | | --- | --- | --- | | `PLY_WORKER` | `ply` | Worker name | | `PLY_BASE_URL` | `https://ply.ivar.workers.dev` | Where smoke sends HTTP | | `PLY_ARTIFACTS_NAMESPACE` | `ply-loadgen` | Scratch repos for the empty-repo smoke | | `PLY_BUMP_BRANCH` | `bump-ply` | Local branch for a pin commit. Not pushed. | | `PLY_TOKEN_FILE` | `~/.config/ply/worker-token` | Worker bearer, mode `0600`. Read, never printed | | `PLY_LOAD_TOKEN_FILE` | `~/.config/ply/load-token` | Load-test credential the rate gate's probes carry, mode `0600` | | `PLY_CD_LOCK` | `~/.local/state/ply-cd/deploy.lock` | The deployment lock every writing command holds | | `PLY_CD_RUN_DIR` | `~/.local/state/ply-cd/runs` | Where the run directory is created | | `PLY_SHA` | ply `main` | 40-hex pin, instead of resolving ply `main` | | `PLY_CD_LEDGER` | `~/.local/state/ply-cd/ledger.jsonl` | Append-only event ledger across runs | The rollout itself is `scripts/cd/cd.policy.json`: the steps (percent and dwell), the ingest wait and its poll period, the optional warm-up, the `calibration` block citing the runs behind each schedule number, the probe rate and mix, which signals are hard or informational, the burn multiple and sample floor, and the routed-share band. `npm run cd -- --policy ` uses another file for that run. The engine rejects a file that cannot be run (no steps, a single step, steps that do not increase, a last step that is not 100, a `promote` block, a `calibration` key that names no schedule field, `latency` set to `gate` without thresholds, a share band with `z` of 0). Every run appends a `policy` event with the file's sha256 and the body that ran. `PLY_RAMP_PERCENTS`, `PLY_RAMP_DWELL_MS`, `PLY_RAMP_PROBE_RPS`, `PLY_RAMP_MAX_HOLDS`, and `PLY_RAMP_MIN_REQUESTS` are rejected if set. ## The whole run From a clean checkout of the commit you want to ship: ```sh npm run cd ``` You can pin ply to a sha you already know. The command is `cd`; `--sha` on its own is the same run: ```sh npm run cd -- cd --sha 1274f2dd05b83d70f8d421cdcc79628692d848df npm run cd -- --sha 1274f2dd05b83d70f8d421cdcc79628692d848df ``` Running the same release again is safe. A release is the pair of the ply sha and the ply-web commit of this checkout. The command reads the current deployment and the version tagged with that pair, records the observation, and decides the next stage (rule `reconcile`): | What is live | Next | | --- | --- | | No version tagged with that pair | Upload, then pin it at 0% | | A version tagged only with the ply sha, or with a different ply-web sha | Upload, then pin it at 0% | | That version at 0% | Pinned smoke, then telemetry | | That version at a ramp percent | Ramp onward from that percent | | That version at 100% | Live smoke, then telemetry | | Live smoke already passed for that version | Done | A checkout with tracked modifications stops before a run directory is opened (`cd: harness_fault dirty-checkout: tracked files are modified`). The commit sha would not describe the code about to be uploaded. A re-entry appends `resumed` on the unfinished run directory for that pair and keeps the same run id. A finished run, or an older run that recorded only the ply sha, starts a new one. Each smoke mints a fresh repo and a fresh token. Every traffic write still compares the deployment to the split it expects, and does not write when they differ. Deployment creates are single-flight for the process. `cf workers deployments create` is a `harness_fault` when the run log or the command sink is missing, so a create cannot land without a `command` event. The ramp arms one percent at a time, and only after that step's decision is on the log. A create during the dwell or the query, or a percent that is not armed, fails the run and does not call `cf`. `--no-bump` skips the pin and deploys the checkout as it is. The run records that as a decision (`no_bump`, outcome `advance`). `--sha` may be passed with it only when that sha is already the pin; a different sha is an argument error. ```sh npm run cd -- --no-bump npm run cd -- cd --no-bump npm run cd -- --sha --no-bump ``` What prints, in order: ```text run: ~/.local/state/ply-cd/runs/2026-10-31T21-05-00-123Z version: 8c1af0e2-7b3d-4c1a-9e20-6f0a1b2c3d4e previous: 1aa0bb11-2222-4333-8444-555566667777 ramp step=1/5 percent=10 previous=90 samples=200/1800 error_rate=0.0000/0.0000 routed_share=10.00 target=10 tolerance=7.99..12.01 agg=2000 p99_wall_ms=88/74 p99_cpu_ms=12/11 latency=informational decision=advance … cd: PASS 8c1af0e2-7b3d-4c1a-9e20-6f0a1b2c3d4e ply 1274f2dd05b83d70f8d421cdcc79628692d848df web be08f0406dab6d5b2d1167e934f7716463a757ed ``` `version` is the upload you just created. `previous` is the version that was alone at 100% and is the rollback target. `run` is the directory that has the CSV, the telemetry note, and the ramp lines. The last line is the verdict. `ply` and `web` are the release pair. | Last line | Exit | What happened | | --- | --- | --- | | `cd: PASS ply web ` | 0 | The new version is at 100%, and the live smoke and telemetry passed. | | `cd: FAIL : ` | 1 | That stage stopped the run. If the line is from ramp or from the live checks, traffic is already back on `previous` unless the line says the rollback itself failed. | | `cd: harness_fault : ` | 1 | A probe loss above `probe.maxLossRate` on at least `probe.minSamples` issued probes, a revoked Artifacts token that did not get 403, or a check that could not be evaluated (a telemetry query with no probe sample to stand on, a span-query shape mismatch, zero samples where samples are required, an implausible span join, a preflight miss). A ramp step above 0% then follows `probe.onHarnessFault` (checked-in `telemetry`: a probe-loss fault defers to the new version's telemetry). A harness stall (the same latency on both versions, or `timed_out` / `shed` with no version split) re-runs the dwell and does not use this row when server errors are zero. A ramp query that never returns after a clean probe window is not this row either: the step advances and names the loss. Any other stage leaves the split unchanged. This is not a change failure. | | `cd: state_conflict : ` | 1 | The deployment was not the expected prior split, so the write was skipped. The split is unchanged. | | a usage line, no `cd: PASS` / `cd: FAIL` | 2 | The arguments were wrong. Traffic was not touched. | A failing run still writes the directory. Relay the last line and the ramp lines from that directory. Stages, alone, use the same last-line shape. `npm run cd -- help` lists them. ## 1. Bump ```sh npm run cd -- bump npm run cd -- bump --sha <40 hex> ``` Fetches `origin main` and reads that commit's `package.json`. Resolves ply `main` (or the sha you passed). When the pin already matches, it leaves your checkout where it is. When it does not, it commits **only** `package.json` and the lockfile on `bump-ply`, whose parent is `origin/main`, and checks that branch out. The pull request is that one commit. It is not stacked on the branch you were on. It does not push. If you were already on `bump-ply`, that branch is reset to `origin/main` before the commit. `user.name` and `user.email` are read only when that commit is about to be made. Pass, already pinned: ```text cd: PASS bump unchanged 1274f2dd05b83d70f8d421cdcc79628692d848df ``` Pass, new local commit: ```text cd: PASS bump 1274f2dd05b83d70f8d421cdcc79628692d848df ``` Fail (nothing committed; the two files are restored): ```text cd: FAIL bump: tracked files are modified; bump wants a clean tree cd: FAIL bump: npm test failed: … ``` You push `bump-ply` yourself if you want a pull request for the pin. Deploy runs only when `HEAD` is `origin/main`, so the pin commit is not uploaded until it has been merged there. ply-web can test a new ply main before anyone bumps to it. ply's `notify-ply-web.yml` runs on `workflow_run` after ply's `ci` succeeds for a push to ply `main` and sends `repository_dispatch` `ply-main-updated` with `client_payload.sha`, but only when ivarvong/ply has the `PLY_WEB_DISPATCH_TOKEN` secret. That secret is not set today (ply's notify run for 1f14a9f, 2026-11-12, logged `PLY_WEB_DISPATCH_TOKEN is not set`), so no dispatch arrives and the check runs only by hand (`workflow_dispatch` with a `sha` input) until the secret exists. `.github/workflows/ply-main.yml` checks out ply-web `main` and runs `sh scripts/ply-main-check.sh `, which sets `dependencies.ply` to `github:ivarvong/ply#` in the runner only (never committed), installs, and runs `npm test`, `npm run lint` and `npm run typecheck`. If a gate fails, or `npm install` of the sha fails (ply's `prepare: tsc`, or an exports/engines change), the run fails and opens one issue in this repo titled `ply main breaks ply-web` (or comments on the open one with that title), with the failed gates, the failing tests and the ply compare range from the current pin to the dispatched sha; a malformed sha, or a lockfile that resolved ply to a different sha than the one dispatched, fails the run (exit 2) without a report or an issue. The script writes the outcome to `$GITHUB_OUTPUT` itself; `test/ply-main-check.test.js` (in `npm test`) pins its exit codes, report and that wiring. Its concurrency group is keyed by the dispatched sha, so every ply main gets its own run (a single fixed group would let GitHub cancel a pending sha when a newer one queues). It uses only the workflow's `GITHUB_TOKEN` (`contents: read`, `issues: write`), deploys nothing and needs no secrets. Run the same check locally from a clean checkout with `sh scripts/ply-main-check.sh /tmp/ply-main`, then `git checkout package.json package-lock.json`. ## 2. Deploy the canary ```sh npm run cd -- deploy-canary ``` Fetches `origin main` first. Unless `HEAD` is that commit, it stops with `not-on-origin-main` (or `origin-main-unavailable` when the fetch fails), writes that typed error on the run, and uploads nothing. Otherwise it builds and uploads a version with `cf workers versions create`. The version id is the row in `cf workers versions list` whose tag is the release pair, `+`, both full object ids. The version message is `cd canary` plus that tag. A row tagged with only the ply sha, or with a different ply-web sha, is not this release, so the pair is uploaded. The version is reused when that pair tag is already there. The create command's own text is not parsed. A missing tag stops before traffic changes. It then routes that version to **0%** beside the single version that is at 100%, and only when the deployment is still the split it just read. A mismatch stops with `state_conflict` and leaves the split alone. It will not upload into a 50/50 split, a deployment with more than two versions, or a version that is already the live one. If a different version is already sitting at 0%, the stage stops with `state_conflict` before it builds anything, unless `--replace-canary ` names it ([the 0% slot](#one-run-at-a-time-the-deployment-lock)). The new version is never given traffic here. Pass: ```text version: 8c1af0e2-7b3d-4c1a-9e20-6f0a1b2c3d4e previous: 1aa0bb11-2222-4333-8444-555566667777 cd: PASS deploy-canary 8c1af0e2-7b3d-4c1a-9e20-6f0a1b2c3d4e ``` The command then waits until five `GET /` requests in a row, sent with `Cloudflare-Workers-Version-Overrides: ply=""`, come back with `x-ply-version` equal to the new id. Fail, pin never stuck. The 0% version stays in the deployment. The old version is still at 100%. Traffic was not raised, and it was not rolled back, because there is nothing to undo: ```text cd: FAIL deploy-canary: pinned requests were not answered by 8c1af0e2-7b3d-4c1a-9e20-6f0a1b2c3d4e; left the 0% canary in place ``` Fail, the deployment after create is not 0/100. The previous version is put back at 100% alone: ```text cd: FAIL deploy-canary: 8c1af0e2-… is at 10%, wanted 0% ``` ## 3. Smoke Pinned (the canary). Every request sends the version override, and every response must carry that same id in `x-ply-version`: ```sh npm run cd -- smoke --version 8c1af0e2-7b3d-4c1a-9e20-6f0a1b2c3d4e ``` Live (after the ramp, or any time you want to see who is actually serving). No override. `x-ply-version` must still be that id, which is how you know traffic moved: ```sh npm run cd -- smoke --expect-version 8c1af0e2-7b3d-4c1a-9e20-6f0a1b2c3d4e ``` The smoke creates an **empty** repo in `ply-loadgen` (prefix `ply-web-cd-`). Repo create returns a token. That auto-created token is revoked before the smoke mints its own write token, and the revoked bearer is checked with a `GET` of the repo's `info/refs`: Artifacts must answer 403. The smoke then mints a write token and replays the git lifecycle over `/v1`. It does not seed the repo with stock git. The minted token is revoked after those requests finish and checked the same way. Each check is a `request` event (method, url, status). The token is not in the event, the URL, or the run directory. On a pass the repo is deleted. The delete is idempotent: a timeout or other transient error is retried with backoff, and the bound is `cleanup.attempts` in the policy (the wait before retry n is `cleanup.backoffMs * 2^(n-1)`). A 404, `[10200]`, or `Repository not found` means the repo is already gone. Each attempt's latency is a `repo-delete` observation. The measured delete on run `2026-11-01T17-29-43-122Z` (web `9558b10`) took 103s and returned `APIError "timeout"`, then the retry was `404 [10200] Repository not found`. If the delete still cannot be confirmed, the repo name is a `cleanup_leak` on `verdict.json` and in the ledger. That record is informational: a release that passed correctness still passes. `npm run cd -- sweep` deletes leaked `ply-web-cd-*` repos in the namespace older than `cleanup.staleMinutes`, and the next smoke sweeps the same set before it creates a repo. On a fail the repo name is in the verdict and the token is still revoked. A request that gets no response, a 403 naming that token as invalid or expired, or a revoke that does not stick as 403, stops the run as `harness_fault`. The split stays where it is. The operator decides what to do next. There is no retry and no rollback for that fault. A 409 is a pass when that step expects it. The ones that matter after the current write API: - Cherry-pick or revert of a patch that is already on the target, with the same adds and deletes, is `200` `empty`. - Cherry-pick of that patch onto a target that also has its own extra file is `409` `merge_conflict`. `allowEmpty` does not turn that into a commit. Revert of the picked commit onto a branch that then added an unrelated file is `200` `revert`. - A merge may stamp author and committer as two different people. - `from` and `onto` may be a 40-hex commit. Forty zeros is `400` and pushes nothing. Pass: ```text cd: PASS smoke 8c1af0e2-7b3d-4c1a-9e20-6f0a1b2c3d4e ``` Fail. The step name is first. The repo is left so you can look, without a token: ```text cd: FAIL smoke: cherry-wider: status 200, wanted 409; repo ply-web-cd-20261031210500-ab12cd ``` `smoke.csv` in the run directory is one row per request. `pass` is the step verdict, not the HTTP status. A row with status 409 and `pass` is an expected conflict. Columns: `step`, `method`, `path`, `status`, `ms`, `pass`, `cf_ray`, `x_ply_request_id`, `x_artifacts_ray`, `x_ply_version`, `x_ply_build`, `server_timing`, `detail`. The token is not in the file. On a full `npm run cd`, a smoke failure leaves the canary at 0%. The ramp does not start. Nothing is rolled back: the previous version is already at 100%. The smoke CSV is still written. Telemetry for that canary does not run. ## 4. Telemetry ```sh npm run cd -- telemetry --version 8c1af0e2-7b3d-4c1a-9e20-6f0a1b2c3d4e \ --since 2026-10-31T21:05:00Z \ --report ~/.local/state/ply-cd/runs/2026-10-31T21-05-00-123Z/smoke.json ``` `--since` is the start of the smoke window (ISO-8601, epoch milliseconds, or epoch seconds). The report is the smoke JSON from the same run. The check queries `cf o11y telemetry query` for that version in that window. It fails when any of these are true: - an exception, a non-ok invocation, a 5xx, or an error log on that version - a smoke id logged by a different version - smoke ids missing from telemetry (no `ply.request` log, or no trace) above `telemetry.maxLossRate` in `scripts/cd/cd.policy.json` (default 0.5). Workers Observability loses whole invocations in bursts, so loss at or below that rate is informational: the telemetry stage records the count, rate, and every missing id in `telemetry.md`, `telemetry.json` (`loss`), a `telemetry-loss` observation in `events.jsonl`, and the debrief's Telemetry loss section. An id the smoke did not see answered by this version is never waived. Status mismatches, 5xx, exceptions, error logs, and burn stay hard gates regardless of loss - a response status the smoke did not expect - an exception, a 5xx log, a non-ok outcome, or an error log - wall, CPU, or upstream `fetch` span p50/p99 over the ceilings below, on samples the query returned A 409 the smoke expected is not an unexpected status. A 500 the smoke did not expect is. On the pinned window (not the live one after 100%) it also checks ply's own spans (issue #67). The smoke's `edit-traced` step is a single-file find/replace on the `squash` branch (`POST /v1/edit` with `path`, `find`, `replace`), which ply runs as `editFile`. Every ply span of a `/v1` request is a Workers span under the request span (`src/edge-trace.ts`), so that request's trace must hold `git.edit`, `git.edit.tree`, `git.edit.blob`, `git.upload-pack fetch`, and `git.receive-pack` besides the platform's own `POST` and `fetch` spans. The check reads the same span-events query (`$metadata.type = span`), joins on the request's `cf-ray` from the smoke report and then on the `traceId` those events carry, and re-queries on the telemetry polls (5s, 10s, 15s, up to 60s) while names are missing, because span events of one trace are ingested independently. Source: `scripts/cd/span-check.ts`. - Every name is there: a second line, `spans: edit-traced trace has git.edit, …`. - The trace is there and a ply span is not: `cd: FAIL telemetry: spans: edit-traced trace is missing ply spans ; saw (#67: ply spans are not reaching Workers tracing)`. That is a worker fault. The canary stays at 0% and the ramp does not start. - No span event joins to the request within the wait, or the report has no `edit-traced` row (an older `smoke.json`): `harness_fault`, with `spans: no span event joined to edit-traced …` as the reason. The split is unchanged. The result is `spans` in `telemetry.json` and the last line of `telemetry.md`, and a `ply_spans` rule on the telemetry decision in `events.jsonl`. It decides only when every other telemetry check passed. Pass prints the verdict line, then: ```text telemetry: 40/42 matched, 2 lost (tolerated) spans: edit-traced 3f9c2a1e-… trace 6ae804bdd2731f488b904cf72e929540 has git.edit, git.edit.tree, git.edit.blob, git.upload-pack fetch, git.receive-pack (14 span events) cd: PASS telemetry 8c1af0e2-7b3d-4c1a-9e20-6f0a1b2c3d4e ``` Fail: ```text cd: FAIL telemetry: telemetry: 0/42 matched FAIL: 42 lost without x-ply-version … recorded by the smoke ``` `telemetry.md` in the run directory is that verdict plus the percentiles. `telemetry.json` is the machine-readable pass or fail. The ramp will not move traffic unless that file is a pass for this version. Ceilings for this smoke (a tiny repo, cold isolates included; tighten them after a few real runs): | | p50 | p99 | | --- | --- | --- | | Worker wall | 4000 ms | 25000 ms | | Worker CPU | 250 ms | 3000 ms | | Artifacts `fetch` span | 2000 ms | 15000 ms | Span events have `$metadata.requestId` null. They carry `traceId` and `rayId`. The smoke CSV records each request's `cf-ray`, and the join uses that ray, then `traceId` when the ray does not match. A request id still on a span is only a last resort. The traces view returns one summary row per trace (`traceId`, a span count, a duration, a root span name) and no child spans. Outbound `fetch` durations come from a second query, `view: events` with `$metadata.type = span`. On those events the duration is `source.durationMS` and an Artifacts call has `source.name` `fetch` with a path ending in `git-upload-pack`, `git-receive-pack`, or `info/refs`. A check the harness cannot evaluate is `harness_fault`: the query threw, the span join matches implausibly few of the invocations the worker logged, or the span query came back as trace summaries. The split stays where it is. There is no rollback and no retry. That result is not a worker fault. A 5xx, an exception, an error log, or a non-ok outcome is a worker fault, because the value was observed. A wall, CPU, or fetch percentile over the ceiling above is a worker fault only when the policy's latency mode is `gate`. The checked-in policy is `informational`, so those percentiles are printed and do not fail the stage. A missing latency series is a harness fault only in `gate` mode. ## 5. Ramp Only after the pinned smoke and the telemetry file are both a pass for this version, and the deployment is still new-at-0% / previous-at-100%. An empty split, a percent that is not an integer 0–100, or two copies of one version id is refused before `cf` is called: ```sh npm run cd -- ramp --version 8c1af0e2-7b3d-4c1a-9e20-6f0a1b2c3d4e \ --previous 1aa0bb11-2222-4333-8444-555566667777 \ --verdict ~/.local/state/ply-cd/runs/2026-10-31T21-05-00-123Z/telemetry.json ``` Without `--verdict`, or with a verdict that is not a pass for that version, the command exits and does not change traffic. The checked-in steps are `10, 25, 50, 75, 100` with dwells `30000, 25000, 25000, 25000, 40000` ms (40000 each until 2026-11-08). The list is only in the policy file, and `calibration["steps[*].dwellMs"]` there is the derivation: at 50 probes/s the 10% step issues 1500, which at the worst measured 10% share (0.0874, n=10 steps in the ledger) is 131 new-version requests against `split.minNewRequests` 100; the 25/50/75% steps issue 1250, 25% above `probe.minSamples` 1000; the 100% step keeps 40 s because it runs beside the live checks (median 49 s over the 10 promote runs) and a shorter dwell would not end the run sooner. After the dwell the step re-queries its window every `ingestPollMs` (`3000`) until the Worker has logged 98% of the probes that were answered, with `ingestWaitMs` (`20000`) as the ceiling: across the 10 polled promote watches, 98% had landed 8.6–19.2 s after the dwell end (median 12.6 s). On the ramp-only runs of 2026-11-08 up to 17-17-40-415Z the 10–75% steps stopped rising at about 93% of issued probes, so the poll ran to the ceiling. That was `queryCapturedSliced` deleting rows that shared a non-unique `$metadata.id` (fixed in ace0eb9), not ingest. A 30000 ceiling (0016dae) only added flat polls. On 2026-11-08T18-30-12-168Z (d3a35bb1), after the fix, every responded probe of the 10–75% steps has exactly one external fetch row (1499/1250/1250/1250, 0 absent). The landed check compares all external fetches with probes, so other traffic counts toward it: that run also carried about 8 req/s of `GET /r/fresh-*`, and its polls stopped at agg 1557/1245/1277/1447 against an estimated 1740/1450/1450/1450 external fetches in the windows (probes plus that traffic). Smoke id-matching still uses `TELEMETRY_WAIT_MS` (`60000`); that budget is not the ramp wait. `warmupMs` is `10000`: after the split is applied the step pauses 10 s before the probes, and that pause is not part of the judged window. It was `0` until 2026-11-06, when two ramps of 86456068 (runs 20-01-37Z, 20-11-49Z) rolled back at 100% on routed share 98.84% and 98.74%: 24 and 25 old-version requests, every one within 4.6 s and 3.7 s of the 100% deployment, none after, 0 errors (the split takes a few seconds to reach every edge). The only pinned traffic is the 0% gate: the lifecycle smoke and the deploy check, both sent with `Cloudflare-Workers-Version-Overrides`. From the first ramp step on, probes are ordinary requests. They do not set that header, and they do not pin the old version to balance the arms. Right now those requests are the only load. Real users would be judged by the same telemetry. During the probe the ramp calls at `probe.rps` (25) ticks per second, once per entry in `probe.calls`. Each call's `path` is the request path and query, and the engine sends that string as written. The checked-in mix is `GET /` (the usage text, no bearer) and `GET /v1/discover?remote=https%3A%2F%2F5ad2236b39de4c8bd8f6270ea63f3ea9.artifacts.cloudflare.net%2Fgit%2Fpublic-v0%2Fply.git` (the public `ply` mirror, with the bearer from `PLY_TOKEN_FILE`). That is 50 requests per second. The scheduler is open-loop: it issues on the period whether or not earlier probes have returned, and the dwell ends on the wall clock (`dwellMs`), not when the in-flight set settles. Each probe has its own deadline, `probe.timeoutMs` (2000). Hitting it is `timed_out`, which is not a transport error. Outstanding probes are capped at `probe.maxInFlight` (50, one second of the old tick cap times the two calls). A tick that would pass that cap is `shed` and counted, and the clock keeps going. There is no `/health` route. The response's `x-ply-version` and latency are on the request event. A lost probe has no version. A probe that gets no response (no status and no ray: a client abort, a network failure, or `timed_out`) is a probe loss, not a server error. A shed was never sent, so it is not in the loss denominator. The step records `probe_loss` (count, rate, and ids), `timed_out`, and `shed` on the ramp line, as a `probe-loss` observation in `events.jsonl`, and the loss on `verdict.json`. Loss at or below `probe.maxLossRate` stays in the sample only as loss: correctness and availability use the probes that responded, and that count still has to reach `burn.minSamples`. Fewer responded probes than that is `insufficient_samples`, not a pass. The loss-rate gate cannot fail a step until the issued count reaches `probe.minSamples` (1000). Above that, a loss rate above `probe.maxLossRate` is `harness_fault`. A lost probe has no version, so a window whose responded probes show the same stall on both versions — or whose losses are `timed_out` or `shed` while both versions stayed healthy — is harness/network loss. That window is re-run, up to `probe.windowAttempts` (2), instead of rolling back a version with zero server errors. When those attempts are exhausted the step stops as `insufficient_samples`. A window cut short because this client stalled (`harness_unhealthy`: the box itself froze, not the Worker) is likewise unjudged. A 5xx on a responded probe inside that window, or inside a window too small for the loss-rate gate, is still a rollback. Above 0%, the step's split write has already moved traffic, so every unjudged step (`insufficient_samples`, `harness_unhealthy`, and a telemetry-unavailable `harness_fault`) settles it through `probe.onHarnessFault` (ply-web#145): `rollback` and `telemetry` restore `previous` at 100%, and `hold` writes the split back to the last fully passed step, or restores `previous` when no step passed. At 0% there is nothing to settle and the reason ends `held at 0%`. The ramp state's reason ends with where traffic is (`rolled back to previous`, `held at N%`); if that rollback or hold write fails it ends `…failed: ; still at N%`, `rolledBack` is false, and the new version is still serving N%: treat it like a failed rollback. For a real probe-loss `harness_fault`, `probe.onHarnessFault` says: `telemetry` (the checked-in choice since 2026-11-08) records a probe-loss fault and judges the step on the new version's telemetry instead, and restores `previous` only when that telemetry is also missing; `rollback` restores `previous` at 100% (the checked-in choice until 2026-11-08); `hold` returns to the last fully passed split. The verdict kind stays `harness_fault`, distinct from a correctness fail. The checked-in rate is 1/790, the single client abort on the 100% step of run `2026-11-01T17-06-08-780Z` (web `789ba93`, ply `1743461`), with no headroom above that measurement. The scheduler bounds are from the 50% step of run `2026-11-01T19-40-35-485Z` (web `22bfb79`): healthy p50 41–101ms on both `4f3d7acf` and `19ee3cab`, then a client stall at about 69830ms, 10 of 128 never answered, and a decision about 3m14s after a 40s dwell. `timeoutMs` sits above 10× that 101ms p50 and below the stall. `minSamples` sits above 128 and below the healthy steps' about 1990 probes. Round bounds that are not themselves a percentile are marked PRIOR on `probe.schedule`. Before each percentage change the ramp reads the deployment and writes only when it is still the previous step's split. Otherwise it stops with `state_conflict` and does not roll back. Five dwells are 145 s, five warm-ups 50 s; the polled ingest is 10–24 s a step. See [Duration](#duration) for measured runs. The original 20s wait was the tail measured on run `2026-11-01T00-10-02-620Z`: the query ran about 1.4s after the dwell and the newest event was still 12640ms behind the dwell end at 1% and 17313ms behind it at 5%. `20000` is above that. Those two steps are no longer in the file; the lag is. Then it reads one `cf o11y` events query for the whole worker over that step's window, and partitions it by `$workers.scriptVersion.id`. The new version, the old version, and the aggregate (every invocation in the window, which is what users feel) are judged separately. Child spans are a second events query with `$metadata.type = span`, not the traces view. Span events have no `$workers.scriptVersion.id` and a null `$metadata.requestId`, so they are queried by service and joined on `rayId`, then `traceId`. The step decision is the policy. A signal set to `hard` rolls back. The same signal set to `informational` is named on the advance reason and does not roll back. Burn rate is `(bad / total) / (1 − target)`. Definitions are in [Metrics](#metrics). The checked-in file: | Signal | Checked-in mode | What happens | | --- | --- | --- | | Smoke | `hard` | The 0% lifecycle only. While the ramp query returns, a probe's HTTP status is not a gate, except a 5xx in a window the loss-rate gate will not judge (a harness stall, or fewer issued probes than `probe.minSamples`): that 5xx rolls back without waiting for the query. When the query is lost, a 5xx on a responded probe is the correctness signal | | Exception, 5xx, error log, non-ok outcome | `hard` | Any, on the new version. One request is enough. The same counts on the old version and across the worker are named on the reason as `informational` and never roll back (`scripts/cd/slo.ts` `judgeStep`) | | SLO burn | `hard` | On the new version: `bad/total` divided by the error budget is above `burn.fastBurn` (**14.4**), and that SLO saw at least `burn.minSamples` (**100**) requests. The old version and the aggregate are judged the same way and reported as `informational` | | Routed share | `rollback` | `routedNewShare` outside the binomial band. `split.onMismatch` may instead be `hold`, which leaves the split | | Thin evidence | `hold` | Fewer than `split.minNewRequests` (**100**) new-version invocations, or no invocations at all. Outcome `insufficient_data`. `onInsufficient: advance` moves on and still says so | | Latency | `informational` | Never, under this file. Wall p99 and CPU p99 are on the line and on the ramp observation. `off` omits them from the line. `gate` needs `minSamples`, `p99Ratio` (> 1), and `effectFloorMs`, and rolls back only when both sides have that many wall samples and p99 clears both thresholds | | Query failed | retry, then probes | Each `cf o11y telemetry query` is retried up to `telemetry.query.attempts` (3). The wait before retry n is `telemetry.query.backoffMs * 2^(n-1)`. A later success is a normal sample. If every attempt fails and the probe window already passed, the step uses those status codes: a 5xx rolls back, and a clean window advances. The lost series is `informational` on the reason and a `telemetry-query` observation per attempt. It is not stored as zero errors, and it is not a `harness_fault`. With no probe sample the step is still `harness_fault`, and above 0% the split follows `probe.onHarnessFault` | ### How many requests that is Correctness here is zero tolerance: one exception, 5xx, error log, or non-ok outcome fails the scope that saw it. With zero observed errors in `n` requests, the error rate `p` at which a step has a 95% chance of seeing at least one is ```text (1 − p) ^ n = 0.05 p = 1 − 0.05 ^ (1 / n) ``` That `p` is the minimum detectable error-rate change from a clean baseline. The rule of three, `3 / n`, is the same bound at these sizes. The checked-in probe schedules `25 × 2 × 30s = 1500` requests at the 10% step, `25 × 2 × 25s = 1250` at 25/50/75%, and `25 × 2 × 40s = 2000` at 100%. The smallest open step is 10%, so the new version's expected count is `1500 × 0.10 = 150`. At 100% the old version is not allocated. | Scope | Expected n | Detectable error rate | | --- | --- | --- | | New version at 10% | 150 | 1.98% | | Floor, `minNewRequests` and `burn.minSamples` | 100 | 2.95% | | New version at 25% | 312 | 0.96% | | New version at 50% | 625 | 0.48% | | Aggregate, and the new version at 100% | 2000 | 0.15% | A step with fewer than 100 new-version invocations does not advance. That floor is the 2.95% row. A normal 10% step sees about 150 and can catch a 2% regression on the new version (it was 200 and 1.5% at the old 40 s dwell). The aggregate, which is what users feel, would resolve 0.15% in the same window, but it is informational: only the new version's signals roll back. Per-version is what keeps a regression on the small arm from being diluted into the aggregate. The share check is the same `n`. For target proportion `p` and `z = 3`, the band is `p ± z √(p(1−p)/n)`. At 10% and `n = 1500` that is 7.68%..12.32%. At 100%, `p(1−p)` is 0, so the share has to be 100%. Counts come from external fetch invocations, not from the probe's own bookkeeping and not from named entrypoints. `test/fixtures/cd/routed-share.json` is a synthetic 10% window of this shape: 200 of 2000 advances, 1000 of 2000 rolls back, 40 of 400 is `insufficient_data`. This probe is a scaled model of production. At about 800 requests per second, a 40s dwell already holds 32000 requests and the 10% arm holds 3200, which can see about a 0.09% error rate. Retune `probe.rps`, `burn.minSamples`, and `split.minNewRequests` in the policy. When real traffic is the load, `probe.rps` can go to 0 and the same gates read whoever called the worker. That is a config change. `samples=` on a ramp line counts external fetch invocations, new / old, in this window. An external fetch is `$metadata.type` `cf-worker-event`, `$workers.eventType` `fetch`, and `$workers.entrypoint` absent or `default`. A named entrypoint (`FlagsCache`, `GitCache`) is an internal `ctx.exports` call. It has its own ray and request id. Rows are deduplicated on `$metadata.id` + `requestId` + `traceId` + `spanId` (`rowKey` in `scripts/cd/telemetry.ts`) because `$metadata.id` alone is a millisecond plus a per-invocation row position: distinct invocations in the same millisecond share it, and keying on it alone deleted 4–10% of rows at every partial ramp step on 2026-11-08 (runs 16-58-16-960Z, 17-07-46-259Z, 17-17-40-415Z). The event page cursor (`pageCursor`) likewise steps back one id, because `offset` returns ids strictly below it and a same-id group split by a page boundary would otherwise lose its remainder. Those counts are `internal_next` / `internal_previous` on the ramp line and `internalInvocations` on the ramp observation. They are not part of `samples`, `agg`, error rates, latency samples, or the share. `agg=` is the external count, the share denominator. Each external request also writes a `ply.request` log, so the event page is larger than the sample count. The query asks for `limit` 1000 and follows the cursor. On the 2026-11-01 run a page came back short because the index held about 28s of the 40s dwell, not because of a 1000-row cap. The window's `to` is the dwell end plus `60000` ms so the tail is inside the query. The ingest wait is what keeps the next step's probes out of this step's judged window. At 100% the old version is dropped, so its wall and CPU numbers are the last step whose old version had at least 5 wall samples. That line includes `previous_sample=carried`. If no earlier step had them, 100% still advances when the new version is clean. Comparing versions on trace latency, and leaving cold starts out of that compare, waits until the rollout itself is trusted. The events this ramp reads have no cold-start or isolate-start field. `$workers` carries `wallTimeMs` and `cpuTimeMs`, plus outcome, execution model, and version. This policy does not try to drop cold starts. Each step prints one line to stdout and appends the same line to `ramp.jsonl` in the run directory. Relay those lines as they appear. A clean ramp: ```text ramp step=1/5 percent=10 previous=90 samples=200/1800 error_rate=0.0000/0.0000 routed_share=10.00 target=10 tolerance=7.99..12.01 agg=2000 p99_wall_ms=88/74 p99_cpu_ms=12/11 latency=informational decision=advance ramp step=2/5 percent=25 previous=75 samples=500/1500 error_rate=0.0000/0.0000 routed_share=25.00 target=25 tolerance=22.10..27.90 agg=2000 p99_wall_ms=90/76 p99_cpu_ms=12/11 latency=informational decision=advance ramp step=3/5 percent=50 previous=50 samples=1000/1000 error_rate=0.0000/0.0000 routed_share=50.00 target=50 tolerance=46.65..53.35 agg=2000 p99_wall_ms=94/78 p99_cpu_ms=13/12 latency=informational decision=advance ramp step=4/5 percent=75 previous=25 samples=1500/500 error_rate=0.0000/0.0000 routed_share=75.00 target=75 tolerance=72.10..77.90 agg=2000 p99_wall_ms=101/80 p99_cpu_ms=14/12 latency=informational decision=advance ramp step=5/5 percent=100 previous=0 samples=2000/0 error_rate=0.0000/- routed_share=100.00 target=100 tolerance=100.00..100.00 agg=2000 p99_wall_ms=114/84 p99_cpu_ms=15/13 latency=informational decision=advance previous_sample=carried cd: PASS ramp 8c1af0e2-7b3d-4c1a-9e20-6f0a1b2c3d4e ``` How to read a line: | Field | Meaning | | --- | --- | | `step=2/5` | Second percentage. 1-based. The count is the policy's step list. | | `percent=25` `previous=75` | Configured new-version share, and what is left on the old one. At the last step `previous=0`. | | `samples=500/1500` | External fetch invocations in this step's window, new / old. Named entrypoints are not included. | | `error_rate=0.0000/0.0000` | Non-ok rate, new / old, four decimal places. | | `routed_share=25.00` | `100 × new / agg`. Null (`-`) when `agg` is 0. | | `target=25` | The step's percent. The same number as `percent`. | | `tolerance=22.10..27.90` | The binomial band in percent, `z` standard errors at this `agg`. `-` when `agg` is 0. | | `agg=2000` | External fetch invocations in the window, every version. | | `p99_wall_ms=90/76` | Wall-time p99 in milliseconds, new / old. `-` means that side had no sample. Omitted when latency mode is `off`. | | `p99_cpu_ms=12/11` | CPU p99, new / old. Informational under the checked-in policy, same as wall. | | `latency=informational` | The percentiles are recorded. They do not roll back. `gate` would. | | `decision=advance` | No hard signal fired, the share is inside the band, and the new version had enough invocations. | | `decision=rollback` | A hard signal fired, or the share is outside the band and `onMismatch` is `rollback`. Traffic goes back to the previous version. | | `decision=insufficient_data` | The new version had fewer than `minNewRequests` invocations (or the window had none) and `onInsufficient` is `hold`. The split stays. | | `decision=hold` | The share is outside the band and `onMismatch` is `hold`. The split stays. The checked-in file rolls back instead. | | `reason="…"` | Present on rollback, hold, insufficient evidence, `harness_fault`, and an advance that names an informational signal. Relay it with the line. | | `internal_next=FlagsCache:92` | Named-entrypoint fetches on that version, `name:count` separated by commas. Omitted when there were none. Not a gate. | | `previous_sample=carried` | The old-version p99 is from the last step that had at least 5 wall samples. The sample counts on that line are still this window. | A large wall shift advances. The 2026-11-01 5% walls (`test/fixtures/cd/ramp-5pct-walls.json`, run `2026-11-01T00-10-02-620Z`) do not roll back on latency. So does a step whose p99 is 27ms against 12ms. `insufficient_data` now means the new version was not seen enough times. It is not the old latency compare. Relay it: nothing was rolled back, and the split did not advance. An exception on the new version, even with one request: ```text ramp step=1/5 percent=10 previous=90 samples=1/1999 error_rate=1.0000/0.0000 routed_share=0.05 target=10 tolerance=7.99..12.01 agg=2000 p99_wall_ms=40/70 p99_cpu_ms=8/9 latency=informational decision=rollback reason="1 exception on the new version" cd: FAIL ramp: 1 exception on the new version ``` A 10% step whose telemetry shows about half the traffic on the new version: ```text ramp step=1/5 percent=10 previous=90 samples=1000/1000 error_rate=0.0000/0.0000 routed_share=50.00 target=10 tolerance=7.99..12.01 agg=2000 p99_wall_ms=80/80 p99_cpu_ms=12/12 latency=informational decision=rollback reason="routed share 50.00% is outside 7.99..12.01 of target 10% (n=2000)" ``` A ramp query that times out is retried. Run `2026-11-01T19-17-18-170Z` (web `07568e0`) lost one `cf o11y telemetry query --body` at 100% to `APIError "timeout" Envelope Error` after every probe had returned, and that single miss rolled the release back. A retry that returns a page is not a fault. If the bound is exhausted and the probes were a full clean sample, the step advances and names the loss: ```text ramp step=5/5 percent=100 previous=0 samples=-/- error_rate=-/- routed_share=- target=100 tolerance=- agg=0 p99_wall_ms=-/- p99_cpu_ms=-/- latency=informational probe_loss=0/790 decision=advance reason="informational: telemetry query lost; 790 responded probes, 0 server errors; …" ``` A 5xx on one of those probes still rolls back, because that status was observed. Routed share, burn, exceptions, and error logs from the missing query are not filled in as zero. With no probe sample at all, the step is `harness_fault` and above 0% follows `probe.onHarnessFault` (checked-in `telemetry`: with no telemetry either, roll back to `previous`). If the rollback request itself does not stick, the last line says so and names the version that should be at 100%. Do not treat that run as rolled back. Put it back yourself with the command in the next section. ## 6. After 100% `npm run cd` does not stop at 100%. It smokes again with **no** version override (`--expect-version`) once unpinned traffic has converged on the new version, then runs telemetry (with the rate gate) on that live window, and runs the CDP visual smoke (`scripts/visual-smoke.mjs`, 375 + 1280 px, light + dark, every page pinned to the new version; screenshots in `/visual/`, verdict in `/visual.json`). All have to pass. They start the moment the 100% split is written and run during the 100% step, which is the live watch; the run waits for both. A Worker failure in any of them rolls back; a harness fault alone (no Chrome, a lost query) stops with the new version live. A visual smoke with no Chrome (`CHROME` unset and no macOS Chrome) is recorded as skipped. Events written while the checks run in parallel may carry the label of the stage that last started; the stage timings in the manifest are exact. Pass is the same `cd: PASS ply web ` as the whole run. Fail rolls back to `previous` at 100% and exits: ```text cd: FAIL smoke: read-readme: x-ply-version 1aa0bb11-…, wanted 8c1af0e2-… cd: FAIL telemetry: telemetry: … 3 5xx ``` Organic 5xx on the new version during the live window fails this telemetry check and rolls back. The canary check was only the pinned smoke ids. The live check is every invocation of that version in the window. ## Rollback Any time, including when a ramp or a live check did not put the old version back: ```sh npm run cd -- rollback --to 1aa0bb11-2222-4333-8444-555566667777 ``` `--to` is the **good** version. It becomes the only version, at 100%. The command reads the deployment back and will not print a pass unless that is what it sees. ```text cd: PASS rollback 1aa0bb11-2222-4333-8444-555566667777 cd: FAIL rollback: rollback to 1aa0bb11-… failed: … ``` ## The run directory `~/.local/state/ply-cd/runs//` (`PLY_CD_RUN_DIR`; an older checkout used `cd-runs/`, which is still gitignored): | File | What you open it for | | --- | --- | | `verdict.json` | `{ "kind": "pass", "version": "…", "failures": [], "probe_loss": [], "cleanup_leak": [] }`, `kind: "fail"` with `stage`, `reason`, and `rolledBack`, `kind: "insufficient_samples"` when responded probes missed `burn.minSamples`, or `kind: "harness_fault"` / `kind: "state_conflict"`. A ramp `harness_fault` names whether traffic was rolled back. `probe_loss` is one record per ramp step (count, rate, ids). `cleanup_leak` is scratch repo names whose delete could not be confirmed. It does not fail a pass | | `smoke.csv` / `live-smoke.csv` | One row per HTTP call. `pass` is the step. `detail` is why a row failed. | | `smoke.json` / `live-smoke.json` | The ids telemetry asks for, plus the scratch repo name if it was kept | | `telemetry.md` / `live-telemetry.md` | The verdict line and the percentiles | | `telemetry.json` | The pass/fail the ramp reads. A ramp with a fail here does not change traffic. | | `ramp.jsonl` | One JSON object per step: `{ "line": "ramp step=…", "probe_loss": { "percent", "count", "rate", "total", "responded", "ids" } }` | | `events.jsonl` | Append-only log. One typed event per line. This is the record a debrief reads | | `manifest.json` | sha256 of every other file, written after `run_finished`, and `blobs`: `dropped` (a clean pass) or `kept`. The pipeline does not modify the directory after this; only retention (below) does | | `blobs/.gz` | Kept only when the run failed or rolled back. Every command's stdout and stderr (redacted) and every observation's body, gzipped and stored once by content: a command event's `stdoutDigest` is the file name. A ramp observation's body lists the digests of the telemetry pages it read instead of copying them | | `visual/` | CDP screenshots | | `pruned.json` | Present only on a compacted run: its status, what retention removed, bytes freed | The gates read only an observation's `measurement` (counts, errors, latencies, shares), and that is in `events.jsonl`, so a clean pass loses nothing it decided on by dropping `blobs/`. Before 2026-11-07 every telemetry poll wrote its full stdout to `captures/.stdout` (up to 2.7 MB) and again, escaped inside a JSON string, to `raw/.json` (up to 29 MB): a ramp run was 120-135 MB, and about 50 runs filled the disk twice. ### Retention Every run that creates a directory prunes `PLY_CD_RUN_DIR` first (`scripts/cd/retention.ts`, `RETENTION`) and prints `retention: N whole, N compacted, N removed, X MB freed`. A run's status is the last `run_finished` in its `events.jsonl`; a rollback is a failure, and an unfinished run older than 24 h counts as a failure (younger ones, and the run being resumed, are not touched). | Tier | Which runs | What stays | | --- | --- | --- | | Whole | The newest 10 passes and the newest 10 failures | Everything. A pass also loses any leftover `blobs/`, `captures/`, `raw/` | | Compacted | The next 500 runs, newest first | Top-level summary files only (`verdict.json`, `telemetry.*`, `smoke.*`, `live-smoke.*`, `visual.json`, `deploy.json`, `ramp.*`): tens of KB. `events.jsonl` (its lines are in the ledger), `manifest.json`, and every subdirectory go; `pruned.json` records it | | Removed | Older | The directory and its `.debrief/`. The ledger still has every event | `ledger.jsonl` is never pruned. `liveSmokeAlreadyPassed` reads `live-smoke.json`, which compaction keeps; `--sha` resumes only an unfinished run, which is whole for 24 h. A debrief of a pass or compacted run lists the captures it cites as not kept. ## When it fails | You see | Traffic now | What you do | | --- | --- | --- | | `cd: FAIL bump:` | Unchanged | Fix the tree or the gate. Nothing was uploaded. A failed pin does not move `bump-ply` unless you were already on it. | | `cd: FAIL deploy-canary: not-on-origin-main` | Unchanged | Check out `origin/main`. A pin commit on `bump-ply` is not deployed from there. | | `cd: FAIL deploy-canary:` and the line says the canary was left at 0% | Old at 100%, new at 0% | Read the pin failure. Rollback only if you want the 0% slot cleared: `npm run cd -- rollback --to `. | | `cd: FAIL deploy-canary:` after a bad split | Previous at 100% if the rollback landed | Confirm with `cf workers deployments list --worker ply`. | | `cd: FAIL smoke:` or `cd: FAIL telemetry:` before any ramp line | Still 0% / 100% | The ramp did not start. The scratch repo name is in the smoke failure. | | A ramp line with `decision=advance`, including one that prints `latency=informational` | The next percentage | Relay the line. Wall and CPU time did not roll anything back. | | A ramp line with `decision=insufficient_data` or `decision=hold` | The percentage where it stopped | The split was not advanced and was not rolled back. Thin evidence, or a share miss whose policy says hold. Read the reason. | | A ramp line with `decision=insufficient_samples` or `decision=harness_unhealthy` | `previous` at 100% (`telemetry` and `rollback`), or the last fully passed split under `hold` (`previous` when none passed); at 0%, unchanged | The step was not judged (a probe window under `burn.minSamples`, a harness stall whose re-runs are exhausted, or a client stall), so its traffic was settled: the reason ends `rolled back to previous` or `held at N%`. If it ends `…failed: ; still at N%`, the new version is still serving N%: **not** safe, run `npm run cd -- rollback --to ` and read the deployment list. | | A ramp line with `decision=rollback`, then `cd: FAIL ramp:` | Previous at 100% | Relay the line. The new version is off traffic. | | `cd: FAIL ramp:` and the text says rollback failed | **Not** safe | Run `npm run cd -- rollback --to ` and read the deployment list. The id to restore is the one in that sentence. | | `cd: FAIL smoke:` or `cd: FAIL telemetry:` after ramp lines that reached `percent=100` | Previous at 100% if rollback landed | Same as a ramp rollback. The live check is what rejected it. | | `cd: state_conflict lock:` | Unchanged | Another run holds the deployment lock. Wait for it; the line names its pid and worktree. | | `cd: state_conflict deploy-canary: … already at 0%` | Unchanged, nothing uploaded | Someone else's version is in the 0% slot. Ask whose it is; pass `--replace-canary ` only when it is a leftover. | | `cd: harness_fault` | On a ramp step above 0% whose telemetry query never returned with no probe sample, `previous` at 100% (`telemetry` and `rollback`), or the last fully passed split under `hold`. A probe-loss fault alone does not move traffic under the checked-in `telemetry`. Elsewhere, the split where the run stopped | The version was not blamed. A probe loss above `probe.maxLossRate` once `probe.minSamples` issued probes were counted, a 403 on the token this stage minted, or a telemetry query that had no probe sample, or a span join. A stalled dwell with zero server errors is re-run, and if it is still stalled the step stops as `insufficient_samples` (see that row) instead of being judged a rollback. A lost ramp query after a clean probe window stays on the advance line. The verdict kind is `harness_fault`, not a correctness fail. | | `cd: state_conflict` | The split that was already there | Something else changed the deployment. Do not apply the step again until you have read `cf workers deployments list`. | ## Metrics SLOs are data in `scripts/cd/slo.ts`. A ramp step does not inline the numbers. | SLO | Good event | Target | Window | | --- | --- | --- | --- | | Availability | Not a 5xx and no exception, over `/v1` requests, excluding expected 4xx | 99.9% | 28 days | | Read latency | `GET /v1/*` wall time under 1000 ms | 99% | 28 days | | Write latency | `POST /v1/*` wall time under 2500 ms | 99% | 28 days | | Correctness | Lifecycle smoke: every step the expected status, `git fsck` clean, refs equal | 99.5% | 28 days | The latency ceilings are initial. Tighten them from a real baseline; the 99% target stays. Error budget and burn rate: ```text budget = 1 − target burn = (bad / total) / budget ``` A burn of 1 spends the budget at the planned rate. The ramp fails a step when burn is above **14.4** and that SLO has at least `burn.minSamples` requests in the step window (100 in the checked-in file). 14.4 is the SRE workbook fast-burn multiple: on a 30-day budget, 14.4× for one hour spends about 2%. Our SLO window is 28 days. The step still uses 14.4, not a recomputed multiple. Availability on a ramp step treats every 4xx as expected, so client errors leave the denominator. The correctness smoke still fails an unexpected status at 0%. One 5xx or one exception fails the ramp step even when the sample is shorter than the floor, on the version that served it and across the worker, so the availability burn is the rate view for a window that is already large. Correctness is one lifecycle smoke, not a ramp step. While `gates.smoke` is `hard`, any failed step fails the smoke stage (`decision=stop`, canary left at 0%). `informational` records the failure and continues. The burn from the pass/fail counts is stored on the run. It is what a later alert would page on. It is not a second gate inside the ramp. Wall time against the old version is not a ramp gate unless the policy sets `gates.latency.mode` to `gate`. The checked-in mode is `informational`: the numbers are stored and printed, and a shift does not roll back. Read and write burn still use the ceilings in the table above. Those are burn gates, not a version-versus-version compare. `gate` rolls back only when both sides have `minSamples` wall samples and the new p99 is above `p99Ratio` times the old p99 and more than `effectFloorMs` slower. A shorter sample does not roll back on latency. Trace-based latency is future work, after the rollout mechanics are trusted. Alerting policy for later, not applied on a ramp step. These are the workbook's 30-day windows: | Severity | Burn | Long window | Short window | Budget spent if both fire | | --- | --- | --- | --- | --- | | page | 14.4× | 1h | 5m | about 2% | | ticket | 6× | 6h | 30m | about 5% | | ticket | 1× | 3d | 6h | about 10% | ### Delivery Every event is appended to the run's `events.jsonl` and to the ledger (`PLY_CD_LEDGER`, default `~/.local/state/ply-cd/ledger.jsonl`). `run_finished` carries the fields a delivery report needs: ply sha, ply merge time, ply-web sha, version id, stage timestamps and durations, verdict, rollback flag and reason, smoke pass/fail counts, and per-SLO burn per step. ```sh npm run cd -- report ``` A past run stays in the ledger. To mark one as a pipeline false positive, append a note. This does not rewrite earlier lines and does not open the run directory: ```sh npm run cd -- annotate 2026-11-01T00-10-02-620Z --false-positive "5% ramp Mann-Whitney flagged a sub-millisecond wall-time shift" ``` That run's `run_finished` line still says `fault: worker`. `report` then drops it from the deploy count and from the change-failure rate, and prints the reason under `false positives`. | Metric | Formula | | --- | --- | | Deployment frequency | deploys / days. A deploy is a `run_finished` that reached 100% (`liveAt` set) and whose `fault` is not `harness_fault`. Aborted runs, held runs, and `harness_fault` runs are counted on their own lines and are not deploys. Days are the span from the first deploy timestamp to the last. One timestamp has no rate. | | Lead time | `liveAt − plyMergedAt` on a passing deploy. `liveAt` is when the ramp reached 100%. `plyMergedAt` is the ply commit's committer time from the GitHub API, or null if that fetch fails (the deploy still finishes). Report p50 and p90. | | Change failure rate | rollbacks / deploys. A harness fault is not in either number. A run annotated as a pipeline false positive is not in either number, and `report` lists it on its own. | | Time to restore | `restoredAt − failedAt` for a rolled-back deploy. p50 and p90. | | Stage duration | p50 and p90 of each stage's `durationMs`, across runs. Percentile is `sorted[floor((n − 1) · p)]`. | ## Debrief The event log is the record of the run. Every line is one JSON object: | Field | What | | --- | --- | | `seq` | 1, 2, 3, … inside that run | | `ts` | ISO-8601 with a numeric offset | | `run_id` | The run directory name | | `stage` | `run`, `bump`, `deploy-canary`, `smoke`, `telemetry`, `ramp`, `rollback`, or the live variants | | `kind` | One of the variants below | | `payload` | The variant's fields | | Kind | Payload | | --- | --- | | `lifecycle` | `run_started` (config, node/cf/git versions, ply and ply-web shas, host, and `release` as `{ply, web}` or null), `resumed` (the same run id, the ply sha, and the release pair), or `run_finished` (verdict, rollback, smoke counts, stage timings, per-step burns, live/fail/restore times, `fault`, and the release pair) | | `command` | argv with secrets already replaced by ``, cwd, exit code, duration, sha256 of stdout and stderr, and the paths of those captures | | `observation` | The measurement, the exact `cf o11y` query (or the smoke source) and its window, and `responsePath` of the raw body (`blobs/.gz`; `raw/.json` before 2026-11-07) | | `decision` | `inputs` (one or more earlier observation seqs), `rules` (`name` and `value` from the policy or `slo.ts`, the smoke rule, or `reconcile` with the next stage), `outcome` (`advance`, `hold`, `insufficient_data`, `insufficient_samples`, `rollback`, `stop`, `harness_fault`, `state_conflict`, `cleanup_leak`), and `reason`. `cleanup_leak` names a scratch repo and does not fail the run | | `policy` | `sha256` of the policy file bytes, the path, the body that ran, and the release pair (`ply`, `web`) or null. The debrief prints this first so the rules of that run are the ones you read | | `state_change` | Traffic split before and after, and the deployment id when cf returned one | | `request` | One smoke or probe call: method, url, status, version, latency, `cf-ray`, `x-ply-request-id`, `x-artifacts-ray` | | `error` | An unexpected failure and a JSON context | | `annotation` | Appended to the ledger for a past run: `note` `false_positive` and a one-line `reason`. `npm run cd -- annotate --false-positive "reason"` writes it. Earlier lines stay as they were | A decision cannot be written unless `inputs` cites an earlier `observation`. A `Secret` cannot be serialized: `JSON.stringify` throws, and command argv stores `` instead of the value. Each Secret's text is removed by exact match from every capture and event. An `art_v` plus a version plus `_` is removed the same way, as a backstop, whatever the version number is. The ramp emits `insufficient_data` when the new version was not seen enough times and the policy holds, and `hold` when a routed-share miss is configured to hold. The checked-in file rolls a share miss back and holds on thin evidence. `stop` is a gate that leaves traffic where it is (the pinned smoke failure, while smoke is hard). `rollback` is traffic returned to the previous version. After `run_finished` the run drops `blobs/` on a clean pass, writes `manifest.json` with the sha256 of every other file, and then refuses further appends. Do not edit the run directory; retention compacts it later. A debrief writes beside it: ```sh npm run cd -- debrief ~/.local/state/ply-cd/runs/2026-10-31T21-05-00-123Z ``` That prints `debrief: ~/.local/state/ply-cd/runs/2026-10-31T21-05-00-123Z.debrief/debrief.md`. The markdown stands alone: the policy body and its sha256, a timeline, every decision with the rule and the observations it used, every traffic change, three charts (`split.png` is every `state_change`, including the 0% canary, against `routedNewShare` from telemetry, `latency.png` wall p99 by version labeled with the run's latency mode, `burn.png` burn per SLO with a line at the policy's fast-burn multiple), and links back to `events.jsonl` and to whichever cited blobs were kept (`raw/` and `captures/` in a pre-2026-11-07 run). ## Later This runbook is the operator path. The same stages are what a later driver should call. That driver is a Worker fed by Cloudflare Artifacts repo events, not a GitHub Actions workflow.