Continuous deployment
You ship ply-web from a box where cf is already logged in. One command uploads the commit you have checked out at 0% beside the current 100% version, smokes it pinned with Cloudflare-Workers-Version-Overrides, checks the telemetry of those pinned requests, then ramps it through the steps in scripts/cd/cd.policy.json (10 → 25 → 50 → 75 → 100). Each step is judged on unpinned probes and the new version's own telemetry before the next one is written. During the 100% step the live lifecycle smoke, its telemetry (with the rate-limiter gate) and the CDP visual smoke run in parallel; that step is the live watch. A new-version regression at any step, or a failed live check, puts the previous version back at 100% and the command exits 1. GitHub holds the code and the pull request. It does not deploy.
PLY_TOKEN_FILE=~/.config/ply/worker-token PLY_NODE=<node >= 24.22> env -u CLOUDFLARE_API_TOKEN npm run cd -- --no-bumpThe stepped ramp is the only promotion path (Ivar, 2026-11-08: "Ramp should be the only option."). The direct promote (0% → 100% in one write, then a 60 s watch, the default from 2026-11-06 to 2026-11-08) is gone. --ramp is still accepted so old habits and scripts keep working, and is ignored: the run prints cd: --ramp is ignored: the stepped ramp is the only promotion path since 2026-11-08. A policy file that does not ramp is rejected before anything runs: no steps, an empty list, a single 100 step (the old direct promote), or the removed promote block.
| Stage | What | Typical |
|---|---|---|
| bump | pin check (--no-bump skips) | 0–9 s |
| deploy-canary | build, upload, route 0% beside the 100% version, record version + previous | 12–40 s |
| smoke | pinned lifecycle smoke against the 0% version | 21–50 s |
| telemetry | the pinned requests' logs and spans, plus the rate gate, polled every 3 s until they land | 13–28 s |
| ramp 10/25/50/75 | per step: write the split (~3 s), warmupMs 10 s, dwell (30/25/25/25 s) of unpinned probes, then the step's telemetry polled every ingestPollMs until 98% of answered probes are logged (ceiling ingestWaitMs 20 s) | 58–64 s a step |
| ramp 100 + live checks | the 100% step (dwell 40 s) beside the live smoke, live telemetry and visual smoke, which start the moment the 100% split is written | ~70 s |
The measured end-to-end duration of the first ramp-only run is in Duration.
Duration
End to end (run_started to run_finished in the ledger), passing runs that uploaded a new version:
| Rollout | Runs | End to end | Ramp / promote stage |
|---|---|---|---|
| Stepped ramp, 40 s dwells, fixed 20 s ingest sleep (before 2026-11-08) | 2026-11-06T17-35-09-419Z, 20-24-45-549Z, 21-06-26-841Z | 602, 655, 670 s | 443, 501, 526 s |
| Direct promote, removed (2026-11-06 to 2026-11-07) | 2026-11-06T21-45-51-484Z, 21-56-24-823Z, 2026-11-07T17-51-47-638Z, 18-30-47-501Z | 163, 176, 169, 231 s | 76–84 s |
| Ramp-only, 30/25/25/25/40 s dwells, polled ingest, 20 s ceiling | 2026-11-08T16-58-16-960Z (16cc8e22, previous 296e081f) | 423 s | 325 s |
| Same, 30 s ingest ceiling (0016dae, reverted) | 2026-11-08T17-07-46-259Z (0b0679bd, previous 16cc8e22) | 478 s | 374 s |
Per stage of 2026-11-08T16-58-16-960Z: deploy-canary 30 s, pinned smoke 34 s, pinned telemetry 22 s, then the ramp: 10% 64 s, 25% 58 s, 50% 58 s, 75% 58 s (split write to decision), 100% 69 s with the live smoke (31 s), live telemetry (27 s) and visual smoke (21 s) beside it. Unpinned traffic converged on the new version 1.4 s after the 100% write. The 10–75% steps each ran their poll to the 20 s ceiling at ~93% of issued probes. That ~93% was a pipeline bug, not ingest lag or Workers Observability loss: queryCapturedSliced deduped rows on $metadata.id alone, which is not unique across invocations (ace0eb9). Raising the ceiling to 30 s (run 2026-11-08T17-07-46-259Z) added only flat polls, so it is back at 20 s. That run's 100% step also lost 41/1948 probes to timed_out (2.1%, above probe.maxLossRate 0.13%): under the pre-2026-11-08 onHarnessFault: rollback that would have rolled back a clean version; under telemetry it was judged on telemetry (0 errors in 1806) and advanced.
What can fail a release, and what cannot
- Only the new version's own signals are hard. Errors on the old version (stragglers while the split propagates, or the old arm at any step) and on the aggregate are named on the line as
informationaland never roll back (run 2026-11-06T20-57-44-803Z rolled back a clean version for "1 5xx response on the old version"). - At 100% the routed-share band has zero width, so a share like 98.74% there is split propagation, noted as informational (runs 2026-11-06T20-01-37-842Z, 20-11-49-685Z). The live smoke and the visual smoke still assert
x-ply-version. - Probe loss (client aborts, the probe tail) is the harness:
probe.onHarnessFaultistelemetryin the checked-in file since 2026-11-08, so the loss is recorded and the step is judged on the new version's telemetry. Underrollback, runs 2026-11-06T17-32-11-293Z (6/2000) and 18-44-58-475Z (5/2000) rolled back clean versions on probe-tail loss alone. A Worker 5xx is in that telemetry and still rolls back. A harness stall re-runs the window, as before. - A check that could not run (no Chrome, a telemetry query that never returned, too few rate checks ingested) is
harness_fault, never a rollback for cause: a live check stops with the new version live and says so; a ramp step whose query never returned and has no probe sample restoresprevious(there is nothing left to judge on) but the verdict staysharness_fault. - The rate-limiter gate (below) runs in the pinned telemetry (fail → stop at 0%) and in the live telemetry during the 100% step (fail → rollback).
These are tested under the checked-in ramp in test/cd-ramp-only.test.ts.
One run at a time: the deployment lock
Two runs overlapped on 2026-11-07: run 2026-11-07T18-30-47-501Z replaced another agent's measurement canary b494b751 in the 0% slot. Two guards now stop that.
The lock. Every command that writes the deployment (cd, deploy-canary, ramp, rollback) takes ~/.local/state/ply-cd/deploy.lock (PLY_CD_LOCK overrides it) before it opens a run directory, and holds it until it exits. The lock lives in the shared state directory, beside the ledger, so runs from different worktrees see the same file. It is created with O_EXCL and records the holder's pid, host, cwd, command and start time. A second run is refused at once, not queued:
cd: state_conflict lock: deployment lock ~/.local/state/ply-cd/deploy.lock is held by a live run: pid 4242 on <host>, started 2026-11-08T16:10:02.120Z, cwd /Users/ivar/code/ply-web-ramp-only, command "cd --no-bump"Exit 1, nothing uploaded, no run directory. A lock whose pid is gone, or whose pid is now some other program (ps does not show scripts/cd/main.ts), is stale: the next run takes it over and prints lock: took over a stale lock from pid …. Takeover is serialised by a deploy.lock.steal directory, so two runs cannot both steal one stale lock. A lock written on another host cannot be checked and is never taken over; remove it by hand only after you have confirmed that run is gone. report, smoke, telemetry, debrief, annotate and sweep do not write the deployment and do not take the lock.
To roll back while a run is in progress, stop that run first (Ctrl-C or kill <pid> from the lock line); its lock is then stale and rollback takes it over.
The 0% slot. deploy-canary (and so cd) refuses with state_conflict to replace a version sitting at 0% that this run did not create:
cd: state_conflict deploy-canary: b494b751-… is already at 0% and this run did not create it; to replace it pass --replace-canary b494b751-…It checks before building (nothing is uploaded) and again on the split it reads after upload. Pass --replace-canary <that version id> (npm run cd -- --no-bump --replace-canary <id>, or deploy-canary --replace-canary <id>) when you know the 0% version is a leftover. The id must match exactly; naming a different one is still a conflict. A version tagged with this release (a resumed run) is this run's own and is not a replacement.
Changes the ramp cannot split (flag them to Ivar first)
Two kinds of change reach 100% without a ramp. Neither may be done silently: tell Ivar before you do it, say why the ramp cannot carry it, and use the procedure below.
Exports changes (Durable Object classes, named entrypoints). Every version in a multi-version deployment must declare identical exports. Adding a Durable Object or changing the exported classes makes the 0% split fail with API error 100405 ("All versions in a multi-version deployment must declare identical exports. Deploy the version that changes exports at 100% first, then split"); run 2026-11-06T21-23-36-859Z stopped there as harness_fault. Versions 59926117 ("rate limit exports at 100% (RATE_MODE=legacy; 100405 forbids an exports change in a split)") and fef77eda went straight to 100%. Safest procedure:
- Split the change in two commits. The first adds the export (the class, the binding, the migration) and nothing that uses it, and keeps the request path on the old code (for the rate limiter,
RATE_MODE=legacy). Its gate is green on its own. - With Ivar's go-ahead, and while you hold no one else's run (check the lock file), upload it with
cf workers versions create, note the version at 100%, and deploy the new one at 100% withcf workers deployments create. A version that is not in the deployment cannot be pinned withCloudflare-Workers-Version-Overrides, so there is no 0% smoke for it. Runnpm run cd -- smoke --expect-version <id>, the CDP visual smoke by hand, read the version's telemetry (cf o11y telemetry query, filtered by$workers.scriptVersion.id), and on any failurenpm run cd -- rollback --to <previous>(a single version at 100% is not a split, so the rollback is not refused). - Ship the commit that starts using the export through the normal ramp: both versions now declare the same exports, so the split works.
cf secret and var edits. cf workers secret put (and var edits through cf) create a new version and deploy it at 100% at once: d4e3eea6 (workers/triggered_by: secret, 2026-11-07 17:51Z) did. It bypasses the canary, the ramp and the live watch. Prefer a code change that ships through the pipeline (a binding default in cloudflare.config.ts). When a secret must change, flag it to Ivar first, note the version at 100% before the edit, make the edit, then run npm run cd -- smoke --expect-version <new id> and the visual smoke, and roll back to the noted version on any failure. Never edit a secret while a CD run holds the lock: the run's next split write will see a deployment it did not expect and stop with state_conflict.
The rate-limiter gate
The limiter fails soft (a broken read admits as fail_open, an unreachable counter drops flushes, a missing counter falls back to legacy), and all of those answer 200, so the smoke and error gates pass them. scripts/cd/rate-gate.ts runs inside the pinned telemetry stage (fail → stop at 0%) and inside the live telemetry during the 100% step (fail → rollback). It sends rate.probes (30) pinned probes of rate.probePath as a load client, then counts the version's own ply.rate.check / ply.rate.flush spans, and fails the version when any default-mode check ran a limiter other than rate.expectMode, when warm fail_open checks exceed rate.maxFailOpenRate (10%) over at least rate.minChecks (20), or when more than rate.maxDroppedFlushes (0) flushes were dropped without an injected fault. Fewer than rate.minChecks checks after the wait is a harness fault, not a pass. The thresholds and their evidence are rate in cd.policy.json; the details are in rate-limit.md.
RATE_MODE in cloudflare.config.ts and rate.expectMode in cd.policy.json must change together, in one commit (test/cd-rate-gate.test.ts fails when they differ). Changing only RATE_MODE makes the gate fail the new version as a silent degradation; changing only expectMode fails the version that is already right. Changing RATE_MODE with cf instead of a commit is a var edit (above): it reaches 100% without the gate.
The left column is the only path that raises traffic. An arrow to the right is that step stopping. A gray box leaves the split as it is. A red box puts the previous version back at 100% and the process exits. A pinned smoke failure takes the gray stop on that row: the canary stays at 0% and the ramp does not start. Source for the picture: cd-pipeline.mmd. The vector file is cd-pipeline.svg.
Before you start
On the box:
cf auth loginalready done. The login needs Workers Scripts Edit (versions and deployments; an account token calls this Workers Scripts Write), Workers Observability Read, and Artifacts Edit on the namespace below. It does not need routes, KV, R2, or tail.npm ciin this repo. Node must satisfypackage.jsonengines(^22.23.0 || ^24.22.0 || >=26.12.0)..nvmrcis24.22.1.npm run cdrunsscripts/cd.sh, which usesPLY_NODEif you set it, then/workspace/node-v24.22.1-linux-x64/bin/nodewhen that image directory is present, thennodeonPATH. If none of those satisfy engines, preflight stops and names that search. It does not judge the Worker. The wrapper prepends the chosen Node's bin directory toPATHbefore exec.cfis#!/usr/bin/env node, so a v20nodeleft earlier onPATHused to makecffail and the run recordharness_fault.git config user.nameanduser.emailset only when a bump actually commits. An unchanged pin, and--no-bump, do not need them. The bump does not push.- A clean work tree. Deploy refuses a dirty one.
HEADequal toorigin/mainaftergit fetch origin main. Deploy refuses any other commit (not-on-origin-main) and uploads nothing.- The Worker bearer in
~/.config/ply/worker-token(thePLY_TOKEN_FILEdefault since 2026-11-08; it was~/.ply-token, which no run used), mode0600(no group or other bits). The file is read, never printed. SetPLY_TOKEN_FILEonly to point somewhere else. - Namespace
ply-loadgenalready exists. The smoke does not create it.
mkdir -p "$HOME/.config/ply" && touch "$HOME/.config/ply/worker-token" && chmod 0600 "$HOME/.config/ply/worker-token"
# paste the bearer into that file. That path is the default; export PLY_TOKEN_FILE only to override it.Do not run npm run worker:deploy while this is in progress. It refuses unless PLY_FORCE_FULL_DEPLOY=1. With that set it sends the Worker to 100% in one shot and will replace the split you are ramping. GitHub does not bump the pin and does not deploy.
Optional overrides (anything you do not set keeps the default in the table further down):
| Variable | Default | What it changes |
|---|---|---|
PLY_WORKER | ply | Worker name |
PLY_BASE_URL | https://ply.ivar.workers.dev | Where smoke sends HTTP |
PLY_ARTIFACTS_NAMESPACE | ply-loadgen | Scratch repos for the empty-repo smoke |
PLY_BUMP_BRANCH | bump-ply | Local branch for a pin commit. Not pushed. |
PLY_TOKEN_FILE | ~/.config/ply/worker-token | Worker bearer, mode 0600. Read, never printed |
PLY_LOAD_TOKEN_FILE | ~/.config/ply/load-token | Load-test credential the rate gate's probes carry, mode 0600 |
PLY_CD_LOCK | ~/.local/state/ply-cd/deploy.lock | The deployment lock every writing command holds |
PLY_CD_RUN_DIR | ~/.local/state/ply-cd/runs | Where the run directory is created |
PLY_SHA | ply main | 40-hex pin, instead of resolving ply main |
PLY_CD_LEDGER | ~/.local/state/ply-cd/ledger.jsonl | Append-only event ledger across runs |
The rollout itself is scripts/cd/cd.policy.json: the steps (percent and dwell), the ingest wait and its poll period, the optional warm-up, the calibration block citing the runs behind each schedule number, the probe rate and mix, which signals are hard or informational, the burn multiple and sample floor, and the routed-share band. npm run cd -- --policy <path> uses another file for that run. The engine rejects a file that cannot be run (no steps, a single step, steps that do not increase, a last step that is not 100, a promote block, a calibration key that names no schedule field, latency set to gate without thresholds, a share band with z of 0). Every run appends a policy event with the file's sha256 and the body that ran. PLY_RAMP_PERCENTS, PLY_RAMP_DWELL_MS, PLY_RAMP_PROBE_RPS, PLY_RAMP_MAX_HOLDS, and PLY_RAMP_MIN_REQUESTS are rejected if set.
The whole run
From a clean checkout of the commit you want to ship:
npm run cdYou can pin ply to a sha you already know. The command is cd; --sha on its own is the same run:
npm run cd -- cd --sha 1274f2dd05b83d70f8d421cdcc79628692d848df
npm run cd -- --sha 1274f2dd05b83d70f8d421cdcc79628692d848dfRunning the same release again is safe. A release is the pair of the ply sha and the ply-web commit of this checkout. The command reads the current deployment and the version tagged with that pair, records the observation, and decides the next stage (rule reconcile):
| What is live | Next |
|---|---|
| No version tagged with that pair | Upload, then pin it at 0% |
| A version tagged only with the ply sha, or with a different ply-web sha | Upload, then pin it at 0% |
| That version at 0% | Pinned smoke, then telemetry |
| That version at a ramp percent | Ramp onward from that percent |
| That version at 100% | Live smoke, then telemetry |
| Live smoke already passed for that version | Done |
A checkout with tracked modifications stops before a run directory is opened (cd: harness_fault dirty-checkout: tracked files are modified). The commit sha would not describe the code about to be uploaded. A re-entry appends resumed on the unfinished run directory for that pair and keeps the same run id. A finished run, or an older run that recorded only the ply sha, starts a new one. Each smoke mints a fresh repo and a fresh token. Every traffic write still compares the deployment to the split it expects, and does not write when they differ. Deployment creates are single-flight for the process. cf workers deployments create is a harness_fault when the run log or the command sink is missing, so a create cannot land without a command event. The ramp arms one percent at a time, and only after that step's decision is on the log. A create during the dwell or the query, or a percent that is not armed, fails the run and does not call cf.
--no-bump skips the pin and deploys the checkout as it is. The run records that as a decision (no_bump, outcome advance). --sha may be passed with it only when that sha is already the pin; a different sha is an argument error.
npm run cd -- --no-bump
npm run cd -- cd --no-bump
npm run cd -- --sha <current pin> --no-bumpWhat prints, in order:
run: ~/.local/state/ply-cd/runs/2026-10-31T21-05-00-123Z
version: 8c1af0e2-7b3d-4c1a-9e20-6f0a1b2c3d4e
previous: 1aa0bb11-2222-4333-8444-555566667777
ramp step=1/5 percent=10 previous=90 samples=200/1800 error_rate=0.0000/0.0000 routed_share=10.00 target=10 tolerance=7.99..12.01 agg=2000 p99_wall_ms=88/74 p99_cpu_ms=12/11 latency=informational decision=advance
…
cd: PASS 8c1af0e2-7b3d-4c1a-9e20-6f0a1b2c3d4e ply 1274f2dd05b83d70f8d421cdcc79628692d848df web be08f0406dab6d5b2d1167e934f7716463a757edversion is the upload you just created. previous is the version that was alone at 100% and is the rollback target. run is the directory that has the CSV, the telemetry note, and the ramp lines. The last line is the verdict. ply and web are the release pair.
| Last line | Exit | What happened |
|---|---|---|
cd: PASS <version id> ply <ply sha> web <ply-web sha> | 0 | The new version is at 100%, and the live smoke and telemetry passed. |
cd: FAIL <stage>: <reason> | 1 | That stage stopped the run. If the line is from ramp or from the live checks, traffic is already back on previous unless the line says the rollback itself failed. |
cd: harness_fault <stage>: <reason> | 1 | A probe loss above probe.maxLossRate on at least probe.minSamples issued probes, a revoked Artifacts token that did not get 403, or a check that could not be evaluated (a telemetry query with no probe sample to stand on, a span-query shape mismatch, zero samples where samples are required, an implausible span join, a preflight miss). A ramp step above 0% then follows probe.onHarnessFault (checked-in telemetry: a probe-loss fault defers to the new version's telemetry). A harness stall (the same latency on both versions, or timed_out / shed with no version split) re-runs the dwell and does not use this row when server errors are zero. A ramp query that never returns after a clean probe window is not this row either: the step advances and names the loss. Any other stage leaves the split unchanged. This is not a change failure. |
cd: state_conflict <stage>: <reason> | 1 | The deployment was not the expected prior split, so the write was skipped. The split is unchanged. |
a usage line, no cd: PASS / cd: FAIL | 2 | The arguments were wrong. Traffic was not touched. |
A failing run still writes the directory. Relay the last line and the ramp lines from that directory.
Stages, alone, use the same last-line shape. npm run cd -- help lists them.
1. Bump
npm run cd -- bump
npm run cd -- bump --sha <40 hex>Fetches origin main and reads that commit's package.json. Resolves ply main (or the sha you passed). When the pin already matches, it leaves your checkout where it is. When it does not, it commits only package.json and the lockfile on bump-ply, whose parent is origin/main, and checks that branch out. The pull request is that one commit. It is not stacked on the branch you were on. It does not push. If you were already on bump-ply, that branch is reset to origin/main before the commit. user.name and user.email are read only when that commit is about to be made.
Pass, already pinned:
cd: PASS bump unchanged 1274f2dd05b83d70f8d421cdcc79628692d848dfPass, new local commit:
cd: PASS bump 1274f2dd05b83d70f8d421cdcc79628692d848dfFail (nothing committed; the two files are restored):
cd: FAIL bump: tracked files are modified; bump wants a clean tree
cd: FAIL bump: npm test failed: …You push bump-ply yourself if you want a pull request for the pin. Deploy runs only when HEAD is origin/main, so the pin commit is not uploaded until it has been merged there.
ply-web can test a new ply main before anyone bumps to it. ply's notify-ply-web.yml runs on workflow_run after ply's ci succeeds for a push to ply main and sends repository_dispatch ply-main-updated with client_payload.sha, but only when ivarvong/ply has the PLY_WEB_DISPATCH_TOKEN secret. That secret is not set today (ply's notify run for 1f14a9f, 2026-11-12, logged PLY_WEB_DISPATCH_TOKEN is not set), so no dispatch arrives and the check runs only by hand (workflow_dispatch with a sha input) until the secret exists. .github/workflows/ply-main.yml checks out ply-web main and runs sh scripts/ply-main-check.sh <sha> <dir>, which sets dependencies.ply to github:ivarvong/ply#<sha> in the runner only (never committed), installs, and runs npm test, npm run lint and npm run typecheck. If a gate fails, or npm install of the sha fails (ply's prepare: tsc, or an exports/engines change), the run fails and opens one issue in this repo titled ply main <short sha> breaks ply-web (or comments on the open one with that title), with the failed gates, the failing tests and the ply compare range from the current pin to the dispatched sha; a malformed sha, or a lockfile that resolved ply to a different sha than the one dispatched, fails the run (exit 2) without a report or an issue. The script writes the outcome to $GITHUB_OUTPUT itself; test/ply-main-check.test.js (in npm test) pins its exit codes, report and that wiring. Its concurrency group is keyed by the dispatched sha, so every ply main gets its own run (a single fixed group would let GitHub cancel a pending sha when a newer one queues). It uses only the workflow's GITHUB_TOKEN (contents: read, issues: write), deploys nothing and needs no secrets. Run the same check locally from a clean checkout with sh scripts/ply-main-check.sh <sha> /tmp/ply-main, then git checkout package.json package-lock.json.
2. Deploy the canary
npm run cd -- deploy-canaryFetches origin main first. Unless HEAD is that commit, it stops with not-on-origin-main (or origin-main-unavailable when the fetch fails), writes that typed error on the run, and uploads nothing. Otherwise it builds and uploads a version with cf workers versions create. The version id is the row in cf workers versions list whose tag is the release pair, <ply sha>+<ply-web sha>, both full object ids. The version message is cd canary plus that tag. A row tagged with only the ply sha, or with a different ply-web sha, is not this release, so the pair is uploaded. The version is reused when that pair tag is already there. The create command's own text is not parsed. A missing tag stops before traffic changes. It then routes that version to 0% beside the single version that is at 100%, and only when the deployment is still the split it just read. A mismatch stops with state_conflict and leaves the split alone. It will not upload into a 50/50 split, a deployment with more than two versions, or a version that is already the live one. If a different version is already sitting at 0%, the stage stops with state_conflict before it builds anything, unless --replace-canary <that id> names it (the 0% slot). The new version is never given traffic here.
Pass:
version: 8c1af0e2-7b3d-4c1a-9e20-6f0a1b2c3d4e
previous: 1aa0bb11-2222-4333-8444-555566667777
cd: PASS deploy-canary 8c1af0e2-7b3d-4c1a-9e20-6f0a1b2c3d4eThe command then waits until five GET / requests in a row, sent with Cloudflare-Workers-Version-Overrides: ply="<new>", come back with x-ply-version equal to the new id.
Fail, pin never stuck. The 0% version stays in the deployment. The old version is still at 100%. Traffic was not raised, and it was not rolled back, because there is nothing to undo:
cd: FAIL deploy-canary: pinned requests were not answered by 8c1af0e2-7b3d-4c1a-9e20-6f0a1b2c3d4e; left the 0% canary in placeFail, the deployment after create is not 0/100. The previous version is put back at 100% alone:
cd: FAIL deploy-canary: 8c1af0e2-… is at 10%, wanted 0%3. Smoke
Pinned (the canary). Every request sends the version override, and every response must carry that same id in x-ply-version:
npm run cd -- smoke --version 8c1af0e2-7b3d-4c1a-9e20-6f0a1b2c3d4eLive (after the ramp, or any time you want to see who is actually serving). No override. x-ply-version must still be that id, which is how you know traffic moved:
npm run cd -- smoke --expect-version 8c1af0e2-7b3d-4c1a-9e20-6f0a1b2c3d4eThe smoke creates an empty repo in ply-loadgen (prefix ply-web-cd-). Repo create returns a token. That auto-created token is revoked before the smoke mints its own write token, and the revoked bearer is checked with a GET of the repo's info/refs: Artifacts must answer 403. The smoke then mints a write token and replays the git lifecycle over /v1. It does not seed the repo with stock git. The minted token is revoked after those requests finish and checked the same way. Each check is a request event (method, url, status). The token is not in the event, the URL, or the run directory. On a pass the repo is deleted. The delete is idempotent: a timeout or other transient error is retried with backoff, and the bound is cleanup.attempts in the policy (the wait before retry n is cleanup.backoffMs * 2^(n-1)). A 404, [10200], or Repository not found means the repo is already gone. Each attempt's latency is a repo-delete observation. The measured delete on run 2026-11-01T17-29-43-122Z (web 9558b10) took 103s and returned APIError "timeout", then the retry was 404 [10200] Repository not found. If the delete still cannot be confirmed, the repo name is a cleanup_leak on verdict.json and in the ledger. That record is informational: a release that passed correctness still passes. npm run cd -- sweep deletes leaked ply-web-cd-* repos in the namespace older than cleanup.staleMinutes, and the next smoke sweeps the same set before it creates a repo. On a fail the repo name is in the verdict and the token is still revoked. A request that gets no response, a 403 naming that token as invalid or expired, or a revoke that does not stick as 403, stops the run as harness_fault. The split stays where it is. The operator decides what to do next. There is no retry and no rollback for that fault.
A 409 is a pass when that step expects it. The ones that matter after the current write API:
- Cherry-pick or revert of a patch that is already on the target, with the same adds and deletes, is
200empty. - Cherry-pick of that patch onto a target that also has its own extra file is
409merge_conflict.allowEmptydoes not turn that into a commit. Revert of the picked commit onto a branch that then added an unrelated file is200revert. - A merge may stamp author and committer as two different people.
fromandontomay be a 40-hex commit. Forty zeros is400and pushes nothing.
Pass:
cd: PASS smoke 8c1af0e2-7b3d-4c1a-9e20-6f0a1b2c3d4eFail. The step name is first. The repo is left so you can look, without a token:
cd: FAIL smoke: cherry-wider: status 200, wanted 409; repo ply-web-cd-20261031210500-ab12cdsmoke.csv in the run directory is one row per request. pass is the step verdict, not the HTTP status. A row with status 409 and pass is an expected conflict. Columns: step, method, path, status, ms, pass, cf_ray, x_ply_request_id, x_artifacts_ray, x_ply_version, x_ply_build, server_timing, detail. The token is not in the file.
On a full npm run cd, a smoke failure leaves the canary at 0%. The ramp does not start. Nothing is rolled back: the previous version is already at 100%. The smoke CSV is still written. Telemetry for that canary does not run.
4. Telemetry
npm run cd -- telemetry --version 8c1af0e2-7b3d-4c1a-9e20-6f0a1b2c3d4e \
--since 2026-10-31T21:05:00Z \
--report ~/.local/state/ply-cd/runs/2026-10-31T21-05-00-123Z/smoke.json--since is the start of the smoke window (ISO-8601, epoch milliseconds, or epoch seconds). The report is the smoke JSON from the same run. The check queries cf o11y telemetry query for that version in that window.
It fails when any of these are true:
- an exception, a non-ok invocation, a 5xx, or an error log on that version
- a smoke id logged by a different version
- smoke ids missing from telemetry (no
ply.requestlog, or no trace) abovetelemetry.maxLossRateinscripts/cd/cd.policy.json(default 0.5). Workers Observability loses whole invocations in bursts, so loss at or below that rate is informational: the telemetry stage records the count, rate, and every missing id intelemetry.md,telemetry.json(loss), atelemetry-lossobservation inevents.jsonl, and the debrief's Telemetry loss section. An id the smoke did not see answered by this version is never waived. Status mismatches, 5xx, exceptions, error logs, and burn stay hard gates regardless of loss - a response status the smoke did not expect
- an exception, a 5xx log, a non-ok outcome, or an error log
- wall, CPU, or upstream
fetchspan p50/p99 over the ceilings below, on samples the query returned
A 409 the smoke expected is not an unexpected status. A 500 the smoke did not expect is.
On the pinned window (not the live one after 100%) it also checks ply's own spans (issue #67). The smoke's edit-traced step is a single-file find/replace on the squash branch (POST /v1/edit with path, find, replace), which ply runs as editFile. Every ply span of a /v1 request is a Workers span under the request span (src/edge-trace.ts), so that request's trace must hold git.edit, git.edit.tree, git.edit.blob, git.upload-pack fetch, and git.receive-pack besides the platform's own POST and fetch spans. The check reads the same span-events query ($metadata.type = span), joins on the request's cf-ray from the smoke report and then on the traceId those events carry, and re-queries on the telemetry polls (5s, 10s, 15s, up to 60s) while names are missing, because span events of one trace are ingested independently. Source: scripts/cd/span-check.ts.
- Every name is there: a second line,
spans: edit-traced <request id> trace <traceId> has git.edit, …. - The trace is there and a ply span is not:
cd: FAIL telemetry: spans: edit-traced <id> trace <traceId> is missing ply spans <names>; saw <names> (#67: ply spans are not reaching Workers tracing). That is a worker fault. The canary stays at 0% and the ramp does not start. - No span event joins to the request within the wait, or the report has no
edit-tracedrow (an oldersmoke.json):harness_fault, withspans: no span event joined to edit-traced …as the reason. The split is unchanged.
The result is spans in telemetry.json and the last line of telemetry.md, and a ply_spans rule on the telemetry decision in events.jsonl. It decides only when every other telemetry check passed.
Pass prints the verdict line, then:
telemetry: 40/42 matched, 2 lost (tolerated)
spans: edit-traced 3f9c2a1e-… trace 6ae804bdd2731f488b904cf72e929540 has git.edit, git.edit.tree, git.edit.blob, git.upload-pack fetch, git.receive-pack (14 span events)
cd: PASS telemetry 8c1af0e2-7b3d-4c1a-9e20-6f0a1b2c3d4eFail:
cd: FAIL telemetry: telemetry: 0/42 matched FAIL: 42 lost without x-ply-version … recorded by the smoketelemetry.md in the run directory is that verdict plus the percentiles. telemetry.json is the machine-readable pass or fail. The ramp will not move traffic unless that file is a pass for this version.
Ceilings for this smoke (a tiny repo, cold isolates included; tighten them after a few real runs):
| p50 | p99 | |
|---|---|---|
| Worker wall | 4000 ms | 25000 ms |
| Worker CPU | 250 ms | 3000 ms |
Artifacts fetch span | 2000 ms | 15000 ms |
Span events have $metadata.requestId null. They carry traceId and rayId. The smoke CSV records each request's cf-ray, and the join uses that ray, then traceId when the ray does not match. A request id still on a span is only a last resort.
The traces view returns one summary row per trace (traceId, a span count, a duration, a root span name) and no child spans. Outbound fetch durations come from a second query, view: events with $metadata.type = span. On those events the duration is source.durationMS and an Artifacts call has source.name fetch with a path ending in git-upload-pack, git-receive-pack, or info/refs.
A check the harness cannot evaluate is harness_fault: the query threw, the span join matches implausibly few of the invocations the worker logged, or the span query came back as trace summaries. The split stays where it is. There is no rollback and no retry. That result is not a worker fault. A 5xx, an exception, an error log, or a non-ok outcome is a worker fault, because the value was observed. A wall, CPU, or fetch percentile over the ceiling above is a worker fault only when the policy's latency mode is gate. The checked-in policy is informational, so those percentiles are printed and do not fail the stage. A missing latency series is a harness fault only in gate mode.
5. Ramp
Only after the pinned smoke and the telemetry file are both a pass for this version, and the deployment is still new-at-0% / previous-at-100%. An empty split, a percent that is not an integer 0–100, or two copies of one version id is refused before cf is called:
npm run cd -- ramp --version 8c1af0e2-7b3d-4c1a-9e20-6f0a1b2c3d4e \
--previous 1aa0bb11-2222-4333-8444-555566667777 \
--verdict ~/.local/state/ply-cd/runs/2026-10-31T21-05-00-123Z/telemetry.jsonWithout --verdict, or with a verdict that is not a pass for that version, the command exits and does not change traffic.
The checked-in steps are 10, 25, 50, 75, 100 with dwells 30000, 25000, 25000, 25000, 40000 ms (40000 each until 2026-11-08). The list is only in the policy file, and calibration["steps[*].dwellMs"] there is the derivation: at 50 probes/s the 10% step issues 1500, which at the worst measured 10% share (0.0874, n=10 steps in the ledger) is 131 new-version requests against split.minNewRequests 100; the 25/50/75% steps issue 1250, 25% above probe.minSamples 1000; the 100% step keeps 40 s because it runs beside the live checks (median 49 s over the 10 promote runs) and a shorter dwell would not end the run sooner. After the dwell the step re-queries its window every ingestPollMs (3000) until the Worker has logged 98% of the probes that were answered, with ingestWaitMs (20000) as the ceiling: across the 10 polled promote watches, 98% had landed 8.6–19.2 s after the dwell end (median 12.6 s). On the ramp-only runs of 2026-11-08 up to 17-17-40-415Z the 10–75% steps stopped rising at about 93% of issued probes, so the poll ran to the ceiling. That was queryCapturedSliced deleting rows that shared a non-unique $metadata.id (fixed in ace0eb9), not ingest. A 30000 ceiling (0016dae) only added flat polls. On 2026-11-08T18-30-12-168Z (d3a35bb1), after the fix, every responded probe of the 10–75% steps has exactly one external fetch row (1499/1250/1250/1250, 0 absent). The landed check compares all external fetches with probes, so other traffic counts toward it: that run also carried about 8 req/s of GET /r/fresh-*, and its polls stopped at agg 1557/1245/1277/1447 against an estimated 1740/1450/1450/1450 external fetches in the windows (probes plus that traffic). Smoke id-matching still uses TELEMETRY_WAIT_MS (60000); that budget is not the ramp wait. warmupMs is 10000: after the split is applied the step pauses 10 s before the probes, and that pause is not part of the judged window. It was 0 until 2026-11-06, when two ramps of 86456068 (runs 20-01-37Z, 20-11-49Z) rolled back at 100% on routed share 98.84% and 98.74%: 24 and 25 old-version requests, every one within 4.6 s and 3.7 s of the 100% deployment, none after, 0 errors (the split takes a few seconds to reach every edge).
The only pinned traffic is the 0% gate: the lifecycle smoke and the deploy check, both sent with Cloudflare-Workers-Version-Overrides. From the first ramp step on, probes are ordinary requests. They do not set that header, and they do not pin the old version to balance the arms. Right now those requests are the only load. Real users would be judged by the same telemetry.
During the probe the ramp calls at probe.rps (25) ticks per second, once per entry in probe.calls. Each call's path is the request path and query, and the engine sends that string as written. The checked-in mix is GET / (the usage text, no bearer) and GET /v1/discover?remote=https%3A%2F%2F5ad2236b39de4c8bd8f6270ea63f3ea9.artifacts.cloudflare.net%2Fgit%2Fpublic-v0%2Fply.git (the public ply mirror, with the bearer from PLY_TOKEN_FILE). That is 50 requests per second. The scheduler is open-loop: it issues on the period whether or not earlier probes have returned, and the dwell ends on the wall clock (dwellMs), not when the in-flight set settles. Each probe has its own deadline, probe.timeoutMs (2000). Hitting it is timed_out, which is not a transport error. Outstanding probes are capped at probe.maxInFlight (50, one second of the old tick cap times the two calls). A tick that would pass that cap is shed and counted, and the clock keeps going. There is no /health route. The response's x-ply-version and latency are on the request event. A lost probe has no version. A probe that gets no response (no status and no ray: a client abort, a network failure, or timed_out) is a probe loss, not a server error. A shed was never sent, so it is not in the loss denominator. The step records probe_loss (count, rate, and ids), timed_out, and shed on the ramp line, as a probe-loss observation in events.jsonl, and the loss on verdict.json. Loss at or below probe.maxLossRate stays in the sample only as loss: correctness and availability use the probes that responded, and that count still has to reach burn.minSamples. Fewer responded probes than that is insufficient_samples, not a pass. The loss-rate gate cannot fail a step until the issued count reaches probe.minSamples (1000). Above that, a loss rate above probe.maxLossRate is harness_fault. A lost probe has no version, so a window whose responded probes show the same stall on both versions — or whose losses are timed_out or shed while both versions stayed healthy — is harness/network loss. That window is re-run, up to probe.windowAttempts (2), instead of rolling back a version with zero server errors. When those attempts are exhausted the step stops as insufficient_samples. A window cut short because this client stalled (harness_unhealthy: the box itself froze, not the Worker) is likewise unjudged. A 5xx on a responded probe inside that window, or inside a window too small for the loss-rate gate, is still a rollback. Above 0%, the step's split write has already moved traffic, so every unjudged step (insufficient_samples, harness_unhealthy, and a telemetry-unavailable harness_fault) settles it through probe.onHarnessFault (ply-web#145): rollback and telemetry restore previous at 100%, and hold writes the split back to the last fully passed step, or restores previous when no step passed. At 0% there is nothing to settle and the reason ends held at 0%. The ramp state's reason ends with where traffic is (rolled back to previous, held at N%); if that rollback or hold write fails it ends …failed: <error>; still at N%, rolledBack is false, and the new version is still serving N%: treat it like a failed rollback. For a real probe-loss harness_fault, probe.onHarnessFault says: telemetry (the checked-in choice since 2026-11-08) records a probe-loss fault and judges the step on the new version's telemetry instead, and restores previous only when that telemetry is also missing; rollback restores previous at 100% (the checked-in choice until 2026-11-08); hold returns to the last fully passed split. The verdict kind stays harness_fault, distinct from a correctness fail. The checked-in rate is 1/790, the single client abort on the 100% step of run 2026-11-01T17-06-08-780Z (web 789ba93, ply 1743461), with no headroom above that measurement. The scheduler bounds are from the 50% step of run 2026-11-01T19-40-35-485Z (web 22bfb79): healthy p50 41–101ms on both 4f3d7acf and 19ee3cab, then a client stall at about 69830ms, 10 of 128 never answered, and a decision about 3m14s after a 40s dwell. timeoutMs sits above 10× that 101ms p50 and below the stall. minSamples sits above 128 and below the healthy steps' about 1990 probes. Round bounds that are not themselves a percentile are marked PRIOR on probe.schedule. Before each percentage change the ramp reads the deployment and writes only when it is still the previous step's split. Otherwise it stops with state_conflict and does not roll back.
Five dwells are 145 s, five warm-ups 50 s; the polled ingest is 10–24 s a step. See Duration for measured runs. The original 20s wait was the tail measured on run 2026-11-01T00-10-02-620Z: the query ran about 1.4s after the dwell and the newest event was still 12640ms behind the dwell end at 1% and 17313ms behind it at 5%. 20000 is above that. Those two steps are no longer in the file; the lag is.
Then it reads one cf o11y events query for the whole worker over that step's window, and partitions it by $workers.scriptVersion.id. The new version, the old version, and the aggregate (every invocation in the window, which is what users feel) are judged separately. Child spans are a second events query with $metadata.type = span, not the traces view. Span events have no $workers.scriptVersion.id and a null $metadata.requestId, so they are queried by service and joined on rayId, then traceId.
The step decision is the policy. A signal set to hard rolls back. The same signal set to informational is named on the advance reason and does not roll back. Burn rate is (bad / total) / (1 − target). Definitions are in Metrics. The checked-in file:
| Signal | Checked-in mode | What happens |
|---|---|---|
| Smoke | hard | The 0% lifecycle only. While the ramp query returns, a probe's HTTP status is not a gate, except a 5xx in a window the loss-rate gate will not judge (a harness stall, or fewer issued probes than probe.minSamples): that 5xx rolls back without waiting for the query. When the query is lost, a 5xx on a responded probe is the correctness signal |
| Exception, 5xx, error log, non-ok outcome | hard | Any, on the new version, on the old version, or across the worker. One request is enough |
| SLO burn | hard | The same three scopes. bad/total divided by the error budget is above burn.fastBurn (14.4), and that SLO saw at least burn.minSamples (100) requests |
| Routed share | rollback | routedNewShare outside the binomial band. split.onMismatch may instead be hold, which leaves the split |
| Thin evidence | hold | Fewer than split.minNewRequests (100) new-version invocations, or no invocations at all. Outcome insufficient_data. onInsufficient: advance moves on and still says so |
| Latency | informational | Never, under this file. Wall p99 and CPU p99 are on the line and on the ramp observation. off omits them from the line. gate needs minSamples, p99Ratio (> 1), and effectFloorMs, and rolls back only when both sides have that many wall samples and p99 clears both thresholds |
| Query failed | retry, then probes | Each cf o11y telemetry query is retried up to telemetry.query.attempts (3). The wait before retry n is telemetry.query.backoffMs * 2^(n-1). A later success is a normal sample. If every attempt fails and the probe window already passed, the step uses those status codes: a 5xx rolls back, and a clean window advances. The lost series is informational on the reason and a telemetry-query observation per attempt. It is not stored as zero errors, and it is not a harness_fault. With no probe sample the step is still harness_fault, and above 0% the split follows probe.onHarnessFault |
How many requests that is
Correctness here is zero tolerance: one exception, 5xx, error log, or non-ok outcome fails the scope that saw it. With zero observed errors in n requests, the error rate p at which a step has a 95% chance of seeing at least one is
(1 − p) ^ n = 0.05
p = 1 − 0.05 ^ (1 / n)That p is the minimum detectable error-rate change from a clean baseline. The rule of three, 3 / n, is the same bound at these sizes.
The checked-in probe schedules 25 × 2 × 30s = 1500 requests at the 10% step, 25 × 2 × 25s = 1250 at 25/50/75%, and 25 × 2 × 40s = 2000 at 100%. The smallest open step is 10%, so the new version's expected count is 1500 × 0.10 = 150. At 100% the old version is not allocated.
| Scope | Expected n | Detectable error rate |
|---|---|---|
| New version at 10% | 150 | 1.98% |
Floor, minNewRequests and burn.minSamples | 100 | 2.95% |
| New version at 25% | 312 | 0.96% |
| New version at 50% | 625 | 0.48% |
| Aggregate, and the new version at 100% | 2000 | 0.15% |
A step with fewer than 100 new-version invocations does not advance. That floor is the 2.95% row. A normal 10% step sees about 150 and can catch a 2% regression on the new version (it was 200 and 1.5% at the old 40 s dwell). The aggregate, which is what users feel, can catch 0.15% in the same window. Per-version is what keeps a regression on the small arm from being diluted into the aggregate.
The share check is the same n. For target proportion p and z = 3, the band is p ± z √(p(1−p)/n). At 10% and n = 1500 that is 7.68%..12.32%. At 100%, p(1−p) is 0, so the share has to be 100%. Counts come from external fetch invocations, not from the probe's own bookkeeping and not from named entrypoints. test/fixtures/cd/routed-share.json is a synthetic 10% window of this shape: 200 of 2000 advances, 1000 of 2000 rolls back, 40 of 400 is insufficient_data.
This probe is a scaled model of production. At about 800 requests per second, a 40s dwell already holds 32000 requests and the 10% arm holds 3200, which can see about a 0.09% error rate. Retune probe.rps, burn.minSamples, and split.minNewRequests in the policy. When real traffic is the load, probe.rps can go to 0 and the same gates read whoever called the worker. That is a config change.
samples= on a ramp line counts external fetch invocations, new / old, in this window. An external fetch is $metadata.type cf-worker-event, $workers.eventType fetch, and $workers.entrypoint absent or default. A named entrypoint (FlagsCache, GitCache) is an internal ctx.exports call. It has its own ray and request id. Rows are deduplicated on $metadata.id + requestId + traceId + spanId (rowKey in scripts/cd/telemetry.ts) because $metadata.id alone is a millisecond plus a per-invocation row position: distinct invocations in the same millisecond share it, and keying on it alone deleted 4–10% of rows at every partial ramp step on 2026-11-08 (runs 16-58-16-960Z, 17-07-46-259Z, 17-17-40-415Z). The event page cursor (pageCursor) likewise steps back one id, because offset returns ids strictly below it and a same-id group split by a page boundary would otherwise lose its remainder. Those counts are internal_next / internal_previous on the ramp line and internalInvocations on the ramp observation. They are not part of samples, agg, error rates, latency samples, or the share. agg= is the external count, the share denominator. Each external request also writes a ply.request log, so the event page is larger than the sample count. The query asks for limit 1000 and follows the cursor. On the 2026-11-01 run a page came back short because the index held about 28s of the 40s dwell, not because of a 1000-row cap. The window's to is the dwell end plus 60000 ms so the tail is inside the query. The ingest wait is what keeps the next step's probes out of this step's judged window.
At 100% the old version is dropped, so its wall and CPU numbers are the last step whose old version had at least 5 wall samples. That line includes previous_sample=carried. If no earlier step had them, 100% still advances when the new version is clean.
Comparing versions on trace latency, and leaving cold starts out of that compare, waits until the rollout itself is trusted. The events this ramp reads have no cold-start or isolate-start field. $workers carries wallTimeMs and cpuTimeMs, plus outcome, execution model, and version. This policy does not try to drop cold starts.
Each step prints one line to stdout and appends the same line to ramp.jsonl in the run directory. Relay those lines as they appear.
A clean ramp:
ramp step=1/5 percent=10 previous=90 samples=200/1800 error_rate=0.0000/0.0000 routed_share=10.00 target=10 tolerance=7.99..12.01 agg=2000 p99_wall_ms=88/74 p99_cpu_ms=12/11 latency=informational decision=advance
ramp step=2/5 percent=25 previous=75 samples=500/1500 error_rate=0.0000/0.0000 routed_share=25.00 target=25 tolerance=22.10..27.90 agg=2000 p99_wall_ms=90/76 p99_cpu_ms=12/11 latency=informational decision=advance
ramp step=3/5 percent=50 previous=50 samples=1000/1000 error_rate=0.0000/0.0000 routed_share=50.00 target=50 tolerance=46.65..53.35 agg=2000 p99_wall_ms=94/78 p99_cpu_ms=13/12 latency=informational decision=advance
ramp step=4/5 percent=75 previous=25 samples=1500/500 error_rate=0.0000/0.0000 routed_share=75.00 target=75 tolerance=72.10..77.90 agg=2000 p99_wall_ms=101/80 p99_cpu_ms=14/12 latency=informational decision=advance
ramp step=5/5 percent=100 previous=0 samples=2000/0 error_rate=0.0000/- routed_share=100.00 target=100 tolerance=100.00..100.00 agg=2000 p99_wall_ms=114/84 p99_cpu_ms=15/13 latency=informational decision=advance previous_sample=carried
cd: PASS ramp 8c1af0e2-7b3d-4c1a-9e20-6f0a1b2c3d4eHow to read a line:
| Field | Meaning |
|---|---|
step=2/5 | Second percentage. 1-based. The count is the policy's step list. |
percent=25 previous=75 | Configured new-version share, and what is left on the old one. At the last step previous=0. |
samples=500/1500 | External fetch invocations in this step's window, new / old. Named entrypoints are not included. |
error_rate=0.0000/0.0000 | Non-ok rate, new / old, four decimal places. |
routed_share=25.00 | 100 × new / agg. Null (-) when agg is 0. |
target=25 | The step's percent. The same number as percent. |
tolerance=22.10..27.90 | The binomial band in percent, z standard errors at this agg. - when agg is 0. |
agg=2000 | External fetch invocations in the window, every version. |
p99_wall_ms=90/76 | Wall-time p99 in milliseconds, new / old. - means that side had no sample. Omitted when latency mode is off. |
p99_cpu_ms=12/11 | CPU p99, new / old. Informational under the checked-in policy, same as wall. |
latency=informational | The percentiles are recorded. They do not roll back. gate would. |
decision=advance | No hard signal fired, the share is inside the band, and the new version had enough invocations. |
decision=rollback | A hard signal fired, or the share is outside the band and onMismatch is rollback. Traffic goes back to the previous version. |
decision=insufficient_data | The new version had fewer than minNewRequests invocations (or the window had none) and onInsufficient is hold. The split stays. |
decision=hold | The share is outside the band and onMismatch is hold. The split stays. The checked-in file rolls back instead. |
reason="…" | Present on rollback, hold, insufficient evidence, harness_fault, and an advance that names an informational signal. Relay it with the line. |
internal_next=FlagsCache:92 | Named-entrypoint fetches on that version, name:count separated by commas. Omitted when there were none. Not a gate. |
previous_sample=carried | The old-version p99 is from the last step that had at least 5 wall samples. The sample counts on that line are still this window. |
A large wall shift advances. The 2026-11-01 5% walls (test/fixtures/cd/ramp-5pct-walls.json, run 2026-11-01T00-10-02-620Z) do not roll back on latency. So does a step whose p99 is 27ms against 12ms. insufficient_data now means the new version was not seen enough times. It is not the old latency compare. Relay it: nothing was rolled back, and the split did not advance.
An exception on the new version, even with one request:
ramp step=1/5 percent=10 previous=90 samples=1/1999 error_rate=1.0000/0.0000 routed_share=0.05 target=10 tolerance=7.99..12.01 agg=2000 p99_wall_ms=40/70 p99_cpu_ms=8/9 latency=informational decision=rollback reason="1 exception on the new version"
cd: FAIL ramp: 1 exception on the new versionA 10% step whose telemetry shows about half the traffic on the new version:
ramp step=1/5 percent=10 previous=90 samples=1000/1000 error_rate=0.0000/0.0000 routed_share=50.00 target=10 tolerance=7.99..12.01 agg=2000 p99_wall_ms=80/80 p99_cpu_ms=12/12 latency=informational decision=rollback reason="routed share 50.00% is outside 7.99..12.01 of target 10% (n=2000)"A ramp query that times out is retried. Run 2026-11-01T19-17-18-170Z (web 07568e0) lost one cf o11y telemetry query --body at 100% to APIError "timeout" Envelope Error after every probe had returned, and that single miss rolled the release back. A retry that returns a page is not a fault. If the bound is exhausted and the probes were a full clean sample, the step advances and names the loss:
ramp step=5/5 percent=100 previous=0 samples=-/- error_rate=-/- routed_share=- target=100 tolerance=- agg=0 p99_wall_ms=-/- p99_cpu_ms=-/- latency=informational probe_loss=0/790 decision=advance reason="informational: telemetry query lost; 790 responded probes, 0 server errors; …"A 5xx on one of those probes still rolls back, because that status was observed. Routed share, burn, exceptions, and error logs from the missing query are not filled in as zero. With no probe sample at all, the step is harness_fault and above 0% follows probe.onHarnessFault (checked-in telemetry: with no telemetry either, roll back to previous).
If the rollback request itself does not stick, the last line says so and names the version that should be at 100%. Do not treat that run as rolled back. Put it back yourself with the command in the next section.
6. After 100%
npm run cd does not stop at 100%. It smokes again with no version override (--expect-version) once unpinned traffic has converged on the new version, then runs telemetry (with the rate gate) on that live window, and runs the CDP visual smoke (scripts/visual-smoke.mjs, 375 + 1280 px, light + dark, every page pinned to the new version; screenshots in <run>/visual/, verdict in <run>/visual.json). All have to pass. They start the moment the 100% split is written and run during the 100% step, which is the live watch; the run waits for both. A Worker failure in any of them rolls back; a harness fault alone (no Chrome, a lost query) stops with the new version live. A visual smoke with no Chrome (CHROME unset and no macOS Chrome) is recorded as skipped. Events written while the checks run in parallel may carry the label of the stage that last started; the stage timings in the manifest are exact.
Pass is the same cd: PASS <version id> ply <ply sha> web <ply-web sha> as the whole run.
Fail rolls back to previous at 100% and exits:
cd: FAIL smoke: read-readme: x-ply-version 1aa0bb11-…, wanted 8c1af0e2-…
cd: FAIL telemetry: telemetry: … 3 5xxOrganic 5xx on the new version during the live window fails this telemetry check and rolls back. The canary check was only the pinned smoke ids. The live check is every invocation of that version in the window.
Rollback
Any time, including when a ramp or a live check did not put the old version back:
npm run cd -- rollback --to 1aa0bb11-2222-4333-8444-555566667777--to is the good version. It becomes the only version, at 100%. The command reads the deployment back and will not print a pass unless that is what it sees.
cd: PASS rollback 1aa0bb11-2222-4333-8444-555566667777
cd: FAIL rollback: rollback to 1aa0bb11-… failed: …The run directory
~/.local/state/ply-cd/runs/<timestamp>/ (PLY_CD_RUN_DIR; an older checkout used cd-runs/, which is still gitignored):
| File | What you open it for |
|---|---|
verdict.json | { "kind": "pass", "version": "…", "failures": [], "probe_loss": [], "cleanup_leak": [] }, kind: "fail" with stage, reason, and rolledBack, kind: "insufficient_samples" when responded probes missed burn.minSamples, or kind: "harness_fault" / kind: "state_conflict". A ramp harness_fault names whether traffic was rolled back. probe_loss is one record per ramp step (count, rate, ids). cleanup_leak is scratch repo names whose delete could not be confirmed. It does not fail a pass |
smoke.csv / live-smoke.csv | One row per HTTP call. pass is the step. detail is why a row failed. |
smoke.json / live-smoke.json | The ids telemetry asks for, plus the scratch repo name if it was kept |
telemetry.md / live-telemetry.md | The verdict line and the percentiles |
telemetry.json | The pass/fail the ramp reads. A ramp with a fail here does not change traffic. |
ramp.jsonl | One JSON object per step: { "line": "ramp step=…", "probe_loss": { "percent", "count", "rate", "total", "responded", "ids" } } |
events.jsonl | Append-only log. One typed event per line. This is the record a debrief reads |
manifest.json | sha256 of every other file, written after run_finished, and blobs: dropped (a clean pass) or kept. The pipeline does not modify the directory after this; only retention (below) does |
blobs/<sha256>.gz | Kept only when the run failed or rolled back. Every command's stdout and stderr (redacted) and every observation's body, gzipped and stored once by content: a command event's stdoutDigest is the file name. A ramp observation's body lists the digests of the telemetry pages it read instead of copying them |
visual/ | CDP screenshots |
pruned.json | Present only on a compacted run: its status, what retention removed, bytes freed |
The gates read only an observation's measurement (counts, errors, latencies, shares), and that is in events.jsonl, so a clean pass loses nothing it decided on by dropping blobs/. Before 2026-11-07 every telemetry poll wrote its full stdout to captures/<seq>.stdout (up to 2.7 MB) and again, escaped inside a JSON string, to raw/<seq>.json (up to 29 MB): a ramp run was 120-135 MB, and about 50 runs filled the disk twice.
Retention
Every run that creates a directory prunes PLY_CD_RUN_DIR first (scripts/cd/retention.ts, RETENTION) and prints retention: N whole, N compacted, N removed, X MB freed. A run's status is the last run_finished in its events.jsonl; a rollback is a failure, and an unfinished run older than 24 h counts as a failure (younger ones, and the run being resumed, are not touched).
| Tier | Which runs | What stays |
|---|---|---|
| Whole | The newest 10 passes and the newest 10 failures | Everything. A pass also loses any leftover blobs/, captures/, raw/ |
| Compacted | The next 500 runs, newest first | Top-level summary files only (verdict.json, telemetry.*, smoke.*, live-smoke.*, visual.json, deploy.json, ramp.*): tens of KB. events.jsonl (its lines are in the ledger), manifest.json, and every subdirectory go; pruned.json records it |
| Removed | Older | The directory and its .debrief/. The ledger still has every event |
ledger.jsonl is never pruned. liveSmokeAlreadyPassed reads live-smoke.json, which compaction keeps; --sha resumes only an unfinished run, which is whole for 24 h. A debrief of a pass or compacted run lists the captures it cites as not kept.
When it fails
| You see | Traffic now | What you do |
|---|---|---|
cd: FAIL bump: | Unchanged | Fix the tree or the gate. Nothing was uploaded. A failed pin does not move bump-ply unless you were already on it. |
cd: FAIL deploy-canary: not-on-origin-main | Unchanged | Check out origin/main. A pin commit on bump-ply is not deployed from there. |
cd: FAIL deploy-canary: and the line says the canary was left at 0% | Old at 100%, new at 0% | Read the pin failure. Rollback only if you want the 0% slot cleared: npm run cd -- rollback --to <previous>. |
cd: FAIL deploy-canary: after a bad split | Previous at 100% if the rollback landed | Confirm with cf workers deployments list --worker ply. |
cd: FAIL smoke: or cd: FAIL telemetry: before any ramp line | Still 0% / 100% | The ramp did not start. The scratch repo name is in the smoke failure. |
A ramp line with decision=advance, including one that prints latency=informational | The next percentage | Relay the line. Wall and CPU time did not roll anything back. |
A ramp line with decision=insufficient_data or decision=hold | The percentage where it stopped | The split was not advanced and was not rolled back. Thin evidence, or a share miss whose policy says hold. Read the reason. |
A ramp line with decision=insufficient_samples or decision=harness_unhealthy | previous at 100% (telemetry and rollback), or the last fully passed split under hold (previous when none passed); at 0%, unchanged | The step was not judged (a probe window under burn.minSamples, a harness stall whose re-runs are exhausted, or a client stall), so its traffic was settled: the reason ends rolled back to previous or held at N%. If it ends …failed: <error>; still at N%, the new version is still serving N%: not safe, run npm run cd -- rollback --to <previous> and read the deployment list. |
A ramp line with decision=rollback, then cd: FAIL ramp: | Previous at 100% | Relay the line. The new version is off traffic. |
cd: FAIL ramp: and the text says rollback failed | Not safe | Run npm run cd -- rollback --to <previous> and read the deployment list. The id to restore is the one in that sentence. |
cd: FAIL smoke: or cd: FAIL telemetry: after ramp lines that reached percent=100 | Previous at 100% if rollback landed | Same as a ramp rollback. The live check is what rejected it. |
cd: state_conflict lock: | Unchanged | Another run holds the deployment lock. Wait for it; the line names its pid and worktree. |
cd: state_conflict deploy-canary: … already at 0% | Unchanged, nothing uploaded | Someone else's version is in the 0% slot. Ask whose it is; pass --replace-canary <id> only when it is a leftover. |
cd: harness_fault | On a ramp step above 0% whose telemetry query never returned with no probe sample, previous at 100% (telemetry and rollback), or the last fully passed split under hold. A probe-loss fault alone does not move traffic under the checked-in telemetry. Elsewhere, the split where the run stopped | The version was not blamed. A probe loss above probe.maxLossRate once probe.minSamples issued probes were counted, a 403 on the token this stage minted, or a telemetry query that had no probe sample, or a span join. A stalled dwell with zero server errors is re-run, and if it is still stalled the step stops as insufficient_samples (see that row) instead of being judged a rollback. A lost ramp query after a clean probe window stays on the advance line. The verdict kind is harness_fault, not a correctness fail. |
cd: state_conflict | The split that was already there | Something else changed the deployment. Do not apply the step again until you have read cf workers deployments list. |
Metrics
SLOs are data in scripts/cd/slo.ts. A ramp step does not inline the numbers.
| SLO | Good event | Target | Window |
|---|---|---|---|
| Availability | Not a 5xx and no exception, over /v1 requests, excluding expected 4xx | 99.9% | 28 days |
| Read latency | GET /v1/* wall time under 1000 ms | 99% | 28 days |
| Write latency | POST /v1/* wall time under 2500 ms | 99% | 28 days |
| Correctness | Lifecycle smoke: every step the expected status, git fsck clean, refs equal | 99.5% | 28 days |
The latency ceilings are initial. Tighten them from a real baseline; the 99% target stays.
Error budget and burn rate:
budget = 1 − target
burn = (bad / total) / budgetA burn of 1 spends the budget at the planned rate. The ramp fails a step when burn is above 14.4 and that SLO has at least burn.minSamples requests in the step window (100 in the checked-in file). 14.4 is the SRE workbook fast-burn multiple: on a 30-day budget, 14.4× for one hour spends about 2%. Our SLO window is 28 days. The step still uses 14.4, not a recomputed multiple.
Availability on a ramp step treats every 4xx as expected, so client errors leave the denominator. The correctness smoke still fails an unexpected status at 0%. One 5xx or one exception fails the ramp step even when the sample is shorter than the floor, on the version that served it and across the worker, so the availability burn is the rate view for a window that is already large.
Correctness is one lifecycle smoke, not a ramp step. While gates.smoke is hard, any failed step fails the smoke stage (decision=stop, canary left at 0%). informational records the failure and continues. The burn from the pass/fail counts is stored on the run. It is what a later alert would page on. It is not a second gate inside the ramp.
Wall time against the old version is not a ramp gate unless the policy sets gates.latency.mode to gate. The checked-in mode is informational: the numbers are stored and printed, and a shift does not roll back. Read and write burn still use the ceilings in the table above. Those are burn gates, not a version-versus-version compare. gate rolls back only when both sides have minSamples wall samples and the new p99 is above p99Ratio times the old p99 and more than effectFloorMs slower. A shorter sample does not roll back on latency. Trace-based latency is future work, after the rollout mechanics are trusted.
Alerting policy for later, not applied on a ramp step. These are the workbook's 30-day windows:
| Severity | Burn | Long window | Short window | Budget spent if both fire |
|---|---|---|---|---|
| page | 14.4× | 1h | 5m | about 2% |
| ticket | 6× | 6h | 30m | about 5% |
| ticket | 1× | 3d | 6h | about 10% |
Delivery
Every event is appended to the run's events.jsonl and to the ledger (PLY_CD_LEDGER, default ~/.local/state/ply-cd/ledger.jsonl). run_finished carries the fields a delivery report needs: ply sha, ply merge time, ply-web sha, version id, stage timestamps and durations, verdict, rollback flag and reason, smoke pass/fail counts, and per-SLO burn per step.
npm run cd -- reportA past run stays in the ledger. To mark one as a pipeline false positive, append a note. This does not rewrite earlier lines and does not open the run directory:
npm run cd -- annotate 2026-11-01T00-10-02-620Z --false-positive "5% ramp Mann-Whitney flagged a sub-millisecond wall-time shift"That run's run_finished line still says fault: worker. report then drops it from the deploy count and from the change-failure rate, and prints the reason under false positives.
| Metric | Formula |
|---|---|
| Deployment frequency | deploys / days. A deploy is a run_finished that reached 100% (liveAt set) and whose fault is not harness_fault. Aborted runs, held runs, and harness_fault runs are counted on their own lines and are not deploys. Days are the span from the first deploy timestamp to the last. One timestamp has no rate. |
| Lead time | liveAt − plyMergedAt on a passing deploy. liveAt is when the ramp reached 100%. plyMergedAt is the ply commit's committer time from the GitHub API, or null if that fetch fails (the deploy still finishes). Report p50 and p90. |
| Change failure rate | rollbacks / deploys. A harness fault is not in either number. A run annotated as a pipeline false positive is not in either number, and report lists it on its own. |
| Time to restore | restoredAt − failedAt for a rolled-back deploy. p50 and p90. |
| Stage duration | p50 and p90 of each stage's durationMs, across runs. Percentile is sorted[floor((n − 1) · p)]. |
Debrief
The event log is the record of the run. Every line is one JSON object:
| Field | What |
|---|---|
seq | 1, 2, 3, … inside that run |
ts | ISO-8601 with a numeric offset |
run_id | The run directory name |
stage | run, bump, deploy-canary, smoke, telemetry, ramp, rollback, or the live variants |
kind | One of the variants below |
payload | The variant's fields |
| Kind | Payload |
|---|---|
lifecycle | run_started (config, node/cf/git versions, ply and ply-web shas, host, and release as {ply, web} or null), resumed (the same run id, the ply sha, and the release pair), or run_finished (verdict, rollback, smoke counts, stage timings, per-step burns, live/fail/restore times, fault, and the release pair) |
command | argv with secrets already replaced by <redacted>, cwd, exit code, duration, sha256 of stdout and stderr, and the paths of those captures |
observation | The measurement, the exact cf o11y query (or the smoke source) and its window, and responsePath of the raw body (blobs/<sha256>.gz; raw/<seq>.json before 2026-11-07) |
decision | inputs (one or more earlier observation seqs), rules (name and value from the policy or slo.ts, the smoke rule, or reconcile with the next stage), outcome (advance, hold, insufficient_data, insufficient_samples, rollback, stop, harness_fault, state_conflict, cleanup_leak), and reason. cleanup_leak names a scratch repo and does not fail the run |
policy | sha256 of the policy file bytes, the path, the body that ran, and the release pair (ply, web) or null. The debrief prints this first so the rules of that run are the ones you read |
state_change | Traffic split before and after, and the deployment id when cf returned one |
request | One smoke or probe call: method, url, status, version, latency, cf-ray, x-ply-request-id, x-artifacts-ray |
error | An unexpected failure and a JSON context |
annotation | Appended to the ledger for a past run: note false_positive and a one-line reason. npm run cd -- annotate <run-id> --false-positive "reason" writes it. Earlier lines stay as they were |
A decision cannot be written unless inputs cites an earlier observation. A Secret cannot be serialized: JSON.stringify throws, and command argv stores <redacted> instead of the value. Each Secret's text is removed by exact match from every capture and event. An art_v plus a version plus _ is removed the same way, as a backstop, whatever the version number is.
The ramp emits insufficient_data when the new version was not seen enough times and the policy holds, and hold when a routed-share miss is configured to hold. The checked-in file rolls a share miss back and holds on thin evidence. stop is a gate that leaves traffic where it is (the pinned smoke failure, while smoke is hard). rollback is traffic returned to the previous version.
After run_finished the run drops blobs/ on a clean pass, writes manifest.json with the sha256 of every other file, and then refuses further appends. Do not edit the run directory; retention compacts it later. A debrief writes beside it:
npm run cd -- debrief ~/.local/state/ply-cd/runs/2026-10-31T21-05-00-123ZThat prints debrief: ~/.local/state/ply-cd/runs/2026-10-31T21-05-00-123Z.debrief/debrief.md. The markdown stands alone: the policy body and its sha256, a timeline, every decision with the rule and the observations it used, every traffic change, three charts (split.png is every state_change, including the 0% canary, against routedNewShare from telemetry, latency.png wall p99 by version labeled with the run's latency mode, burn.png burn per SLO with a line at the policy's fast-burn multiple), and links back to events.jsonl and to whichever cited blobs were kept (raw/ and captures/ in a pre-2026-11-07 run).
Later
This runbook is the operator path. The same stages are what a later driver should call. That driver is a Worker fed by Cloudflare Artifacts repo events, not a GitHub Actions workflow.