{ "steps": [ { "percent": 10, "dwellMs": 30000 }, { "percent": 25, "dwellMs": 25000 }, { "percent": 50, "dwellMs": 25000 }, { "percent": 75, "dwellMs": 25000 }, { "percent": 100, "dwellMs": 40000 } ], "ingestWaitMs": 20000, "ingestPollMs": 3000, "warmupMs": 10000, "calibration": { "steps[*].dwellMs": { "source": "ledger ramp-step observations, runs 2026-11-06T17-35-09-419Z through 2026-11-06T21-06-26-841Z (10 steps at 10%, 8 at 25%, 8 at 50%, 7 at 75%); live-check stage durations of the 10 direct-promote runs 2026-11-06T21-45-51-484Z through 2026-11-07T19-39-00-604Z", "date": "2026-11-08; re-observed 2026-11-10", "window": "every 10/25/50/75% ramp step in the ledger; smoke-live + telemetry-live of every promote run", "n": 34, "observed": "probes issue 50 req/s (25 rps x 2 calls). At 40 s the 10% steps logged agg 1978-2000 (0.989-1.000 of issued) and new 174-214, share 0.0874-0.1076 (0.0712 on 18-54-31-261Z, which rolled back on that share before warm-up existed; since warm-up 0.1003-0.107). Min share at 25/50/75: 0.232/0.4765/0.734. Live smoke + live telemetry in the promote runs took 42-86 s (median 49 s) after the 100% write. 2026-11-10 (run 2026-11-10T14-56-52-021Z, 480225db): the live checks ran 92 s after the 100% write (visual 2.5 s, smoke-live 68.5 s, telemetry-live 18.9 s), past the 42-86 s range; inside the 100% step's query window they were 129 of 2,091 new-version invocations (74 live-smoke, 55 visual), and they set its p99 wall to 696 ms where its 1,962 probes were 14 ms (ply-web#73).", "headroom": "10%: 30000 issues 1500; at the worst post-stall share 0.0874 that is 131 new against split.minNewRequests 100 (2.8 sd above it; at the post-warm-up 0.10 it is 150, 4.6 sd). 25/50/75%: 25000 issues 1250, 25% above probe.minSamples 1000 (a clean window sheds 0), and the smallest new arm is 290 (0.232 x 1250), 2.9x the floor; detectable new-version error rate at 25% goes from 0.60% (n=500) to about 1.0% (n=290). 100%: 40000 unchanged: the step runs beside the live checks (median 49 s), so a shorter dwell would not end the run sooner, and it keeps n=2000 (0.15%) for the live watch. 2026-11-10: dwell unchanged; the 100% step's latency is now read from its probes only (ramp.ts LatencySample), so its p99 compares with the 10-75% steps whatever the live checks do inside the dwell." }, "ingestPollMs": { "source": "the polled ingest wait of the 10 direct-promote runs 2026-11-06T21-45-51-484Z through 2026-11-07T19-39-00-604Z (promote.ingestPollMs 3000, ceiling 30000)", "date": "2026-11-08; re-observed 2026-11-10", "window": "the 100% watch of each run, polls after the dwell end", "n": 11, "observed": "the first poll returned 5.5-8.6 s after the dwell end and each later one 5.4-11.1 s after the one before (3 s sleep + the query); 98% of answered probes were logged by 8.6-19.2 s after the dwell end (median 12.6 s); the first poll saw 0.877-0.947 (one run 1.132 with live-smoke traffic counted). CORRECTION 2026-11-08: these runs postdate cdb8e00, whose row dedupe deleted rows sharing a $metadata.id (fixed ace0eb9), so the landed fractions read low and 8.6-19.2 s are upper bounds. 3000 is a PRIOR period and does not change. 2026-11-10 (run 2026-11-10T14-56-52-021Z): the stop rule was met at +21.5 s (10%), +23.9 s (25%), +23.2 s (75%) and +19.8 s (100%) after the dwell end, past the 8.6-19.2 s predicted; at 50% the second query returned at +42.7 s with 1,037 of 1,250 probes (83%) and the ceiling ended the wait, so that step was judged on a partial window. At 100% the rule counted the live checks' requests (2,024 events met 98% of 2,000 probes; a later read-only re-query found 1,962 probes of 2,091).", "headroom": "PRIOR period: 3000 keeps one query in flight. Replaces a fixed 20000 sleep per step that also ran before a 20-30 s query on the pre-slicing ramps (2026-11-06T20-24-45-549Z, 21-06-26-841Z: 44-66 s from dwell end to decision). 2026-11-10: the rule now counts the latency sample's invocations (Observation.landed: the probes joined on cf-ray), not every event in the window. Period unchanged." }, "ingestWaitMs": { "source": "runs 2026-11-08T16-58-16-960Z (ceiling 20000) and 2026-11-08T17-07-46-259Z (ceiling 30000), every ramp step; run 2026-11-01T00-10-02-620Z (12640 ms and 17313 ms gaps); the 10 polled promote watches 2026-11-06T21-45-51-484Z through 2026-11-07T19-39-00-604Z", "date": "2026-11-08; re-observed 2026-11-10", "window": "the polls after each dwell end", "n": 27, "observed": "2026-11-08T17-07-46-259Z (ceiling 30000): every step's logged count stopped rising at the 4th poll, 21.8-24.1 s after the dwell end (10% 1405, 25% 1160, 50% 1150, 75% 1142, 100% 1806), and the two later polls returned the same count. The plateau of ~93% of issued probes at partial splits was the row-dedupe bug fixed in ace0eb9 (rows sharing a $metadata.id were deleted), not ingest lag and not Workers Observability loss. 2026-11-08T16-58-16-960Z (ceiling 20000) also took 4 polls per step, the last at about +22-24 s. 2026-11-10 (run 2026-11-10T14-56-52-021Z, n=5 steps): final polls at +19.8-23.9 s after the dwell end on four steps; the 50% step's last query returned at +42.7 s, past the ceiling, holding 83% of its probes.", "headroom": "20000 (unchanged since 2026-11-01): the poll's last query starts at the ceiling and returns at about +22-24 s, where the 2026-11-08 plateau was reached. 30000 (tried in 0016dae) added only flat polls: ramp stage 374 s against 325 s. Re-checked 2026-11-08 after ace0eb9 and 633bfdb: value unchanged. Run 2026-11-08T18-30-12-168Z (d3a35bb1) has one external fetch row per responded probe at 10/25/50/75% (1499/1250/1250/1250, 0 absent); its polls stopped early because other traffic counts toward INGEST_LANDED, so it does not re-derive this ceiling. 2026-11-10: the +22-24 s return assumed queries of a few seconds; one 50% query took about 36 s and the step was judged on 83% of its probes. Value unchanged pending more runs on the probe-counted stop rule (prediction: with landed counted on probes, steps stop at +20-24 s; a step that reaches the ceiling below 98% recurs at most once in 5 runs)." } }, "probe": { "rps": 25, "maxLossRate": 0.0012658227848101266, "onHarnessFault": "telemetry", "calibration": { "source": "run 2026-11-01T17-06-08-780Z", "date": "2026-11-01", "window": "100% ramp step", "n": 790, "lost": 1, "web": "789ba93", "ply": "1743461", "observedRate": "1/790", "percentile": "one measured step, not a percentile", "headroom": "none above the measured loss; a greater rate is a fault" }, "calls": [ { "method": "GET", "path": "/", "auth": false }, { "method": "GET", "path": "/v1/discover?remote=https%3A%2F%2F5ad2236b39de4c8bd8f6270ea63f3ea9.artifacts.cloudflare.net%2Fgit%2Fpublic-v0%2Fply.git", "auth": true } ], "timeoutMs": 2000, "maxInFlight": 50, "minSamples": 1000, "stallLatencyMs": 1000, "windowAttempts": 2, "windowBackoffMs": 1000, "schedule": { "source": "run 2026-11-01T19-40-35-485Z", "date": "2026-11-01", "window": "50% ramp step", "n": 128, "responded": 118, "lost": 10, "healthyN": 1990, "web": "22bfb79", "oldVersion": "4f3d7acf", "newVersion": "19ee3cab", "healthyP50Ms": "41-101", "observedStallMs": 69830, "dwellMs": 40000, "observedRate": "10/128", "percentile": "healthy responses p50 41-101ms on both versions in the first seconds of the step (n=118); the stalled cohort latencyMs ~69830 on both versions; not a worker percentile", "headroom": "timeoutMs 2000 is above 10× the healthy p50 upper bound (101ms → 1010ms) and below observedStallMs 69830, so a client stall is timed_out instead of holding the dwell. stallLatencyMs 1000 is PRIOR: a round threshold above that 101ms p50 and below timeoutMs. maxInFlight 50 is PRIOR: the previous cap was one second of ticks (ceil(rps)=25) times 2 calls; healthy in-flight at p50 101ms and 50 requests/s is about 5. The 50% step hit that cap and issuance stopped until 14:49:39 EST, about 3m14s after 14:46:25 instead of dwellMs 40000. minSamples 1000 is PRIOR: a round floor above the stalled denominator 128 and below the healthy 10/25 steps of about 1990, so the loss-rate gate cannot fail a window as small as this stall. windowAttempts 2 is the one stalled window plus one re-run; a further stall with 0 server errors is insufficient_samples (no 5xx verdict) and its traffic is settled through onHarnessFault. windowBackoffMs 1000 is PRIOR: the run did not wait between windows." } }, "gates": { "smoke": "hard", "fivexx": "hard", "exceptions": "hard", "errorLogs": "hard", "nonOk": "hard", "burn": "hard", "latency": { "mode": "informational" } }, "burn": { "fastBurn": 14.4, "minSamples": 100 }, "split": { "z": 3, "minNewRequests": 100, "onMismatch": "rollback", "onInsufficient": "hold" }, "telemetry": { "maxLossRate": 0.5, "query": { "attempts": 3, "backoffMs": 1000, "calibration": { "source": "run 2026-11-01T19-17-18-170Z", "date": "2026-11-01", "window": "100% ramp step", "n": 1, "web": "07568e0", "ply": "1743461", "observed": "cf o11y telemetry query --body APIError timeout Envelope Error, exit 1", "percentile": "one measured query, not a percentile", "headroom": "attempts is 3, two past the single timeout that rolled the step back. backoffMs 1000 is PRIOR: the run did not retry." } } }, "cleanup": { "attempts": 3, "backoffMs": 1000, "staleMinutes": 60, "calibration": { "source": "run 2026-11-01T17-29-43-122Z", "date": "2026-11-01", "window": "post-ramp scratch repo delete", "n": 1, "latencyMs": 103000, "web": "9558b10", "namespace": "ply-loadgen", "observed": "APIError timeout exit 1, then 404 [10200] Repository not found", "percentile": "one measured delete, not a percentile", "headroom": "attempts is 3, one past the timeout-then-404 that confirmed deletion. backoffMs 1000 is PRIOR: the confirming retry was immediate. staleMinutes 60 is PRIOR: the previous sweep age, not a measured leak age." } }, "rate": { "expectMode": "memo", "probes": 30, "probeGapMs": 300, "probePath": "/view?remote=https%3A%2F%2F5ad2236b39de4c8bd8f6270ea63f3ea9.artifacts.cloudflare.net%2Fgit%2Fpublic-v0%2Fply.git", "minChecks": 20, "maxFailOpenRate": 0.1, "maxDroppedFlushes": 0, "calibration": { "source": "Workers Observability ply.rate.check and ply.rate.flush spans, the 7 days to 2026-11-07T17:30Z: every version since the batched limiter shipped (6ed2e746, ae21dee2, a020b1b1, 78ddd5c3, b4bd90a3, 2a4741c0, ccad274e, fc120c0d, 5bd53910, 8df1e5a6, e8846911), runs lat1-lat8, abuse1-5, fault1-2", "date": "2026-11-07", "n": "128,753 ply.rate.flush; ply.rate.check: 36-566 real (non-load) per version, 5,714-115,585 load per version", "observed": "dropped flushes: 1 of 128,753, and it carried an injected fault (a020b1b1, error;attempts=2, run fault2); overflow drops 0. fail_open, batched: load runs 0-0.13% (a020b1b1 13/15,128, 78ddd5c3 9/10,782, 2a4741c0 109/83,574, e8846911 4/11,263); real traffic per version, cold isolates included, worst 4/52 = 7.7% (ccad274e) and 3/41 = 7.3% (2a4741c0), the rest <= 2.8%. Mode: every version configured batched ran 0 real checks on another limiter; 59926117 and e61a5742 (the ctx.exports fallback bug) ran 100% legacy", "headroom": "maxDroppedFlushes 0: the no-fault baseline is 0 in 128,753 (rule of three: < 2.3e-5 per flush), so one drop in a gate window (hundreds of flushes) is far beyond it. maxFailOpenRate 0.1 is above the worst real-traffic share (7.7%, which counted cold-isolate first reads; the gate excludes those) and 75x the load baseline (0.13%); a broken read path fails open on ~100%. minChecks 20 of 30 probes is PRIOR: 30 probes 300 ms apart take ~9 s, and 20 tolerates a third lost to ingest. expectMode memo is RATE_MODE in cloudflare.config.ts since 2026-11-07 (test/cd-rate-gate.test.ts asserts they match): a deliberate RATE_MODE rollback (batched or legacy) changes this too. Memo cold reads while an isolate's first refresh is in flight are stamped rate.cold = isolate_start (start-up, excluded); an idle isolate's document that aged past MEMO_FALLBACK_MS after an ok refresh is rate.cold = memo_expired (excluded and counted on its own line, unless its own refresh fails: ply-web#156, CD 2026-11-12T13-06-51-511Z counted one as 1/29 warm); a cold read after a refresh settled with no document counts as warm fail_open." } } }