main
1{
2 "steps": [
3 { "percent": 10, "dwellMs": 30000 },
4 { "percent": 25, "dwellMs": 25000 },
5 { "percent": 50, "dwellMs": 25000 },
6 { "percent": 75, "dwellMs": 25000 },
7 { "percent": 100, "dwellMs": 40000 }
8 ],
9 "ingestWaitMs": 20000,
10 "ingestPollMs": 3000,
11 "warmupMs": 10000,
12 "calibration": {
13 "steps[*].dwellMs": {
14 "source": "ledger ramp-step observations, runs 2026-11-06T17-35-09-419Z through 2026-11-06T21-06-26-841Z (10 steps at 10%, 8 at 25%, 8 at 50%, 7 at 75%); live-check stage durations of the 10 direct-promote runs 2026-11-06T21-45-51-484Z through 2026-11-07T19-39-00-604Z",
15 "date": "2026-11-08; re-observed 2026-11-10",
16 "window": "every 10/25/50/75% ramp step in the ledger; smoke-live + telemetry-live of every promote run",
17 "n": 34,
18 "observed": "probes issue 50 req/s (25 rps x 2 calls). At 40 s the 10% steps logged agg 1978-2000 (0.989-1.000 of issued) and new 174-214, share 0.0874-0.1076 (0.0712 on 18-54-31-261Z, which rolled back on that share before warm-up existed; since warm-up 0.1003-0.107). Min share at 25/50/75: 0.232/0.4765/0.734. Live smoke + live telemetry in the promote runs took 42-86 s (median 49 s) after the 100% write. 2026-11-10 (run 2026-11-10T14-56-52-021Z, 480225db): the live checks ran 92 s after the 100% write (visual 2.5 s, smoke-live 68.5 s, telemetry-live 18.9 s), past the 42-86 s range; inside the 100% step's query window they were 129 of 2,091 new-version invocations (74 live-smoke, 55 visual), and they set its p99 wall to 696 ms where its 1,962 probes were 14 ms (ply-web#73).",
19 "headroom": "10%: 30000 issues 1500; at the worst post-stall share 0.0874 that is 131 new against split.minNewRequests 100 (2.8 sd above it; at the post-warm-up 0.10 it is 150, 4.6 sd). 25/50/75%: 25000 issues 1250, 25% above probe.minSamples 1000 (a clean window sheds 0), and the smallest new arm is 290 (0.232 x 1250), 2.9x the floor; detectable new-version error rate at 25% goes from 0.60% (n=500) to about 1.0% (n=290). 100%: 40000 unchanged: the step runs beside the live checks (median 49 s), so a shorter dwell would not end the run sooner, and it keeps n=2000 (0.15%) for the live watch. 2026-11-10: dwell unchanged; the 100% step's latency is now read from its probes only (ramp.ts LatencySample), so its p99 compares with the 10-75% steps whatever the live checks do inside the dwell."
20 },
21 "ingestPollMs": {
22 "source": "the polled ingest wait of the 10 direct-promote runs 2026-11-06T21-45-51-484Z through 2026-11-07T19-39-00-604Z (promote.ingestPollMs 3000, ceiling 30000)",
23 "date": "2026-11-08; re-observed 2026-11-10",
24 "window": "the 100% watch of each run, polls after the dwell end",
25 "n": 11,
26 "observed": "the first poll returned 5.5-8.6 s after the dwell end and each later one 5.4-11.1 s after the one before (3 s sleep + the query); 98% of answered probes were logged by 8.6-19.2 s after the dwell end (median 12.6 s); the first poll saw 0.877-0.947 (one run 1.132 with live-smoke traffic counted). CORRECTION 2026-11-08: these runs postdate cdb8e00, whose row dedupe deleted rows sharing a $metadata.id (fixed ace0eb9), so the landed fractions read low and 8.6-19.2 s are upper bounds. 3000 is a PRIOR period and does not change. 2026-11-10 (run 2026-11-10T14-56-52-021Z): the stop rule was met at +21.5 s (10%), +23.9 s (25%), +23.2 s (75%) and +19.8 s (100%) after the dwell end, past the 8.6-19.2 s predicted; at 50% the second query returned at +42.7 s with 1,037 of 1,250 probes (83%) and the ceiling ended the wait, so that step was judged on a partial window. At 100% the rule counted the live checks' requests (2,024 events met 98% of 2,000 probes; a later read-only re-query found 1,962 probes of 2,091).",
27 "headroom": "PRIOR period: 3000 keeps one query in flight. Replaces a fixed 20000 sleep per step that also ran before a 20-30 s query on the pre-slicing ramps (2026-11-06T20-24-45-549Z, 21-06-26-841Z: 44-66 s from dwell end to decision). 2026-11-10: the rule now counts the latency sample's invocations (Observation.landed: the probes joined on cf-ray), not every event in the window. Period unchanged."
28 },
29 "ingestWaitMs": {
30 "source": "runs 2026-11-08T16-58-16-960Z (ceiling 20000) and 2026-11-08T17-07-46-259Z (ceiling 30000), every ramp step; run 2026-11-01T00-10-02-620Z (12640 ms and 17313 ms gaps); the 10 polled promote watches 2026-11-06T21-45-51-484Z through 2026-11-07T19-39-00-604Z",
31 "date": "2026-11-08; re-observed 2026-11-10",
32 "window": "the polls after each dwell end",
33 "n": 27,
34 "observed": "2026-11-08T17-07-46-259Z (ceiling 30000): every step's logged count stopped rising at the 4th poll, 21.8-24.1 s after the dwell end (10% 1405, 25% 1160, 50% 1150, 75% 1142, 100% 1806), and the two later polls returned the same count. The plateau of ~93% of issued probes at partial splits was the row-dedupe bug fixed in ace0eb9 (rows sharing a $metadata.id were deleted), not ingest lag and not Workers Observability loss. 2026-11-08T16-58-16-960Z (ceiling 20000) also took 4 polls per step, the last at about +22-24 s. 2026-11-10 (run 2026-11-10T14-56-52-021Z, n=5 steps): final polls at +19.8-23.9 s after the dwell end on four steps; the 50% step's last query returned at +42.7 s, past the ceiling, holding 83% of its probes.",
35 "headroom": "20000 (unchanged since 2026-11-01): the poll's last query starts at the ceiling and returns at about +22-24 s, where the 2026-11-08 plateau was reached. 30000 (tried in 0016dae) added only flat polls: ramp stage 374 s against 325 s. Re-checked 2026-11-08 after ace0eb9 and 633bfdb: value unchanged. Run 2026-11-08T18-30-12-168Z (d3a35bb1) has one external fetch row per responded probe at 10/25/50/75% (1499/1250/1250/1250, 0 absent); its polls stopped early because other traffic counts toward INGEST_LANDED, so it does not re-derive this ceiling. 2026-11-10: the +22-24 s return assumed queries of a few seconds; one 50% query took about 36 s and the step was judged on 83% of its probes. Value unchanged pending more runs on the probe-counted stop rule (prediction: with landed counted on probes, steps stop at +20-24 s; a step that reaches the ceiling below 98% recurs at most once in 5 runs)."
36 }
37 },
38 "probe": {
39 "rps": 25,
40 "maxLossRate": 0.0012658227848101266,
41 "onHarnessFault": "telemetry",
42 "calibration": {
43 "source": "run 2026-11-01T17-06-08-780Z",
44 "date": "2026-11-01",
45 "window": "100% ramp step",
46 "n": 790,
47 "lost": 1,
48 "web": "789ba93",
49 "ply": "1743461",
50 "observedRate": "1/790",
51 "percentile": "one measured step, not a percentile",
52 "headroom": "none above the measured loss; a greater rate is a fault"
53 },
54 "calls": [
55 { "method": "GET", "path": "/", "auth": false },
56 { "method": "GET", "path": "/v1/discover?remote=https%3A%2F%2F5ad2236b39de4c8bd8f6270ea63f3ea9.artifacts.cloudflare.net%2Fgit%2Fpublic-v0%2Fply.git", "auth": true }
57 ],
58 "timeoutMs": 2000,
59 "maxInFlight": 50,
60 "minSamples": 1000,
61 "stallLatencyMs": 1000,
62 "windowAttempts": 2,
63 "windowBackoffMs": 1000,
64 "schedule": {
65 "source": "run 2026-11-01T19-40-35-485Z",
66 "date": "2026-11-01",
67 "window": "50% ramp step",
68 "n": 128,
69 "responded": 118,
70 "lost": 10,
71 "healthyN": 1990,
72 "web": "22bfb79",
73 "oldVersion": "4f3d7acf",
74 "newVersion": "19ee3cab",
75 "healthyP50Ms": "41-101",
76 "observedStallMs": 69830,
77 "dwellMs": 40000,
78 "observedRate": "10/128",
79 "percentile": "healthy responses p50 41-101ms on both versions in the first seconds of the step (n=118); the stalled cohort latencyMs ~69830 on both versions; not a worker percentile",
80 "headroom": "timeoutMs 2000 is above 10× the healthy p50 upper bound (101ms → 1010ms) and below observedStallMs 69830, so a client stall is timed_out instead of holding the dwell. stallLatencyMs 1000 is PRIOR: a round threshold above that 101ms p50 and below timeoutMs. maxInFlight 50 is PRIOR: the previous cap was one second of ticks (ceil(rps)=25) times 2 calls; healthy in-flight at p50 101ms and 50 requests/s is about 5. The 50% step hit that cap and issuance stopped until 14:49:39 EST, about 3m14s after 14:46:25 instead of dwellMs 40000. minSamples 1000 is PRIOR: a round floor above the stalled denominator 128 and below the healthy 10/25 steps of about 1990, so the loss-rate gate cannot fail a window as small as this stall. windowAttempts 2 is the one stalled window plus one re-run; a further stall with 0 server errors is insufficient_samples (no 5xx verdict) and its traffic is settled through onHarnessFault. windowBackoffMs 1000 is PRIOR: the run did not wait between windows."
81 }
82 },
83 "gates": {
84 "smoke": "hard",
85 "fivexx": "hard",
86 "exceptions": "hard",
87 "errorLogs": "hard",
88 "nonOk": "hard",
89 "burn": "hard",
90 "latency": { "mode": "informational" }
91 },
92 "burn": {
93 "fastBurn": 14.4,
94 "minSamples": 100
95 },
96 "split": {
97 "z": 3,
98 "minNewRequests": 100,
99 "onMismatch": "rollback",
100 "onInsufficient": "hold"
101 },
102 "telemetry": {
103 "maxLossRate": 0.5,
104 "query": {
105 "attempts": 3,
106 "backoffMs": 1000,
107 "calibration": {
108 "source": "run 2026-11-01T19-17-18-170Z",
109 "date": "2026-11-01",
110 "window": "100% ramp step",
111 "n": 1,
112 "web": "07568e0",
113 "ply": "1743461",
114 "observed": "cf o11y telemetry query --body APIError timeout Envelope Error, exit 1",
115 "percentile": "one measured query, not a percentile",
116 "headroom": "attempts is 3, two past the single timeout that rolled the step back. backoffMs 1000 is PRIOR: the run did not retry."
117 }
118 }
119 },
120 "cleanup": {
121 "attempts": 3,
122 "backoffMs": 1000,
123 "staleMinutes": 60,
124 "calibration": {
125 "source": "run 2026-11-01T17-29-43-122Z",
126 "date": "2026-11-01",
127 "window": "post-ramp scratch repo delete",
128 "n": 1,
129 "latencyMs": 103000,
130 "web": "9558b10",
131 "namespace": "ply-loadgen",
132 "observed": "APIError timeout exit 1, then 404 [10200] Repository not found",
133 "percentile": "one measured delete, not a percentile",
134 "headroom": "attempts is 3, one past the timeout-then-404 that confirmed deletion. backoffMs 1000 is PRIOR: the confirming retry was immediate. staleMinutes 60 is PRIOR: the previous sweep age, not a measured leak age."
135 }
136 },
137 "rate": {
138 "expectMode": "memo",
139 "probes": 30,
140 "probeGapMs": 300,
141 "probePath": "/view?remote=https%3A%2F%2F5ad2236b39de4c8bd8f6270ea63f3ea9.artifacts.cloudflare.net%2Fgit%2Fpublic-v0%2Fply.git",
142 "minChecks": 20,
143 "maxFailOpenRate": 0.1,
144 "maxDroppedFlushes": 0,
145 "calibration": {
146 "source": "Workers Observability ply.rate.check and ply.rate.flush spans, the 7 days to 2026-11-07T17:30Z: every version since the batched limiter shipped (6ed2e746, ae21dee2, a020b1b1, 78ddd5c3, b4bd90a3, 2a4741c0, ccad274e, fc120c0d, 5bd53910, 8df1e5a6, e8846911), runs lat1-lat8, abuse1-5, fault1-2",
147 "date": "2026-11-07",
148 "n": "128,753 ply.rate.flush; ply.rate.check: 36-566 real (non-load) per version, 5,714-115,585 load per version",
149 "observed": "dropped flushes: 1 of 128,753, and it carried an injected fault (a020b1b1, error;attempts=2, run fault2); overflow drops 0. fail_open, batched: load runs 0-0.13% (a020b1b1 13/15,128, 78ddd5c3 9/10,782, 2a4741c0 109/83,574, e8846911 4/11,263); real traffic per version, cold isolates included, worst 4/52 = 7.7% (ccad274e) and 3/41 = 7.3% (2a4741c0), the rest <= 2.8%. Mode: every version configured batched ran 0 real checks on another limiter; 59926117 and e61a5742 (the ctx.exports fallback bug) ran 100% legacy",
150 "headroom": "maxDroppedFlushes 0: the no-fault baseline is 0 in 128,753 (rule of three: < 2.3e-5 per flush), so one drop in a gate window (hundreds of flushes) is far beyond it. maxFailOpenRate 0.1 is above the worst real-traffic share (7.7%, which counted cold-isolate first reads; the gate excludes those) and 75x the load baseline (0.13%); a broken read path fails open on ~100%. minChecks 20 of 30 probes is PRIOR: 30 probes 300 ms apart take ~9 s, and 20 tolerates a third lost to ingest. expectMode memo is RATE_MODE in cloudflare.config.ts since 2026-11-07 (test/cd-rate-gate.test.ts asserts they match): a deliberate RATE_MODE rollback (batched or legacy) changes this too. Memo cold reads while an isolate's first refresh is in flight are stamped rate.cold = isolate_start (start-up, excluded); an idle isolate's document that aged past MEMO_FALLBACK_MS after an ok refresh is rate.cold = memo_expired (excluded and counted on its own line, unless its own refresh fails: ply-web#156, CD 2026-11-12T13-06-51-511Z counted one as 1/29 warm); a cold read after a refresh settled with no document counts as warm fail_open."
151 }
152 }
153}
Diagnostics4 git calls · 298 ms
CallStatusSizeTime0298 ms
discover200152 B3 ms
git.upload-pack discover http.response.status_code=200 git.upstream.cf_ray=- git.upstream.phase=ls-refs git.upstream.dur_ms=3 git.response.bytes=152 git.server.agent=gitty/1.0 git.server.object_format=sha1 git.server.caps=agent ls-refs fetch object-format git.server.fetch=shallow filter sideband-all git.protocol.version=2 git.pkt.frames=8 git.pkt.data=6 git.pkt.flush=2 git.pkt.delim=0 git.pkt.bytes=152 git.pkt.payload.max=34
ls-refs200145 B · hit3 ms
git.upload-pack ls-refs http.response.status_code=200 git.upstream.cf_ray=- git.upstream.phase=ls-refs git.upstream.dur_ms=3 git.response.bytes=145 git.response.cache=HIT git.response.duration_ms=3 git.protocol.version=2 git.pkt.frames=3 git.pkt.data=2 git.pkt.flush=1 git.pkt.delim=0 git.pkt.bytes=145 git.pkt.payload.max=76 git.ls_refs.count=2 git.ls_refs.head=refs/heads/main
fetchblob:none · deepen 1020040 KiB · hit3 ms
git.upload-pack fetch http.response.status_code=200 git.upstream.cf_ray=- git.upstream.phase=fetch git.upstream.dur_ms=2 git.response.bytes=41349 git.response.cache=HIT git.response.duration_ms=3 git.fetch.filter=blob:none git.fetch.filter.requested=tree:0 git.fetch.filter.fallback=blob:none git.fetch.filter.learned=false git.fetch.wants=526b305930b8ef382f2d7a1f9c0238730a585921 git.fetch.haves.count=0 git.fetch.deepen=10 git.fetch.thin_pack=false git.fetch.ofs_delta=true git.protocol.version=2 git.pkt.frames=6 git.pkt.data=4 git.pkt.flush=1 git.pkt.delim=1 git.pkt.bytes=41349 git.pkt.payload.max=41254 git.sideband.band1.bytes=41253 git.sideband.band2.bytes=0 git.sideband.band3.bytes=0 git.pack.objects=61 git.pack.bytes=41253 git.pack.ofs_delta=0 git.pack.ref_delta=0 git.pack.chain.max=0 git.pack.trailer=ab60efef4b7243678c75de64a4386986ee885c1c git.pack.full=61Open pack map
fetch2005.0 KiB292 ms
git.upload-pack fetch http.response.status_code=200 git.upstream.cf_ray=- git.upstream.phase=fetch git.upstream.dur_ms=292 git.response.bytes=5165 git.response.cache=MISS git.response.duration_ms=292 git.fetch.wants=88e9690533b0d457dd497e63b44ac8a0c5155dd9 git.fetch.haves.count=0 git.fetch.deepen=0 git.fetch.thin_pack=false git.fetch.ofs_delta=true git.protocol.version=2 git.pkt.frames=3 git.pkt.data=2 git.pkt.flush=1 git.pkt.delim=0 git.pkt.bytes=5165 git.pkt.payload.max=5144 git.sideband.band1.bytes=5143 git.sideband.band2.bytes=0 git.sideband.band3.bytes=0
Each row is one HTTP round trip ply made to the git server for this page, on one timeline. discover asks what the server speaks, ls-refs lists branch tips, fetch brings objects by id. A hit was served by Workers Cache. Tap a row for its wire details.
- Remote
- https://5ad2236b39de4c8bd8f6270ea63f3ea9.artifacts.cloudflare.net/git/public-v0/ply-web.git
- Ref
- refs/heads/main
- Commit
- 526b305930b8ef382f2d7a1f9c0238730a585921
- cf-ray
- a5e2c8c66fbb4b67
- Request
- 0da1cc16…
- Protocol
- v2 · sha1
- Agent
- gitty/1.0
- Filter
- advertised
- Fetch
- shallow filter sideband-all
- Capabilities
- agent ls-refs fetch object-format