main
ply4 git calls182 ms53 KiB2 cachedprotocol v2
319 lines28 KiBRaw

Rate limit: batched writes, cached reads

Status: live. Batched since 78ddd5c3 (2026-11-06); memo (no I/O on the request path) is the default since dbabbca6 (2026-11-07). Replaces the per-request ViewerRateLimit call on the request path. Measurements land in the Results section with version ids and run ids.

Why

ViewerRateLimit is a SQLite Durable Object sharded by SHA-256(CF-Connecting-IP), called synchronously before any git work. On 6f2897ff / c43f01cd that call was 22–49 ms of a 45–57 ms warm /r/ply page and 117 ms on a fresh isolate. The limiter is the largest single bucket of a warm page.

Shape

request ─▶ default entrypoint
            ├─ READ   ctx.exports.RateCache.fetch(/blocked/v1)   ◀── Workers Cache (max-age 1, SWR 30), one key per colo
            │           └─ MISS/STALE revalidation: RateCounter.blockedDoc()
            │         blocked? ─▶ 429 (no git work)
            └─ WRITE  allowed ─▶ isolate accumulator {key,bucket → n}
                        the request that opens a batch: ctx.waitUntil(linger 200 ms → flush)
                        flush: RateCounter.apply({id: <isolate>:<seq>, entries}) raced against a per-attempt timer,
                               retried with the SAME id until acked or 5 s from the first attempt
RateCounter (ONE SQLite Durable Object, name "global-v1", reached as ctx.exports.RateCounter)
   rate_batches(id)          idempotency keys, kept 10 min: a replayed id is acked and changes nothing
   rate_hits(k,b,slot,n)     10 s slots, sliding 60 s window
   rate_blocked(k,b,until)   keys at or over the limit; the read side's whole document

Nothing on the request path waits for the Durable Object except a cache MISS/EXPIRED of the one document.

Write side

  • Accumulator. Per isolate, one open batch: Map<"<key16>:<bucket>", count>. The first admitted request after a flush opens a batch and schedules its flush 200 ms later (LINGER_MS) from ctx.waitUntil. Workers runs no timers outside an invocation, so the waitUntil is what keeps the isolate alive through the linger and the retries. Every request that joins the batch also waitUntils the same promise (free, and the batch survives any one request context ending). A batch that reaches 2048 keys (MAX_BATCH_KEYS) is closed and flushed at once. At most MAX_INFLIGHT (64) batches flush at once (beyond: dropped, rate.batch.dropped=overflow), and at most MAX_KEYS_HELD (16,384) distinct keys are held across the open batch and every flushing batch (since e6baadd; the old bound was 64 × 2048 = 131,072 while the counter is slow). A hit for a new key past it is dropped and recorded (rate.batch.dropped=budget); a key already in the open batch still counts. rate.batch.keys_held is on every check. Node 24, 200k random keys with every flush stuck: 3.0 MB held at the budget vs 20.7 MB unbounded.
  • Idempotency key. <isolate uuid>:<seq>. The uuid is minted on the isolate's first request (randomness is not available at global scope). The DO inserts the id into rate_batches in the same synchronous SQL transaction that adds the counts, so an id applies at most once. Ids are kept 10 min (DEDUPE_RETENTION_MS), 120× the client's 5 s deadline; an attempt still in flight after that is not possible (the isolate's waitUntil is gone by then).
  • Retry (Ivar, 2026-11-06: "we can always retry the DO RPC … race a setTimeout … 5 second max"). Each attempt races apply() against a timer of ATTEMPT_TIMEOUT_MS, capped by what is left of the deadline. Timeout or error → backoff (50 ms, doubling) → the same id again. 5 s after the first attempt (FLUSH_DEADLINE_MS) the batch is dropped and recorded. A timed-out attempt may still apply later; that is exactly what the id covers (tested: a DO that applies after the client gave up, plus the retry, counts once).
  • Per-attempt timeout. 1500 ms = ≈ 2× the measured apply-RPC p99.9 (722 ms, ae21dee2 run lat2, 5,649 flushes). The first cut used 1000 ms: Marseille's RPCs (436–984 ms) timed out and were replayed (counted once, but extra DO load). Lower is not better: an attempt that times out and is retried while the first is merely slow adds a replay, never a double count.
  • Drop, not merge. An expired batch is dropped, not merged into the next batch. Merging moves its counts under a new id, and if the original's last attempt applies late the counts land twice: the one property the id exists to give would be gone. Dropping is a bounded under-count (below).
  • Isolate death before flush. Lost hits ≤ the admitted hits that isolate took in the last ≤ 200 ms plus its batches still retrying (≤ 5 s). A crash or eviction during a waitUntil is rare (it outlives the response by up to 30 s); a deploy does not cut a running waitUntil short. For one client the under-count is ≤ its own requests to that isolate in that window: at the 60/min limit, ≤ 1 request (200 ms) and ≤ 5 requests (5 s of retries).

Durable Object side

  • One object (RateCounter, a new SQLite class, additive: ViewerRateLimit and its namespace stay). It is reached through ctx.exports.RateCounter, not an env binding: the versions API refuses a binding to a class that no deploy has provisioned yet (first upload, 2026-11-06). apply is synchronous SQL (no await between the dedupe check and the writes), so it is atomic under the input gate.
  • Window. Sliding 60 s over 10 s slots. A slot counts while its end is within the last 60 s, so a hit counts for 60–70 s: never more lenient than an exact sliding window (worst case 1/6 stricter at the edge).
  • Blocked set. After applying, every touched key whose window sum is ≥ its limit is upserted into rate_blocked with until (the time the sum next drops below the limit), crossed (when it first went over) and peak (the highest admitted sum). peak − limit is the measured overshoot.
  • Pruning every ≥ 5 s inside apply: expired slots, ids older than 10 min, ended blocks. Per table at most 128 rows (PRUNE_BATCH_ROWS) plus the rows the apply itself added (rate_hits: + its entries; rate_blocked: + its new blocks; since e6baadd). A full slice leaves a backlog and the next apply prunes again instead of waiting 5 s, so a backlog (an idle spell, a restarted object) drains in bounded slices, and it shrinks under any inflow: each apply takes out at least what it put in (unit test: 61,440 expired rows drained by full 2,048-key applies while the inflow continued). One apply writes at most its own rows + its own entries + 128 per table. Before the bound one apply on 78ddd5c3 wrote 12,892 rows. ply.rate.apply logs pruned_hits/batches/blocked and prune_backlog; ply.rate.stats (every 30 s) logs the expired rows still stored per table and databaseSize.
  • Index. rate_blocked(until) (e6baadd) serves the document query (WHERE until > ? ORDER BY until DESC LIMIT 4096, every revalidation) and the blocked prune; before it the plan was SCAN rate_blocked | USE TEMP B-TREE FOR ORDER BY (node:sqlite EXPLAIN QUERY PLAN; asserted in test/rate.test.ts).
  • Budget. Flush RPCs/s ≤ isolates × 5. Measured in Results (apply ms, rows written, DO CPU per invocation).

Read side

  • One document, one key. RateCache is a cache-enabled named entrypoint (the GitCache / Pinned / FlagsCache pattern; never caches.default). GET https://rate.internal/blocked/v1 answers RateCounter.blockedDoc(): every blocked [key16, bucket, until, crossed], newest 4096 (MAX_DOC_ENTRIES), with Cache-Control: public, max-age=1, stale-while-revalidate=30. A full document is 208,947 bytes; parse + index p50 / p99 1.05 / 2.21 ms (Node 24, 300 runs), ~228 KB of index.
  • Reader bound (e6baadd). The isolate reads the body streamed and stops past 512 KiB (MAX_DOC_BYTES: the body is cancelled, the read is TOO_LARGE, the last document stays in use) and indexes at most 4096 entries (rate.doc_truncated). rate.doc_bytes is on refresh spans. A bad or oversized document cannot grow the isolate.
  • Why not per-client keys. A per-client key misses on every new client and on every client's first request after each TTL: the median viewer client makes a handful of requests, so the hit ratio would sit near the miss path and every miss is a DO round trip on the request path, which is the cost being removed. One document is one MISS per colo per second, whatever the client count. The cost is size: the document is O(blocked clients), capped at 4096 entries × ~40 bytes. Measured hit ratio in Results.
  • In the isolate the parsed document is memoized by its etag: a HIT costs the loopback fetch, not a JSON parse.
  • Read budget (never block on the DO). The cached fetch gets 25 ms (READ_BUDGET_MS; warm HIT/SWR reads on lat2: p99 12 ms, max 46 ms). A MISS is a synchronous DO read (403–481 ms on lat2) and an isolate's first loopback ~55 ms: past the budget the request decides from the isolate's last document (any age ≤ 5 min), or admits when the isolate has none, and the fetch completes in waitUntil, refreshing that document. Extra lag this adds: only in an isolate that has never read the document, for ≤ one DO read (~0.5 s); measured in Results.
  • Memo mode (RATE_MODE=memo, the default since 2026-11-07, dbabbca6; or x-ply-rate-mode: memo). The batched read's p99 (12 ms on a020b1b1 / 78ddd5c3) is the loopback fetch itself: of 401 checks ≥ 10 ms, 363 were cache HIT or UPDATING. In memo mode the request never fetches: it looks its key up in the isolate's parsed document. When that document is ≥ 500 ms old (MEMO_REFRESH_MS) the request still decides from it and starts ONE refresh through the same RateCache entrypoint in ctx.waitUntil (single flight: a request that finds one in flight joins it; a flight unsettled after 10 s, MEMO_REFRESH_ABANDON_MS, is abandoned). An isolate with no document (or one older than 5 min) admits (fail_open, rate.read=cold) and refreshes. Extra lag over batched: ≤ 500 ms + the refresh's read + the isolate's own gap between requests (the memo only refreshes when a request arrives).
  • Failure. If the read fails, the last good document (≤ 5 min old) is used; with none, the request is admitted (fail open) and recorded. Today's limiter fails closed (503); with one object instead of one per client, failing closed would turn a DO incident into a viewer outage. The limit bounds cost; it is not an access control.

Semantics

Approximate. Worst-case enforcement lag after the request that takes a client to its limit:

lag ≤ LINGER (200 ms) + apply RPC (+ retries, ≤ 5 s worst) + max-age (1 s) + one revalidation + the client's next-request gap

Steady state (RPC ≈ tens of ms, traffic flowing) that is ≈ 1.5 s. A client sending 2× the limit (2 req/s at 60/min) overshoots by about lag × rate ≈ 3 requests per window. The DO measures the true overshoot (peak − limit); the load generator measures lag end to end (first 429 after the request that should have been the first refusal).

Identical to today: the limit is on admitted requests (a 429 does not count), page 60/min and pack 20/min per client, retry-after from the block's until.

Privacy

As today: the client key is SHA-256 of ply-viewer-rate/1\n<CF-Connecting-IP>; only its first 16 hex (64 bits) is sent, stored and cached. No address is stored. The blocked document is served only over ctx.exports, never to a browser.

Load-test controls

A request carrying x-ply-load-token: <PLY_LOAD_TOKEN> (compared in constant time; never forwarded, logged or echoed; Workers Logs records the header as REDACTED) may also send x-ply-rate-client (a label hashed in place of the address, so one generator can be many clients), x-ply-rate-mode: legacy|batched|memo (choose the limiter for this request: an A/B inside one version), and x-ply-rate-fault (fault injection on the DO, below). Without a valid token those headers are ignored.

PLY_LOAD_TOKEN is a credential of its own (2026-11-07): it unlocks these headers and nothing else. PLY_TOKEN (the /v1 bearer) does not unlock them, and the load generator (ply-load) holds only PLY_LOAD_TOKEN (for its control API and the header). With no PLY_LOAD_TOKEN bound, no request is a load request. Local copy: ~/.config/ply/load-token (0600).

x-ply-rate-fault: slow-ack=<ms>;attempts=<n> makes the DO apply, then hold its answer <ms> on attempts ≤ n (the client times out and retries; the retry must be a replay). error;attempts=<n> makes the DO throw before applying on attempts ≤ n. A faulted request flushes in a batch of its own, at once, never the isolate's shared batch: a fault slows, errors or drops only that request's own count, never a real client's.

Instrumentation

Workers spans (ctx.tracing), each stamped with ply.route and ply.script_version like ply's spans:

spanattributes
ply.rate.checkrate.mode legacy/batched, rate.mode_source default/header (header: a load request chose the arm), rate.cold isolate_start/memo_expired (an expected cold check: see "CD gate"), rate.bucket, rate.decision allow/deny/fail_open, rate.fail_open_reason (on every fail_open: isolate_start / memo_expired / read_failed / read_late), rate.check_ms, rate.cache (HIT/STALE/MISS/EXPIRED/…), rate.read ok/memo/fail, rate.doc_entries, rate.doc_age_ms, rate.blocked_for_ms (deny: now − crossed)
ply.rate.check (memo)rate.read memo_fresh / memo_stale_refresh / cold, rate.cache MEMO, rate.memo_age_ms (now − the isolate's last read), rate.doc_age_ms (now − the DO's at), rate.refresh.started; cold: rate.memo_expired_ms (the expired document's age, when the isolate held one), rate.cold_last_read (read_failed: how the last refresh ended, fail/memo/abandoned/threw)
ply.rate.refreshrate.refresh.trigger cold/stale, .cold (trigger cold: the starting check's cause), .memo_expired_ms, .read ok/memo/fail, .cache (the RateCache read's cf-cache-status), .parsed, .ms, .joins (requests that found it in flight), rate.doc_age_ms
ply.rate.flushrate.batch.id, .keys, .hits, .linger_ms, .attempts, .attempt_log (ok:34,timeout:1000,…), .timeouts, .errors, .outcome ok/dropped, .deadline_hit, .replay (DO said duplicate), .flush_ms, .alive_ms (batch open → flush end, the waitUntil lifetime), rate.do.apply_ms, rate.do.rows_written, rate.do.new_blocks

DO log lines: ply.rate.apply (id, attempt, replay, keys, hits, apply_ms, rows_written, new_blocks, fault) and ply.rate.over (a key at or over its limit: sum, limit, crossed, peak). Queried by $workers.scriptVersion.id.

CD gate (silent degradation)

The limiter fails soft: a broken read admits (fail_open), a flush that cannot reach the counter is dropped after 5 s, and a missing counter sends every request to legacy (59926117 / e61a5742: nothing alerted). All of those answer 200, so the smoke and the error gates pass. scripts/cd/rate-gate.ts runs inside the telemetry stage of the 0% canary (fail → stop) and of the live watch (fail → rollback): 30 pinned probes of /view?remote=<public ply> (the redirect route: rate-checked, no git work) as a load client with no mode header, then the version's own spans since the stage began, by aggregate count. It fails the version when

  • any default-mode check (rate.mode_source default) ran a limiter other than rate.expectMode (memo since 2026-11-07; test/cd-rate-gate.test.ts asserts it equals RATE_MODE in cloudflare.config.ts);
  • warm default-mode checks (no rate.cold, plus memo_expired checks whose own refresh failed) failed open above 10% (with ≥ 20 of them; fewer is a harness fault);
  • any flush was dropped (deadline) or any batch overflowed without an injected fault.

The thresholds and their evidence (7 days of spans, 128,753 flushes, every version since the limiter shipped) are in scripts/cd/cd.policy.json rate. A deliberate RATE_MODE rollback (batched or legacy) changes rate.expectMode too, in the same commit, shipped through the pipeline: a cf var or secret edit creates and deploys a version at 100% directly (d4e3eea6 did).

Expected cold checks (rate.cold, counted on their own line and kept out of the warm share; ply-web#156):

  • isolate_start: an isolate's first check, and any memo check that is cold while that isolate's first refresh is still in flight.
  • memo_expired: the isolate's last refresh read the document ok, and the document then aged past MEMO_FALLBACK_MS (5 min) because no request came to refresh it (an idle isolate). The check admits and starts a refresh stamped rate.refresh.cold = memo_expired; if that refresh does not read ok, the check is put back as a warm failure. CD makes this likely: the canary isolate takes the 30 probes in ~11 s and then idles through the ramp, so the live watch's first request on it is past 5 min (2026-11-12T13-06-51-511Z: 339.4 s idle, refresh ok in 6 ms, counted 1/29 warm before this).

A cold read after a refresh has settled with no document (rate.fail_open_reason = read_failed: the read path is broken) is a warm fail_open and counts. The gate prints both, e.g. fail_open warm 0/28 = 0.00%; expected fail_open isolate_start 4/4, memo_expired 1/1 (refresh ok 1, failed 0).

Results

Load: ply-load (bench/load), generator DOs in wnam/enam/weur/apac (4 shards each, 5 req/s each, open loop), GET /r/ply, the two arms interleaved per request inside ONE version via x-ply-rate-mode, pinned with Cloudflare-Workers-Version-Overrides. Server numbers: ply.rate.check spans (rate.check_ms) and ply.request logs joined by trace, filtered by ply.script_version. Synthetic clients each stay under the limit.

run · versionarmn checkscheck p50 / p99 / p99.9 / max mspage wall p50 / p99 / p99.9 ms
lat3 · 6ed2e746 (no read budget)legacy13,18715 / 404 / 533 / 137527 / 462 / 1567
lat3 · 6ed2e746 (no read budget)batched13,1883 / 13 / 49 / 37316 / 107 / 1446
lat4 · a020b1b1 (25 ms read budget)legacy13,17714 / 403 / 588 / 327428 / 463 / 3447
lat4 · a020b1b1 (25 ms read budget)batched15,0993 / 12 / 25 / 2516 / 71 / 3049
lat5 · 78ddd5c3 (promoted, unpinned)legacy10,77813 / 424 / 584 / 81926 / 531 / 2739
lat5 · 78ddd5c3 (promoted, unpinned)default (batched)10,7823 / 12 / 25 / 2515 / 193 / 2747
lat6 · 2a4741c0 (pinned, interleaved)batched13,5713 / 14 / 25 / 2512 / 165 / 4690
lat6 · 2a4741c0 (pinned, interleaved)memo13,5720 / 0 / 0 / 09 / 153 / 4702
lat7 · 2a4741c0 (pinned, memo only)memo13,5830 / 0 / 0 / 010 / 69 / 1302

The page p99.9 (≈3 s in both arms) is git work, not the limiter (lat1, A/A legacy, has the same tail).

  • Read cache (lat4): HIT 77.9%, UPDATING (served stale while one request revalidates) 21.9%, past the 25 ms budget 0.19% (29); 13 of those had no document in the isolate and were admitted (fail_open). HIT+SWR 99.8%.

  • Enforcement (abuse runs, 12 clients/arm at 2× the 60/min limit for 100 s): batched lag p50 511 / max 1007 ms (abuse2, a020b1b1; abuse1 6ed2e746: p50 1007, max 1012), overshoot p50 1 / max 2 requests per client (the DO's peak − limit agrees: 2). Legacy: lag 19–44 ms, overshoot 0. Admitted totals over the run are equal (1,440 vs 1,440).

  • Flushes (lat4, a020b1b1, no fault): 12,525 flushes, 99.9% one attempt; 11 timeouts in 12,536 attempts at 1.5 s (lat3 at 1.0 s: 175 in 11,655, every one later a DO replay). RPC p50 / p99 / p99.9 142 / 762 / 1363 ms. Keys per batch p50 1, p99 3: at ~1 req/s per colo an isolate rarely sees two hits in 200 ms, so the DO is written about once per admitted request; the batching pays off only with denser per-isolate traffic.

  • Counter DO (lat3 window): 11,228 invocations (≈47/s); CPU p50 0 / p99 2 / p99.9 7 ms; wall p50 133 / p99 1317 ms. The wall is the SQLite commit behind the output gate, and it is the RPC latency the flush sees.

  • Fault injection in prod (a020b1b1, fault1 SJC / fault2 LHR+MRS, 20–60 s of each run): slow-ack=2500 → 305/305 batches: attempt 1 timed out at 1.5 s, attempt 2 acked as a replay. error;attempts=2 → 280 batches ok on attempt 3, 3 on attempt 4, 1 hit the 5 s deadline (recorded dropped), yet the DO log shows its attempt 3 applied and attempt 4 replayed: counted once. Across 13,978 applies, no id applied twice.

  • After promotion (lat5, 78ddd5c3 at 100%, unpinned, default vs x-ply-rate-mode: legacy, 10,800 requests/arm): all 200, 0 wrong-version. Client TTFB p50 / p99 / p99.9 45 / 935 / 4505 ms default vs 59 / 1035 / 4717 legacy. Read cache HIT 74.7%, UPDATING 25.1%, LATE 0.25% (27; 9 fail_open). 10,563 flushes, all ok on attempt 1 (0 timeouts, 0 replays, 0 deadline); RPC p50 / p99 / p99.9 150 / 290 / 453 ms, a much tighter tail than lat4's 762 / 1363. Real (non-load) traffic on batched in the window: 58 checks, p99 5 ms. DO rows_written p50 6, p99.9 519, max 3983: the large ones are the retention prunes (rate_hits, rate_batches older than 10 min), not hit writes.

  • Memo mode (2a4741c0, 2026-11-07; ply-load ad3db19e). lat6: batched vs memo interleaved per request, pinned, 27,200 requests (13,600/arm), all 200. The memo check is a map lookup: rate.check_ms 0 at every percentile (Workers clocks only advance on I/O, so the page wall from ply.request joined by trace is the comparison). Page wall memo − batched, bootstrap 95% CI (400 resamples): p50 −3 ms [−3, −3], p90 −3 [−4, −3], p99 −12 [−39, +12]. So a 3 ms win at the median and no measurable change at p99, which is git work. Batched reads: ok 13,542, LATE 29 (7 fail_open). The interleaved memo arm shares the isolate's document with the batched arm (it is module state), which keeps it fresh: memo_age p50 200 ms, 26 refreshes. So refresh and lag were measured memo-only. lat7, memo only, 13,583 checks: memo_fresh 9,046, memo_stale_refresh 4,495, cold 42 (fail_open; batched lat6: 7). memo_age p50/p99/p99.9 400 / 602 / 653 ms. 4,517 refreshes (one per 3 requests), refresh ms p50/p99/p99.9/max 4 / 8 / 50 / 301, RateCache HIT 2,944, UPDATING 1,569, EXPIRED 4. 20 single-flight joins (max 5 on one flight).

  • Enforcement, memo vs batched (2a4741c0, 12 clients at 2× the limit, 100 s, client-side lag quantised by the 500 ms request gap): abuse3 interleaved (memo kept fresh by batched): batched lag p50 500 / max 1500 ms, overshoot p50 1 / max 3; memo the same. abuse5 batched only: lag p50 500 / max 1500 ms, overshoot max 3, DO peak − limit 3. abuse4 memo only: lag p50 1500 / max 2000 ms, overshoot p50 3 / max 4, DO peak − limit 4. Memo adds about one request gap plus ≤ 500 ms of memo age on top of the cache's max-age and SWR.

  • Decision (superseded 2026-11-07, below): memo stays opt-in (x-ply-rate-mode: memo / RATE_MODE=memo) and the default stays batched. Its page p99 gain is not measurable and its lag max (2.0 s) is over the ~1.5 s bar. 2a4741c0 was promoted with the default unchanged, which ships the bounded prune.

  • Counter prune bound (PRUNE_BATCH_ROWS 128). ply.rate.apply rows_written max before: 12,892 (78ddd5c3), 7,915 (a020b1b1). After: lat8, 2a4741c0 at 100% unpinned, default arm, 13,600 requests all 200, starting on the backlog that lat6/lat7 left: 6,550 applies, rows_written median 9 / p99 140 / max 268.

  • Bursts (2026-11-07, canary 9d33f400 pinned, ply-load de344fdb, runs burst2-c{1,5}-r{50,100,200}). One client (shared label, 4 regions × 2 generators) at 50 / 100 / 200 req/s for 60 s: admitted in a 60 s window 116 / 125 / 140 (overshoot 56 / 65 / 80; the DO's peak equals the client's count every time). Five clients at 10 / 20 / 40 req/s each: overshoot 6-10 / 14-20 / 28-32. Overshoot ≈ client rate × 0.4-1.1 s. DO crossing → first deny per colo p50 0.5-1.8 s, max 4.5 s. The counter stayed healthy: every flush one attempt, 0 timeouts / replays / drops, apply RPC p99 ≤ 294 ms, CPU p99 ≤ 4 ms, rows_written p99 ≤ 51; denied requests never flush. The most a client's requests landed in one colo before it was refused was 59, so a per-isolate pre-count at the limit would not have bitten here.

  • Memo becomes the default (2026-11-07; Ivar: "Overshoot is not horrible, we budget that into the limit. I care most about hot path perf."). Commits 53a8e47 (RATE_MODE memo, gate expects memo) + e6baadd (memory bounds). Canary dbabbca6-b606-4c97-856b-22423a4a65da at 0% beside 935e8eb1-bdb5-4572-b7de-058d4c51492e (batched; it already ran REFS_POLICY=watched, so the arms differ only in the limiter). ply-load de344fdb, both versions pinned, interleaved per request, GET /r/ply, load checks joined to ply.request by trace:

    run · armcheckscheck p50 / p99 / p99.9 / max mspageswall p50 / p90 / p99 / p99.9 ms
    memoab1 · 935e8eb1 batched14,1383 / 8 / 25 / 2513,87411 / 15 / 34 / 755
    memoab1 · dbabbca6 memo14,1420 / 0 / 0 / 013,9769 / 14 / 30 / 562
    mem1m2 · 935e8eb1 batched531,6563 / 19 / 25 / 25531,76414 / – / 154 / 295
    mem1m2 · dbabbca6 memo531,7280 / 0 / 0 / 0531,59811 / – / 139 / 256

    memoab1: 6 regions × 4 generators × 4 req/s × 300 s, 16 colos. Memo − batched, bootstrap 95%: check −3 [−3, −3] / −4 [−4, −4] (p90) / −8 [−8, −8] / −25 [−25, −25]; wall p50 −2 [−2, −2], p90 −1 [−2, −1], p99 −4 [−8, −1], p99.9 −193 [−720, +409] (the tail is git work). mem1m2 (48 generators × 25 req/s × 900 s, a new client per request, 1,063,558 distinct): every percentile lower on memo (Workers Observability calculations). Memo reads (memoab1): memo_fresh 3,200, memo_stale_refresh 10,918, cold 24 (all start-up, then stamped rate.cold_isolate, now rate.cold = isolate_start); warm fail_open 0 vs 1 on batched. Refreshes 10,977 (one per 1.3 requests), all ok, HIT 80% / UPDATING 20%, ms p50 / p99 / p99.9 / max 3 / 8 / 40 / 173; memo age p50 / p99 500 / 1001 ms. Promoted with npm run cd (canary + live rate gate: memo on every default check, warm fail_open 0/26 and 0/37, 0 drops; visual smoke 32/32). Previous: 935e8eb1.

  • Enforcement with memo (the overshoot budget). abuse6 (dbabbca6 pinned, 12 clients at 2 req/s = 2× the limit, 4 regions, 100 s): lag p50 / p90 / max 1000 / 2000 / 2500 ms (client clock, quantised by the 500 ms gap), overshoot p50 / p90 / max 2 / 4 / 5 requests per 60 s window; DO peak max 65. blk2 (1,536 clients at 1.2× the limit, each on one version, under counter load): batched lag p50 / p99 / max 833 / 1667 / 2500 ms, overshoot max 3; memo 833 / 2500 / 3334 ms, overshoot max 4. Budget about +5 per 60/min client at 2× and ≈ rate × 1–3 s for bursts.

  • A million distinct clients (mem1m1, mem1m2, 2026-11-07). keys held per isolate p50 / p99 / max 6 / 18 / 40 (mem1m2), max 81 (mem1m1): far under MAX_KEYS_HELD; 0 budget or overflow drops. Heap profiles (12 captures of dbabbca6 under mem1m2, 10 s each; the profile carries allocation samples, not a live peak): rate code is 0.10% of allocation (4.5 of 4,626 MiB); the page render dominates. The one counter is the ceiling: at ~1,180 admitted req/s from ~50 colos (mem1m2) flush p50 / p99 / p99.9 321 / 1766 / 4167 ms, 4,692 attempt timeouts and 195 batches dropped at the 5 s deadline (under-count; requests unaffected). At ~470 req/s (mem1m1) it was 212 / 948 / 1360 ms with 0 drops. So ≥ 5k concurrently blocked clients (≥ 5k admitted req/s) is beyond one RateCounter; the largest measured document was 1,488 entries / 75,937 bytes (blk2), refresh p99 36 ms. A 0% canary's Durable Object calls run the 100% version's code, so the counter changes in e6baadd (index, adaptive prune, stats) are measured after promotion (below).

  • Counter after promotion (do1, dbabbca6 at 100%, 48 generators × 20 req/s × 300 s, a new client per request, 283,913 sent). 17,167 apply logs: rows_written p50 / p99 / max 18 / 151 / 296; pruned per apply max 138 rate_hits, 129 rate_batches (the budgets). The mem runs had left 30,662 expired rate_batches rows (one id per apply at ~330 applies/s outran the old 128-row prune); ply.rate.stats showed 30,662 → 1,037 → 377 → 152 within 3.5 min while the inflow continued; expired rate_hits 0; databaseSize 7.8–10.2 MB. Flushes 64,289 ok + 66 dropped, flush p50 / p99 214 / 2051 ms at ~950 admitted req/s: the throughput ceiling is the one object's commits, not its prune.

1# Rate limit: batched writes, cached reads
3Status: live. Batched since 78ddd5c3 (2026-11-06); memo (no I/O on the request path) is the default since dbabbca6
4(2026-11-07). Replaces the per-request `ViewerRateLimit` call on the request path. Measurements land in the Results
5section with version ids and run ids.
7## Why
9`ViewerRateLimit` is a SQLite Durable Object sharded by SHA-256(`CF-Connecting-IP`), called synchronously before any git
10work. On `6f2897ff` / `c43f01cd` that call was 22–49 ms of a 45–57 ms warm `/r/ply` page and 117 ms on a fresh isolate.
11The limiter is the largest single bucket of a warm page.
13## Shape
15```
16request ─▶ default entrypoint
17 ├─ READ ctx.exports.RateCache.fetch(/blocked/v1) ◀── Workers Cache (max-age 1, SWR 30), one key per colo
18 │ └─ MISS/STALE revalidation: RateCounter.blockedDoc()
19 │ blocked? ─▶ 429 (no git work)
20 └─ WRITE allowed ─▶ isolate accumulator {key,bucket → n}
21 the request that opens a batch: ctx.waitUntil(linger 200 ms → flush)
22 flush: RateCounter.apply({id: <isolate>:<seq>, entries}) raced against a per-attempt timer,
23 retried with the SAME id until acked or 5 s from the first attempt
24RateCounter (ONE SQLite Durable Object, name "global-v1", reached as ctx.exports.RateCounter)
25 rate_batches(id) idempotency keys, kept 10 min: a replayed id is acked and changes nothing
26 rate_hits(k,b,slot,n) 10 s slots, sliding 60 s window
27 rate_blocked(k,b,until) keys at or over the limit; the read side's whole document
28```
30Nothing on the request path waits for the Durable Object except a cache MISS/EXPIRED of the one document.
32## Write side
34- **Accumulator.** Per isolate, one open batch: `Map<"<key16>:<bucket>", count>`. The first admitted request after a flush
35 opens a batch and schedules its flush 200 ms later (`LINGER_MS`) from `ctx.waitUntil`. Workers runs no timers
36 outside an invocation, so the waitUntil is what keeps the isolate alive through the linger and the retries. Every
37 request that joins the batch also `waitUntil`s the same promise (free, and the batch survives any one request
38 context ending). A batch that reaches 2048 keys (`MAX_BATCH_KEYS`) is closed and flushed at once. At most
39 `MAX_INFLIGHT` (64) batches flush at once (beyond: dropped, `rate.batch.dropped=overflow`), and at most
40 `MAX_KEYS_HELD` (16,384) distinct keys are held across the open batch and every flushing batch (since e6baadd; the
41 old bound was 64 × 2048 = 131,072 while the counter is slow). A hit for a new key past it is dropped and recorded
42 (`rate.batch.dropped=budget`); a key already in the open batch still counts. `rate.batch.keys_held` is on every
43 check. Node 24, 200k random keys with every flush stuck: 3.0 MB held at the budget vs 20.7 MB unbounded.
44- **Idempotency key.** `<isolate uuid>:<seq>`. The uuid is minted on the isolate's first request (randomness is not
45 available at global scope). The DO inserts the id into `rate_batches` in the same synchronous SQL transaction that adds
46 the counts, so an id applies at most once. Ids are kept 10 min (`DEDUPE_RETENTION_MS`), 120× the client's 5 s deadline;
47 an attempt still in flight after that is not possible (the isolate's waitUntil is gone by then).
48- **Retry (Ivar, 2026-11-06: "we can always retry the DO RPC … race a setTimeout … 5 second max").** Each attempt races
49 `apply()` against a timer of `ATTEMPT_TIMEOUT_MS`, capped by what is left of the deadline. Timeout or error → backoff
50 (50 ms, doubling) → the same id again. 5 s after the first attempt (`FLUSH_DEADLINE_MS`) the batch is dropped and
51 recorded. A timed-out attempt may still apply later; that is exactly what the id covers (tested: a DO that applies
52 after the client gave up, plus the retry, counts once).
53- **Per-attempt timeout.** 1500 ms = ≈ 2× the measured apply-RPC p99.9 (722 ms, ae21dee2 run lat2, 5,649 flushes). The
54 first cut used 1000 ms: Marseille's RPCs (436–984 ms) timed out and were replayed (counted once, but extra DO load). Lower is not better: an
55 attempt that times out and is retried while the first is merely slow adds a replay, never a double count.
56- **Drop, not merge.** An expired batch is dropped, not merged into the next batch. Merging moves its counts under a new
57 id, and if the original's last attempt applies late the counts land twice: the one property the id exists to give
58 would be gone. Dropping is a bounded under-count (below).
59- **Isolate death before flush.** Lost hits ≤ the admitted hits that isolate took in the last ≤ 200 ms plus its batches
60 still retrying (≤ 5 s). A crash or eviction during a waitUntil is rare (it outlives the response by up to 30 s); a
61 deploy does not cut a running waitUntil short. For one client the under-count is ≤ its own requests to that isolate
62 in that window: at the 60/min limit, ≤ 1 request (200 ms) and ≤ 5 requests (5 s of retries).
64## Durable Object side
66- One object (`RateCounter`, a new SQLite class, additive: `ViewerRateLimit` and its namespace stay). It is reached
67 through `ctx.exports.RateCounter`, not an env binding: the versions API refuses a binding to a class that no deploy
68 has provisioned yet (first upload, 2026-11-06). `apply` is
69 synchronous SQL (no await between the dedupe check and the writes), so it is atomic under the input gate.
70- **Window.** Sliding 60 s over 10 s slots. A slot counts while its end is within the last 60 s, so a hit counts for
71 60–70 s: never more lenient than an exact sliding window (worst case 1/6 stricter at the edge).
72- **Blocked set.** After applying, every touched key whose window sum is ≥ its limit is upserted into `rate_blocked`
73 with `until` (the time the sum next drops below the limit), `crossed` (when it first went over) and `peak` (the
74 highest admitted sum). `peak − limit` is the measured overshoot.
75- **Pruning** every ≥ 5 s inside `apply`: expired slots, ids older than 10 min, ended blocks. Per table at most 128
76 rows (`PRUNE_BATCH_ROWS`) plus the rows the apply itself added (rate_hits: + its entries; rate_blocked: + its new
77 blocks; since e6baadd). A full slice leaves a backlog and the next apply prunes again instead of waiting 5 s, so a
78 backlog (an idle spell, a restarted object) drains in bounded slices, and it shrinks under any inflow: each apply
79 takes out at least what it put in (unit test: 61,440 expired rows drained by full 2,048-key applies while the
80 inflow continued). One apply writes at most its own rows + its own entries + 128 per table. Before the bound one
81 apply on 78ddd5c3 wrote 12,892 rows. `ply.rate.apply` logs `pruned_hits/batches/blocked` and `prune_backlog`;
82 `ply.rate.stats` (every 30 s) logs the expired rows still stored per table and `databaseSize`.
83- **Index.** `rate_blocked(until)` (e6baadd) serves the document query (`WHERE until > ? ORDER BY until DESC LIMIT
84 4096`, every revalidation) and the blocked prune; before it the plan was `SCAN rate_blocked | USE TEMP B-TREE FOR
85 ORDER BY` (node:sqlite EXPLAIN QUERY PLAN; asserted in test/rate.test.ts).
86- **Budget.** Flush RPCs/s ≤ isolates × 5. Measured in Results (apply ms, rows written, DO CPU per invocation).
88## Read side
90- **One document, one key.** `RateCache` is a cache-enabled named entrypoint (the GitCache / Pinned / FlagsCache pattern;
91 never `caches.default`). `GET https://rate.internal/blocked/v1` answers `RateCounter.blockedDoc()`: every blocked
92 `[key16, bucket, until, crossed]`, newest 4096 (`MAX_DOC_ENTRIES`), with
93 `Cache-Control: public, max-age=1, stale-while-revalidate=30`. A full document is 208,947 bytes; parse + index p50 /
94 p99 1.05 / 2.21 ms (Node 24, 300 runs), ~228 KB of index.
95- **Reader bound (e6baadd).** The isolate reads the body streamed and stops past 512 KiB (`MAX_DOC_BYTES`: the body is
96 cancelled, the read is `TOO_LARGE`, the last document stays in use) and indexes at most 4096 entries
97 (`rate.doc_truncated`). `rate.doc_bytes` is on refresh spans. A bad or oversized document cannot grow the isolate.
98- **Why not per-client keys.** A per-client key misses on every new client and on every client's first request after each
99 TTL: the median viewer client makes a handful of requests, so the hit ratio would sit near the miss path and every
100 miss is a DO round trip on the request path, which is the cost being removed. One document is one MISS per colo per
101 second, whatever the client count. The cost is size: the document is O(blocked clients), capped at 4096 entries ×
102 ~40 bytes. Measured hit ratio in Results.
103- **In the isolate** the parsed document is memoized by its `etag`: a HIT costs the loopback fetch, not a JSON parse.
104- **Read budget (never block on the DO).** The cached fetch gets 25 ms (`READ_BUDGET_MS`; warm HIT/SWR reads on lat2:
105 p99 12 ms, max 46 ms). A MISS is a synchronous DO read (403–481 ms on lat2) and an isolate's first loopback ~55 ms: past
106 the budget the request decides from the isolate's last document (any age ≤ 5 min), or admits when the isolate has
107 none, and the fetch completes in `waitUntil`, refreshing that document. Extra lag this adds: only in an isolate that
108 has never read the document, for ≤ one DO read (~0.5 s); measured in Results.
109- **Memo mode (`RATE_MODE=memo`, the default since 2026-11-07, dbabbca6; or `x-ply-rate-mode: memo`).** The batched read's p99 (12 ms on a020b1b1 / 78ddd5c3)
110 is the loopback fetch itself: of 401 checks ≥ 10 ms, 363 were cache HIT or UPDATING. In memo mode the request never
111 fetches: it looks its key up in the isolate's parsed document. When that document is ≥ 500 ms old
112 (`MEMO_REFRESH_MS`) the request still decides from it and starts ONE refresh through the same `RateCache` entrypoint
113 in `ctx.waitUntil` (single flight: a request that finds one in flight joins it; a flight unsettled after 10 s,
114 `MEMO_REFRESH_ABANDON_MS`, is abandoned). An isolate with no document (or one older than 5 min) admits
115 (`fail_open`, `rate.read=cold`) and refreshes. Extra lag over batched: ≤ 500 ms + the refresh's read + the
116 isolate's own gap between requests (the memo only refreshes when a request arrives).
117- **Failure.** If the read fails, the last good document (≤ 5 min old) is used; with none, the request is admitted (fail
118 open) and recorded. Today's limiter fails closed (503); with one object instead of one per client, failing closed would
119 turn a DO incident into a viewer outage. The limit bounds cost; it is not an access control.
121## Semantics
123Approximate. Worst-case enforcement lag after the request that takes a client to its limit:
125```
126lag ≤ LINGER (200 ms) + apply RPC (+ retries, ≤ 5 s worst) + max-age (1 s) + one revalidation + the client's next-request gap
127```
129Steady state (RPC ≈ tens of ms, traffic flowing) that is ≈ 1.5 s. A client sending 2× the limit (2 req/s at 60/min)
130overshoots by about lag × rate ≈ 3 requests per window. The DO measures the true overshoot (`peak − limit`); the load
131generator measures lag end to end (first 429 after the request that should have been the first refusal).
133Identical to today: the limit is on admitted requests (a 429 does not count), `page` 60/min and `pack` 20/min per client,
134`retry-after` from the block's `until`.
136## Privacy
138As today: the client key is SHA-256 of `ply-viewer-rate/1\n<CF-Connecting-IP>`; only its first 16 hex (64 bits) is sent,
139stored and cached. No address is stored. The blocked document is served only over `ctx.exports`, never to a browser.
141## Load-test controls
143A request carrying `x-ply-load-token: <PLY_LOAD_TOKEN>` (compared in constant time; never forwarded, logged or echoed;
144Workers Logs records the header as `REDACTED`) may also send `x-ply-rate-client` (a label hashed in place of the
145address, so one generator can be many clients), `x-ply-rate-mode: legacy|batched|memo` (choose the limiter for this
146request: an A/B inside one version), and `x-ply-rate-fault` (fault injection on the DO, below). Without a valid token
147those headers are ignored.
149`PLY_LOAD_TOKEN` is a credential of its own (2026-11-07): it unlocks these headers and nothing else. `PLY_TOKEN` (the
150/v1 bearer) does not unlock them, and the load generator (`ply-load`) holds only `PLY_LOAD_TOKEN` (for its control API and
151the header). With no `PLY_LOAD_TOKEN` bound, no request is a load request. Local copy: `~/.config/ply/load-token` (0600).
153`x-ply-rate-fault: slow-ack=<ms>;attempts=<n>` makes the DO apply, then hold its answer `<ms>` on attempts ≤ n
154(the client times out and retries; the retry must be a replay). `error;attempts=<n>` makes the DO throw before
155applying on attempts ≤ n. A faulted request flushes in a batch of its own, at once, never the isolate's shared batch:
156a fault slows, errors or drops only that request's own count, never a real client's.
158## Instrumentation
160Workers spans (`ctx.tracing`), each stamped with `ply.route` and `ply.script_version` like ply's spans:
162| span | attributes |
163|---|---|
164| `ply.rate.check` | `rate.mode` legacy/batched, `rate.mode_source` default/header (header: a load request chose the arm), `rate.cold` isolate_start/memo_expired (an expected cold check: see "CD gate"), `rate.bucket`, `rate.decision` allow/deny/fail_open, `rate.fail_open_reason` (on every fail_open: isolate_start / memo_expired / read_failed / read_late), `rate.check_ms`, `rate.cache` (HIT/STALE/MISS/EXPIRED/…), `rate.read` ok/memo/fail, `rate.doc_entries`, `rate.doc_age_ms`, `rate.blocked_for_ms` (deny: now − crossed) |
165| `ply.rate.check` (memo) | `rate.read` memo_fresh / memo_stale_refresh / cold, `rate.cache` MEMO, `rate.memo_age_ms` (now − the isolate's last read), `rate.doc_age_ms` (now − the DO's `at`), `rate.refresh.started`; cold: `rate.memo_expired_ms` (the expired document's age, when the isolate held one), `rate.cold_last_read` (read_failed: how the last refresh ended, fail/memo/abandoned/threw) |
166| `ply.rate.refresh` | `rate.refresh.trigger` cold/stale, `.cold` (trigger cold: the starting check's cause), `.memo_expired_ms`, `.read` ok/memo/fail, `.cache` (the RateCache read's cf-cache-status), `.parsed`, `.ms`, `.joins` (requests that found it in flight), `rate.doc_age_ms` |
167| `ply.rate.flush` | `rate.batch.id`, `.keys`, `.hits`, `.linger_ms`, `.attempts`, `.attempt_log` (`ok:34,timeout:1000,…`), `.timeouts`, `.errors`, `.outcome` ok/dropped, `.deadline_hit`, `.replay` (DO said duplicate), `.flush_ms`, `.alive_ms` (batch open → flush end, the waitUntil lifetime), `rate.do.apply_ms`, `rate.do.rows_written`, `rate.do.new_blocks` |
169DO log lines: `ply.rate.apply` (id, attempt, replay, keys, hits, apply_ms, rows_written, new_blocks, fault) and
170`ply.rate.over` (a key at or over its limit: sum, limit, crossed, peak). Queried by `$workers.scriptVersion.id`.
172## CD gate (silent degradation)
174The limiter fails soft: a broken read admits (`fail_open`), a flush that cannot reach the counter is dropped after 5 s,
175and a missing counter sends every request to `legacy` (59926117 / e61a5742: nothing alerted). All of those answer 200,
176so the smoke and the error gates pass. `scripts/cd/rate-gate.ts` runs inside the telemetry stage of the 0% canary
177(fail → stop) and of the live watch (fail → rollback): 30 pinned probes of `/view?remote=<public ply>` (the redirect
178route: rate-checked, no git work) as a load client with no mode header, then the version's own spans since the stage
179began, by aggregate count. It fails the version when
181- any default-mode check (`rate.mode_source` default) ran a limiter other than `rate.expectMode` (memo since
182 2026-11-07; test/cd-rate-gate.test.ts asserts it equals `RATE_MODE` in cloudflare.config.ts);
183- warm default-mode checks (no `rate.cold`, plus memo_expired checks whose own refresh failed) failed open above 10%
184 (with ≥ 20 of them; fewer is a harness fault);
185- any flush was dropped (deadline) or any batch overflowed without an injected fault.
187The thresholds and their evidence (7 days of spans, 128,753 flushes, every version since the limiter shipped) are in
188`scripts/cd/cd.policy.json` `rate`. A deliberate `RATE_MODE` rollback (batched or legacy) changes `rate.expectMode`
189too, in the same commit, shipped through the pipeline: a `cf` var or secret edit creates and deploys a version at 100%
190directly (d4e3eea6 did).
192Expected cold checks (`rate.cold`, counted on their own line and kept out of the warm share; ply-web#156):
194- `isolate_start`: an isolate's first check, and any memo check that is cold while that isolate's first refresh is
195 still in flight.
196- `memo_expired`: the isolate's last refresh read the document ok, and the document then aged past MEMO_FALLBACK_MS
197 (5 min) because no request came to refresh it (an idle isolate). The check admits and starts a refresh stamped
198 `rate.refresh.cold = memo_expired`; if that refresh does not read ok, the check is put back as a warm failure. CD
199 makes this likely: the canary isolate takes the 30 probes in ~11 s and then idles through the ramp, so the live
200 watch's first request on it is past 5 min (2026-11-12T13-06-51-511Z: 339.4 s idle, refresh ok in 6 ms, counted 1/29
201 warm before this).
203A cold read after a refresh has settled with no document (`rate.fail_open_reason = read_failed`: the read path is
204broken) is a warm `fail_open` and counts. The gate prints both, e.g. `fail_open warm 0/28 = 0.00%; expected fail_open
205isolate_start 4/4, memo_expired 1/1 (refresh ok 1, failed 0)`.
207## Results
209Load: ply-load (`bench/load`), generator DOs in wnam/enam/weur/apac (4 shards each, 5 req/s each, open loop),
210`GET /r/ply`, the two arms interleaved per request inside ONE version via `x-ply-rate-mode`, pinned with
211`Cloudflare-Workers-Version-Overrides`. Server numbers: `ply.rate.check` spans (`rate.check_ms`) and `ply.request` logs
212joined by trace, filtered by `ply.script_version`. Synthetic clients each stay under the limit.
214| run · version | arm | n checks | check p50 / p99 / p99.9 / max ms | page wall p50 / p99 / p99.9 ms |
215|---|---|---|---|---|
216| lat3 · 6ed2e746 (no read budget) | legacy | 13,187 | 15 / 404 / 533 / 1375 | 27 / 462 / 1567 |
217| lat3 · 6ed2e746 (no read budget) | batched | 13,188 | 3 / 13 / 49 / 373 | 16 / 107 / 1446 |
218| lat4 · a020b1b1 (25 ms read budget) | legacy | 13,177 | 14 / 403 / 588 / 3274 | 28 / 463 / 3447 |
219| lat4 · a020b1b1 (25 ms read budget) | batched | 15,099 | **3 / 12 / 25 / 25** | 16 / 71 / 3049 |
220| lat5 · 78ddd5c3 (promoted, unpinned) | legacy | 10,778 | 13 / 424 / 584 / 819 | 26 / 531 / 2739 |
221| lat5 · 78ddd5c3 (promoted, unpinned) | default (batched) | 10,782 | **3 / 12 / 25 / 25** | 15 / 193 / 2747 |
222| lat6 · 2a4741c0 (pinned, interleaved) | batched | 13,571 | 3 / 14 / 25 / 25 | 12 / 165 / 4690 |
223| lat6 · 2a4741c0 (pinned, interleaved) | memo | 13,572 | **0 / 0 / 0 / 0** | 9 / 153 / 4702 |
224| lat7 · 2a4741c0 (pinned, memo only) | memo | 13,583 | 0 / 0 / 0 / 0 | 10 / 69 / 1302 |
226The page p99.9 (≈3 s in both arms) is git work, not the limiter (lat1, A/A legacy, has the same tail).
228- **Read cache (lat4):** HIT 77.9%, UPDATING (served stale while one request revalidates) 21.9%, past the 25 ms budget
229 0.19% (29); 13 of those had no document in the isolate and were admitted (`fail_open`). HIT+SWR 99.8%.
230- **Enforcement (abuse runs, 12 clients/arm at 2× the 60/min limit for 100 s):** batched lag p50 511 / max 1007 ms
231 (abuse2, a020b1b1; abuse1 6ed2e746: p50 1007, max 1012), overshoot p50 1 / max 2 requests per client (the DO's
232 `peak − limit` agrees: 2). Legacy: lag 19–44 ms, overshoot 0. Admitted totals over the run are equal (1,440 vs 1,440).
233- **Flushes (lat4, a020b1b1, no fault):** 12,525 flushes, 99.9% one attempt; 11 timeouts in 12,536 attempts at 1.5 s
234 (lat3 at 1.0 s: 175 in 11,655, every one later a DO replay). RPC p50 / p99 / p99.9 142 / 762 / 1363 ms. Keys per
235 batch p50 1, p99 3: at ~1 req/s per colo an isolate rarely sees two hits in 200 ms, so the DO is written about once per
236 admitted request; the batching pays off only with denser per-isolate traffic.
237- **Counter DO (lat3 window):** 11,228 invocations (≈47/s); CPU p50 0 / p99 2 / p99.9 7 ms; wall p50 133 / p99 1317 ms.
238 The wall is the SQLite commit behind the output gate, and it is the RPC latency the flush sees.
239- **Fault injection in prod (a020b1b1, fault1 SJC / fault2 LHR+MRS, 20–60 s of each run):**
240 `slow-ack=2500` → 305/305 batches: attempt 1 timed out at 1.5 s, attempt 2 acked as a replay. `error;attempts=2` → 280
241 batches ok on attempt 3, 3 on attempt 4, 1 hit the 5 s deadline (recorded `dropped`), yet the DO log shows its attempt
242 3 applied and attempt 4 replayed: counted once. Across 13,978 applies, no id applied twice.
243- **After promotion (lat5, 78ddd5c3 at 100%, unpinned, default vs `x-ply-rate-mode: legacy`, 10,800 requests/arm):**
244 all 200, 0 wrong-version. Client TTFB p50 / p99 / p99.9 45 / 935 / 4505 ms default vs 59 / 1035 / 4717 legacy. Read
245 cache HIT 74.7%, UPDATING 25.1%, LATE 0.25% (27; 9 `fail_open`). 10,563 flushes, all ok on attempt 1 (0 timeouts,
246 0 replays, 0 deadline); RPC p50 / p99 / p99.9 150 / 290 / 453 ms, a much tighter tail than lat4's 762 / 1363. Real
247 (non-load) traffic on batched in the window: 58 checks, p99 5 ms. DO `rows_written` p50 6, p99.9 519, max 3983: the
248 large ones are the retention prunes (`rate_hits`, `rate_batches` older than 10 min), not hit writes.
249- **Memo mode (2a4741c0, 2026-11-07; ply-load ad3db19e).** lat6: batched vs memo interleaved per request, pinned,
250 27,200 requests (13,600/arm), all 200. The memo check is a map lookup: `rate.check_ms` 0 at every percentile (Workers
251 clocks only advance on I/O, so the page wall from `ply.request` joined by trace is the comparison). Page wall memo −
252 batched, bootstrap 95% CI (400 resamples): p50 −3 ms [−3, −3], p90 −3 [−4, −3], p99 −12 [−39, +12]. So a 3 ms win at
253 the median and no measurable change at p99, which is git work. Batched reads: ok 13,542, LATE 29 (7 `fail_open`).
254 The interleaved memo arm shares the isolate's document with the batched arm (it is module state), which keeps it
255 fresh: memo_age p50 200 ms, 26 refreshes. So refresh and lag were measured memo-only. lat7, memo only, 13,583
256 checks: memo_fresh 9,046, memo_stale_refresh 4,495, cold 42 (`fail_open`; batched lat6: 7). memo_age p50/p99/p99.9
257 400 / 602 / 653 ms. 4,517 refreshes (one per 3 requests), refresh ms p50/p99/p99.9/max 4 / 8 / 50 / 301, RateCache
258 HIT 2,944, UPDATING 1,569, EXPIRED 4. 20 single-flight joins (max 5 on one flight).
259- **Enforcement, memo vs batched (2a4741c0, 12 clients at 2× the limit, 100 s, client-side lag quantised by the 500 ms
260 request gap):** abuse3 interleaved (memo kept fresh by batched): batched lag p50 500 / max 1500 ms, overshoot p50 1 /
261 max 3; memo the same. abuse5 batched only: lag p50 500 / max 1500 ms, overshoot max 3, DO `peak − limit` 3. abuse4
262 memo only: **lag p50 1500 / max 2000 ms, overshoot p50 3 / max 4**, DO `peak − limit` 4. Memo adds about one request
263 gap plus ≤ 500 ms of memo age on top of the cache's max-age and SWR.
264- **Decision (superseded 2026-11-07, below):** memo stays opt-in (`x-ply-rate-mode: memo` / `RATE_MODE=memo`) and the default stays batched. Its page
265 p99 gain is not measurable and its lag max (2.0 s) is over the ~1.5 s bar. 2a4741c0 was promoted with the default
266 unchanged, which ships the bounded prune.
267- **Counter prune bound (PRUNE_BATCH_ROWS 128).** `ply.rate.apply` rows_written max before: 12,892 (78ddd5c3), 7,915
268 (a020b1b1). After: lat8, 2a4741c0 at 100% unpinned, default arm, 13,600 requests all 200, starting on the backlog
269 that lat6/lat7 left: 6,550 applies, rows_written median 9 / p99 140 / max 268.
270- **Bursts (2026-11-07, canary 9d33f400 pinned, ply-load de344fdb, runs burst2-c{1,5}-r{50,100,200}).** One client
271 (shared label, 4 regions × 2 generators) at 50 / 100 / 200 req/s for 60 s: admitted in a 60 s window 116 / 125 / 140
272 (overshoot 56 / 65 / 80; the DO's `peak` equals the client's count every time). Five clients at 10 / 20 / 40 req/s
273 each: overshoot 6-10 / 14-20 / 28-32. Overshoot ≈ client rate × 0.4-1.1 s. DO crossing → first deny per colo p50
274 0.5-1.8 s, max 4.5 s. The counter stayed healthy: every flush one attempt, 0 timeouts / replays / drops, apply RPC
275 p99 ≤ 294 ms, CPU p99 ≤ 4 ms, rows_written p99 ≤ 51; denied requests never flush. The most a client's requests landed
276 in one colo before it was refused was 59, so a per-isolate pre-count at the limit would not have bitten here.
277- **Memo becomes the default (2026-11-07; Ivar: "Overshoot is not horrible, we budget that into the limit. I care
278 most about hot path perf.").** Commits 53a8e47 (RATE_MODE memo, gate expects memo) + e6baadd (memory bounds). Canary
279 dbabbca6-b606-4c97-856b-22423a4a65da at 0% beside 935e8eb1-bdb5-4572-b7de-058d4c51492e (batched; it already ran
280 REFS_POLICY=watched, so the arms differ only in the limiter). ply-load de344fdb, both versions pinned, interleaved
281 per request, GET /r/ply, load checks joined to `ply.request` by trace:
283 | run · arm | checks | check p50 / p99 / p99.9 / max ms | pages | wall p50 / p90 / p99 / p99.9 ms |
284 |---|---|---|---|---|
285 | memoab1 · 935e8eb1 batched | 14,138 | 3 / 8 / 25 / 25 | 13,874 | 11 / 15 / 34 / 755 |
286 | memoab1 · dbabbca6 memo | 14,142 | **0 / 0 / 0 / 0** | 13,976 | **9 / 14 / 30 / 562** |
287 | mem1m2 · 935e8eb1 batched | 531,656 | 3 / 19 / 25 / 25 | 531,764 | 14 / – / 154 / 295 |
288 | mem1m2 · dbabbca6 memo | 531,728 | **0 / 0 / 0 / 0** | 531,598 | **11 / – / 139 / 256** |
290 memoab1: 6 regions × 4 generators × 4 req/s × 300 s, 16 colos. Memo − batched, bootstrap 95%: check −3 [−3, −3] /
291 −4 [−4, −4] (p90) / −8 [−8, −8] / −25 [−25, −25]; wall p50 −2 [−2, −2], p90 −1 [−2, −1], p99 −4 [−8, −1], p99.9
292 −193 [−720, +409] (the tail is git work). mem1m2 (48 generators × 25 req/s × 900 s, a new client per request,
293 1,063,558 distinct): every percentile lower on memo (Workers Observability calculations). Memo reads (memoab1):
294 memo_fresh 3,200, memo_stale_refresh 10,918, cold 24 (all start-up, then stamped `rate.cold_isolate`, now `rate.cold = isolate_start`); warm fail_open 0 vs 1 on
295 batched. Refreshes 10,977 (one per 1.3 requests), all ok, HIT 80% / UPDATING 20%, ms p50 / p99 / p99.9 / max
296 3 / 8 / 40 / 173; memo age p50 / p99 500 / 1001 ms. Promoted with `npm run cd` (canary + live rate gate: memo on every
297 default check, warm fail_open 0/26 and 0/37, 0 drops; visual smoke 32/32). Previous: 935e8eb1.
298- **Enforcement with memo (the overshoot budget).** abuse6 (dbabbca6 pinned, 12 clients at 2 req/s = 2× the limit,
299 4 regions, 100 s): lag p50 / p90 / max 1000 / 2000 / 2500 ms (client clock, quantised by the 500 ms gap), overshoot
300 p50 / p90 / max 2 / 4 / 5 requests per 60 s window; DO `peak` max 65. blk2 (1,536 clients at 1.2× the limit, each on
301 one version, under counter load): batched lag p50 / p99 / max 833 / 1667 / 2500 ms, overshoot max 3; memo 833 /
302 2500 / 3334 ms, overshoot max 4. Budget about +5 per 60/min client at 2× and ≈ rate × 1–3 s for bursts.
303- **A million distinct clients (mem1m1, mem1m2, 2026-11-07).** keys held per isolate p50 / p99 / max 6 / 18 / 40
304 (mem1m2), max 81 (mem1m1): far under MAX_KEYS_HELD; 0 budget or overflow drops. Heap profiles (12 captures of
305 dbabbca6 under mem1m2, 10 s each; the profile carries allocation samples, not a live peak): rate code is 0.10% of
306 allocation (4.5 of 4,626 MiB); the page render dominates. **The one counter is the ceiling:** at ~1,180 admitted
307 req/s from ~50 colos (mem1m2) flush p50 / p99 / p99.9 321 / 1766 / 4167 ms, 4,692 attempt timeouts and 195 batches
308 dropped at the 5 s deadline (under-count; requests unaffected). At ~470 req/s (mem1m1) it was 212 / 948 / 1360 ms
309 with 0 drops. So ≥ 5k concurrently blocked clients (≥ 5k admitted req/s) is beyond one RateCounter; the largest
310 measured document was 1,488 entries / 75,937 bytes (blk2), refresh p99 36 ms. A 0% canary's Durable Object calls
311 run the 100% version's code, so the counter changes in e6baadd (index, adaptive prune, stats) are measured after
312 promotion (below).
314- **Counter after promotion (do1, dbabbca6 at 100%, 48 generators × 20 req/s × 300 s, a new client per request,
315 283,913 sent).** 17,167 apply logs: rows_written p50 / p99 / max 18 / 151 / 296; pruned per apply max 138 rate_hits,
316 129 rate_batches (the budgets). The mem runs had left 30,662 expired `rate_batches` rows (one id per apply at ~330
317 applies/s outran the old 128-row prune); `ply.rate.stats` showed 30,662 → 1,037 → 377 → 152 within 3.5 min while
318 the inflow continued; expired rate_hits 0; databaseSize 7.8–10.2 MB. Flushes 64,289 ok + 66 dropped, flush p50 / p99
319 214 / 2051 ms at ~950 admitted req/s: the throughput ceiling is the one object's commits, not its prune.
Diagnostics4 git calls · 182 ms
CallStatusSizeTime0182 ms
discover200152 B3 ms
git.upload-pack discover
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=ls-refs
git.upstream.dur_ms=3
git.response.bytes=152
git.server.agent=gitty/1.0
git.server.object_format=sha1
git.server.caps=agent ls-refs fetch object-format
git.server.fetch=shallow filter sideband-all
git.protocol.version=2
git.pkt.frames=8
git.pkt.data=6
git.pkt.flush=2
git.pkt.delim=0
git.pkt.bytes=152
git.pkt.payload.max=34
ls-refs200145 B · hit3 ms
git.upload-pack ls-refs
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=ls-refs
git.upstream.dur_ms=3
git.response.bytes=145
git.response.cache=HIT
git.response.duration_ms=3
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=145
git.pkt.payload.max=76
git.ls_refs.count=2
git.ls_refs.head=refs/heads/main
fetchblob:none · deepen 1020040 KiB · hit3 ms
git.upload-pack fetch
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=3
git.response.bytes=41349
git.response.cache=HIT
git.response.duration_ms=3
git.fetch.filter=blob:none
git.fetch.filter.requested=tree:0
git.fetch.filter.fallback=blob:none
git.fetch.filter.learned=false
git.fetch.wants=526b305930b8ef382f2d7a1f9c0238730a585921
git.fetch.haves.count=0
git.fetch.deepen=10
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=6
git.pkt.data=4
git.pkt.flush=1
git.pkt.delim=1
git.pkt.bytes=41349
git.pkt.payload.max=41254
git.sideband.band1.bytes=41253
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0
git.pack.objects=61
git.pack.bytes=41253
git.pack.ofs_delta=0
git.pack.ref_delta=0
git.pack.chain.max=0
git.pack.trailer=ab60efef4b7243678c75de64a4386986ee885c1c
git.pack.full=61
Open pack map
fetch20012 KiB176 ms
git.upload-pack fetch
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=175
git.response.bytes=12381
git.response.cache=MISS
git.response.duration_ms=176
git.fetch.wants=b8d8dc591d0152eb9a475cb4c4be2f724571deac
git.fetch.haves.count=0
git.fetch.deepen=0
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=12381
git.pkt.payload.max=12360
git.sideband.band1.bytes=12359
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0

Each row is one HTTP round trip ply made to the git server for this page, on one timeline. discover asks what the server speaks, ls-refs lists branch tips, fetch brings objects by id. A hit was served by Workers Cache. Tap a row for its wire details.

Remote
https://5ad2236b39de4c8bd8f6270ea63f3ea9.artifacts.cloudflare.net/git/public-v0/ply-web.git
Ref
refs/heads/main
Commit
526b305930b8ef382f2d7a1f9c0238730a585921
cf-ray
a5e3012ea8ed7929
Request
af8d8383…
Protocol
v2 · sha1
Agent
gitty/1.0
Filter
advertised
Fetch
shallow filter sideband-all
Capabilities
agent ls-refs fetch object-format