# Rate limit: batched writes, cached reads Status: live. Batched since 78ddd5c3 (2026-11-06); memo (no I/O on the request path) is the default since dbabbca6 (2026-11-07). Replaces the per-request `ViewerRateLimit` call on the request path. Measurements land in the Results section with version ids and run ids. ## Why `ViewerRateLimit` is a SQLite Durable Object sharded by SHA-256(`CF-Connecting-IP`), called synchronously before any git work. On `6f2897ff` / `c43f01cd` that call was 22–49 ms of a 45–57 ms warm `/r/ply` page and 117 ms on a fresh isolate. The limiter is the largest single bucket of a warm page. ## Shape ``` request ─▶ default entrypoint ├─ READ ctx.exports.RateCache.fetch(/blocked/v1) ◀── Workers Cache (max-age 1, SWR 30), one key per colo │ └─ MISS/STALE revalidation: RateCounter.blockedDoc() │ blocked? ─▶ 429 (no git work) └─ WRITE allowed ─▶ isolate accumulator {key,bucket → n} the request that opens a batch: ctx.waitUntil(linger 200 ms → flush) flush: RateCounter.apply({id: :, entries}) raced against a per-attempt timer, retried with the SAME id until acked or 5 s from the first attempt RateCounter (ONE SQLite Durable Object, name "global-v1", reached as ctx.exports.RateCounter) rate_batches(id) idempotency keys, kept 10 min: a replayed id is acked and changes nothing rate_hits(k,b,slot,n) 10 s slots, sliding 60 s window rate_blocked(k,b,until) keys at or over the limit; the read side's whole document ``` Nothing on the request path waits for the Durable Object except a cache MISS/EXPIRED of the one document. ## Write side - **Accumulator.** Per isolate, one open batch: `Map<":", count>`. The first admitted request after a flush opens a batch and schedules its flush 200 ms later (`LINGER_MS`) from `ctx.waitUntil`. Workers runs no timers outside an invocation, so the waitUntil is what keeps the isolate alive through the linger and the retries. Every request that joins the batch also `waitUntil`s the same promise (free, and the batch survives any one request context ending). A batch that reaches 2048 keys (`MAX_BATCH_KEYS`) is closed and flushed at once. At most `MAX_INFLIGHT` (64) batches flush at once (beyond: dropped, `rate.batch.dropped=overflow`), and at most `MAX_KEYS_HELD` (16,384) distinct keys are held across the open batch and every flushing batch (since e6baadd; the old bound was 64 × 2048 = 131,072 while the counter is slow). A hit for a new key past it is dropped and recorded (`rate.batch.dropped=budget`); a key already in the open batch still counts. `rate.batch.keys_held` is on every check. Node 24, 200k random keys with every flush stuck: 3.0 MB held at the budget vs 20.7 MB unbounded. - **Idempotency key.** `:`. The uuid is minted on the isolate's first request (randomness is not available at global scope). The DO inserts the id into `rate_batches` in the same synchronous SQL transaction that adds the counts, so an id applies at most once. Ids are kept 10 min (`DEDUPE_RETENTION_MS`), 120× the client's 5 s deadline; an attempt still in flight after that is not possible (the isolate's waitUntil is gone by then). - **Retry (Ivar, 2026-11-06: "we can always retry the DO RPC … race a setTimeout … 5 second max").** Each attempt races `apply()` against a timer of `ATTEMPT_TIMEOUT_MS`, capped by what is left of the deadline. Timeout or error → backoff (50 ms, doubling) → the same id again. 5 s after the first attempt (`FLUSH_DEADLINE_MS`) the batch is dropped and recorded. A timed-out attempt may still apply later; that is exactly what the id covers (tested: a DO that applies after the client gave up, plus the retry, counts once). - **Per-attempt timeout.** 1500 ms = ≈ 2× the measured apply-RPC p99.9 (722 ms, ae21dee2 run lat2, 5,649 flushes). The first cut used 1000 ms: Marseille's RPCs (436–984 ms) timed out and were replayed (counted once, but extra DO load). Lower is not better: an attempt that times out and is retried while the first is merely slow adds a replay, never a double count. - **Drop, not merge.** An expired batch is dropped, not merged into the next batch. Merging moves its counts under a new id, and if the original's last attempt applies late the counts land twice: the one property the id exists to give would be gone. Dropping is a bounded under-count (below). - **Isolate death before flush.** Lost hits ≤ the admitted hits that isolate took in the last ≤ 200 ms plus its batches still retrying (≤ 5 s). A crash or eviction during a waitUntil is rare (it outlives the response by up to 30 s); a deploy does not cut a running waitUntil short. For one client the under-count is ≤ its own requests to that isolate in that window: at the 60/min limit, ≤ 1 request (200 ms) and ≤ 5 requests (5 s of retries). ## Durable Object side - One object (`RateCounter`, a new SQLite class, additive: `ViewerRateLimit` and its namespace stay). It is reached through `ctx.exports.RateCounter`, not an env binding: the versions API refuses a binding to a class that no deploy has provisioned yet (first upload, 2026-11-06). `apply` is synchronous SQL (no await between the dedupe check and the writes), so it is atomic under the input gate. - **Window.** Sliding 60 s over 10 s slots. A slot counts while its end is within the last 60 s, so a hit counts for 60–70 s: never more lenient than an exact sliding window (worst case 1/6 stricter at the edge). - **Blocked set.** After applying, every touched key whose window sum is ≥ its limit is upserted into `rate_blocked` with `until` (the time the sum next drops below the limit), `crossed` (when it first went over) and `peak` (the highest admitted sum). `peak − limit` is the measured overshoot. - **Pruning** every ≥ 5 s inside `apply`: expired slots, ids older than 10 min, ended blocks. Per table at most 128 rows (`PRUNE_BATCH_ROWS`) plus the rows the apply itself added (rate_hits: + its entries; rate_blocked: + its new blocks; since e6baadd). A full slice leaves a backlog and the next apply prunes again instead of waiting 5 s, so a backlog (an idle spell, a restarted object) drains in bounded slices, and it shrinks under any inflow: each apply takes out at least what it put in (unit test: 61,440 expired rows drained by full 2,048-key applies while the inflow continued). One apply writes at most its own rows + its own entries + 128 per table. Before the bound one apply on 78ddd5c3 wrote 12,892 rows. `ply.rate.apply` logs `pruned_hits/batches/blocked` and `prune_backlog`; `ply.rate.stats` (every 30 s) logs the expired rows still stored per table and `databaseSize`. - **Index.** `rate_blocked(until)` (e6baadd) serves the document query (`WHERE until > ? ORDER BY until DESC LIMIT 4096`, every revalidation) and the blocked prune; before it the plan was `SCAN rate_blocked | USE TEMP B-TREE FOR ORDER BY` (node:sqlite EXPLAIN QUERY PLAN; asserted in test/rate.test.ts). - **Budget.** Flush RPCs/s ≤ isolates × 5. Measured in Results (apply ms, rows written, DO CPU per invocation). ## Read side - **One document, one key.** `RateCache` is a cache-enabled named entrypoint (the GitCache / Pinned / FlagsCache pattern; never `caches.default`). `GET https://rate.internal/blocked/v1` answers `RateCounter.blockedDoc()`: every blocked `[key16, bucket, until, crossed]`, newest 4096 (`MAX_DOC_ENTRIES`), with `Cache-Control: public, max-age=1, stale-while-revalidate=30`. A full document is 208,947 bytes; parse + index p50 / p99 1.05 / 2.21 ms (Node 24, 300 runs), ~228 KB of index. - **Reader bound (e6baadd).** The isolate reads the body streamed and stops past 512 KiB (`MAX_DOC_BYTES`: the body is cancelled, the read is `TOO_LARGE`, the last document stays in use) and indexes at most 4096 entries (`rate.doc_truncated`). `rate.doc_bytes` is on refresh spans. A bad or oversized document cannot grow the isolate. - **Why not per-client keys.** A per-client key misses on every new client and on every client's first request after each TTL: the median viewer client makes a handful of requests, so the hit ratio would sit near the miss path and every miss is a DO round trip on the request path, which is the cost being removed. One document is one MISS per colo per second, whatever the client count. The cost is size: the document is O(blocked clients), capped at 4096 entries × ~40 bytes. Measured hit ratio in Results. - **In the isolate** the parsed document is memoized by its `etag`: a HIT costs the loopback fetch, not a JSON parse. - **Read budget (never block on the DO).** The cached fetch gets 25 ms (`READ_BUDGET_MS`; warm HIT/SWR reads on lat2: p99 12 ms, max 46 ms). A MISS is a synchronous DO read (403–481 ms on lat2) and an isolate's first loopback ~55 ms: past the budget the request decides from the isolate's last document (any age ≤ 5 min), or admits when the isolate has none, and the fetch completes in `waitUntil`, refreshing that document. Extra lag this adds: only in an isolate that has never read the document, for ≤ one DO read (~0.5 s); measured in Results. - **Memo mode (`RATE_MODE=memo`, the default since 2026-11-07, dbabbca6; or `x-ply-rate-mode: memo`).** The batched read's p99 (12 ms on a020b1b1 / 78ddd5c3) is the loopback fetch itself: of 401 checks ≥ 10 ms, 363 were cache HIT or UPDATING. In memo mode the request never fetches: it looks its key up in the isolate's parsed document. When that document is ≥ 500 ms old (`MEMO_REFRESH_MS`) the request still decides from it and starts ONE refresh through the same `RateCache` entrypoint in `ctx.waitUntil` (single flight: a request that finds one in flight joins it; a flight unsettled after 10 s, `MEMO_REFRESH_ABANDON_MS`, is abandoned). An isolate with no document (or one older than 5 min) admits (`fail_open`, `rate.read=cold`) and refreshes. Extra lag over batched: ≤ 500 ms + the refresh's read + the isolate's own gap between requests (the memo only refreshes when a request arrives). - **Failure.** If the read fails, the last good document (≤ 5 min old) is used; with none, the request is admitted (fail open) and recorded. Today's limiter fails closed (503); with one object instead of one per client, failing closed would turn a DO incident into a viewer outage. The limit bounds cost; it is not an access control. ## Semantics Approximate. Worst-case enforcement lag after the request that takes a client to its limit: ``` lag ≤ LINGER (200 ms) + apply RPC (+ retries, ≤ 5 s worst) + max-age (1 s) + one revalidation + the client's next-request gap ``` Steady state (RPC ≈ tens of ms, traffic flowing) that is ≈ 1.5 s. A client sending 2× the limit (2 req/s at 60/min) overshoots by about lag × rate ≈ 3 requests per window. The DO measures the true overshoot (`peak − limit`); the load generator measures lag end to end (first 429 after the request that should have been the first refusal). Identical to today: the limit is on admitted requests (a 429 does not count), `page` 60/min and `pack` 20/min per client, `retry-after` from the block's `until`. ## Privacy As today: the client key is SHA-256 of `ply-viewer-rate/1\n`; only its first 16 hex (64 bits) is sent, stored and cached. No address is stored. The blocked document is served only over `ctx.exports`, never to a browser. ## Load-test controls A request carrying `x-ply-load-token: ` (compared in constant time; never forwarded, logged or echoed; Workers Logs records the header as `REDACTED`) may also send `x-ply-rate-client` (a label hashed in place of the address, so one generator can be many clients), `x-ply-rate-mode: legacy|batched|memo` (choose the limiter for this request: an A/B inside one version), and `x-ply-rate-fault` (fault injection on the DO, below). Without a valid token those headers are ignored. `PLY_LOAD_TOKEN` is a credential of its own (2026-11-07): it unlocks these headers and nothing else. `PLY_TOKEN` (the /v1 bearer) does not unlock them, and the load generator (`ply-load`) holds only `PLY_LOAD_TOKEN` (for its control API and the header). With no `PLY_LOAD_TOKEN` bound, no request is a load request. Local copy: `~/.config/ply/load-token` (0600). `x-ply-rate-fault: slow-ack=;attempts=` makes the DO apply, then hold its answer `` on attempts ≤ n (the client times out and retries; the retry must be a replay). `error;attempts=` makes the DO throw before applying on attempts ≤ n. A faulted request flushes in a batch of its own, at once, never the isolate's shared batch: a fault slows, errors or drops only that request's own count, never a real client's. ## Instrumentation Workers spans (`ctx.tracing`), each stamped with `ply.route` and `ply.script_version` like ply's spans: | span | attributes | |---|---| | `ply.rate.check` | `rate.mode` legacy/batched, `rate.mode_source` default/header (header: a load request chose the arm), `rate.cold` isolate_start/memo_expired (an expected cold check: see "CD gate"), `rate.bucket`, `rate.decision` allow/deny/fail_open, `rate.fail_open_reason` (on every fail_open: isolate_start / memo_expired / read_failed / read_late), `rate.check_ms`, `rate.cache` (HIT/STALE/MISS/EXPIRED/…), `rate.read` ok/memo/fail, `rate.doc_entries`, `rate.doc_age_ms`, `rate.blocked_for_ms` (deny: now − crossed) | | `ply.rate.check` (memo) | `rate.read` memo_fresh / memo_stale_refresh / cold, `rate.cache` MEMO, `rate.memo_age_ms` (now − the isolate's last read), `rate.doc_age_ms` (now − the DO's `at`), `rate.refresh.started`; cold: `rate.memo_expired_ms` (the expired document's age, when the isolate held one), `rate.cold_last_read` (read_failed: how the last refresh ended, fail/memo/abandoned/threw) | | `ply.rate.refresh` | `rate.refresh.trigger` cold/stale, `.cold` (trigger cold: the starting check's cause), `.memo_expired_ms`, `.read` ok/memo/fail, `.cache` (the RateCache read's cf-cache-status), `.parsed`, `.ms`, `.joins` (requests that found it in flight), `rate.doc_age_ms` | | `ply.rate.flush` | `rate.batch.id`, `.keys`, `.hits`, `.linger_ms`, `.attempts`, `.attempt_log` (`ok:34,timeout:1000,…`), `.timeouts`, `.errors`, `.outcome` ok/dropped, `.deadline_hit`, `.replay` (DO said duplicate), `.flush_ms`, `.alive_ms` (batch open → flush end, the waitUntil lifetime), `rate.do.apply_ms`, `rate.do.rows_written`, `rate.do.new_blocks` | DO log lines: `ply.rate.apply` (id, attempt, replay, keys, hits, apply_ms, rows_written, new_blocks, fault) and `ply.rate.over` (a key at or over its limit: sum, limit, crossed, peak). Queried by `$workers.scriptVersion.id`. ## CD gate (silent degradation) The limiter fails soft: a broken read admits (`fail_open`), a flush that cannot reach the counter is dropped after 5 s, and a missing counter sends every request to `legacy` (59926117 / e61a5742: nothing alerted). All of those answer 200, so the smoke and the error gates pass. `scripts/cd/rate-gate.ts` runs inside the telemetry stage of the 0% canary (fail → stop) and of the live watch (fail → rollback): 30 pinned probes of `/view?remote=` (the redirect route: rate-checked, no git work) as a load client with no mode header, then the version's own spans since the stage began, by aggregate count. It fails the version when - any default-mode check (`rate.mode_source` default) ran a limiter other than `rate.expectMode` (memo since 2026-11-07; test/cd-rate-gate.test.ts asserts it equals `RATE_MODE` in cloudflare.config.ts); - warm default-mode checks (no `rate.cold`, plus memo_expired checks whose own refresh failed) failed open above 10% (with ≥ 20 of them; fewer is a harness fault); - any flush was dropped (deadline) or any batch overflowed without an injected fault. The thresholds and their evidence (7 days of spans, 128,753 flushes, every version since the limiter shipped) are in `scripts/cd/cd.policy.json` `rate`. A deliberate `RATE_MODE` rollback (batched or legacy) changes `rate.expectMode` too, in the same commit, shipped through the pipeline: a `cf` var or secret edit creates and deploys a version at 100% directly (d4e3eea6 did). Expected cold checks (`rate.cold`, counted on their own line and kept out of the warm share; ply-web#156): - `isolate_start`: an isolate's first check, and any memo check that is cold while that isolate's first refresh is still in flight. - `memo_expired`: the isolate's last refresh read the document ok, and the document then aged past MEMO_FALLBACK_MS (5 min) because no request came to refresh it (an idle isolate). The check admits and starts a refresh stamped `rate.refresh.cold = memo_expired`; if that refresh does not read ok, the check is put back as a warm failure. CD makes this likely: the canary isolate takes the 30 probes in ~11 s and then idles through the ramp, so the live watch's first request on it is past 5 min (2026-11-12T13-06-51-511Z: 339.4 s idle, refresh ok in 6 ms, counted 1/29 warm before this). A cold read after a refresh has settled with no document (`rate.fail_open_reason = read_failed`: the read path is broken) is a warm `fail_open` and counts. The gate prints both, e.g. `fail_open warm 0/28 = 0.00%; expected fail_open isolate_start 4/4, memo_expired 1/1 (refresh ok 1, failed 0)`. ## Results Load: ply-load (`bench/load`), generator DOs in wnam/enam/weur/apac (4 shards each, 5 req/s each, open loop), `GET /r/ply`, the two arms interleaved per request inside ONE version via `x-ply-rate-mode`, pinned with `Cloudflare-Workers-Version-Overrides`. Server numbers: `ply.rate.check` spans (`rate.check_ms`) and `ply.request` logs joined by trace, filtered by `ply.script_version`. Synthetic clients each stay under the limit. | run · version | arm | n checks | check p50 / p99 / p99.9 / max ms | page wall p50 / p99 / p99.9 ms | |---|---|---|---|---| | lat3 · 6ed2e746 (no read budget) | legacy | 13,187 | 15 / 404 / 533 / 1375 | 27 / 462 / 1567 | | lat3 · 6ed2e746 (no read budget) | batched | 13,188 | 3 / 13 / 49 / 373 | 16 / 107 / 1446 | | lat4 · a020b1b1 (25 ms read budget) | legacy | 13,177 | 14 / 403 / 588 / 3274 | 28 / 463 / 3447 | | lat4 · a020b1b1 (25 ms read budget) | batched | 15,099 | **3 / 12 / 25 / 25** | 16 / 71 / 3049 | | lat5 · 78ddd5c3 (promoted, unpinned) | legacy | 10,778 | 13 / 424 / 584 / 819 | 26 / 531 / 2739 | | lat5 · 78ddd5c3 (promoted, unpinned) | default (batched) | 10,782 | **3 / 12 / 25 / 25** | 15 / 193 / 2747 | | lat6 · 2a4741c0 (pinned, interleaved) | batched | 13,571 | 3 / 14 / 25 / 25 | 12 / 165 / 4690 | | lat6 · 2a4741c0 (pinned, interleaved) | memo | 13,572 | **0 / 0 / 0 / 0** | 9 / 153 / 4702 | | lat7 · 2a4741c0 (pinned, memo only) | memo | 13,583 | 0 / 0 / 0 / 0 | 10 / 69 / 1302 | The page p99.9 (≈3 s in both arms) is git work, not the limiter (lat1, A/A legacy, has the same tail). - **Read cache (lat4):** HIT 77.9%, UPDATING (served stale while one request revalidates) 21.9%, past the 25 ms budget 0.19% (29); 13 of those had no document in the isolate and were admitted (`fail_open`). HIT+SWR 99.8%. - **Enforcement (abuse runs, 12 clients/arm at 2× the 60/min limit for 100 s):** batched lag p50 511 / max 1007 ms (abuse2, a020b1b1; abuse1 6ed2e746: p50 1007, max 1012), overshoot p50 1 / max 2 requests per client (the DO's `peak − limit` agrees: 2). Legacy: lag 19–44 ms, overshoot 0. Admitted totals over the run are equal (1,440 vs 1,440). - **Flushes (lat4, a020b1b1, no fault):** 12,525 flushes, 99.9% one attempt; 11 timeouts in 12,536 attempts at 1.5 s (lat3 at 1.0 s: 175 in 11,655, every one later a DO replay). RPC p50 / p99 / p99.9 142 / 762 / 1363 ms. Keys per batch p50 1, p99 3: at ~1 req/s per colo an isolate rarely sees two hits in 200 ms, so the DO is written about once per admitted request; the batching pays off only with denser per-isolate traffic. - **Counter DO (lat3 window):** 11,228 invocations (≈47/s); CPU p50 0 / p99 2 / p99.9 7 ms; wall p50 133 / p99 1317 ms. The wall is the SQLite commit behind the output gate, and it is the RPC latency the flush sees. - **Fault injection in prod (a020b1b1, fault1 SJC / fault2 LHR+MRS, 20–60 s of each run):** `slow-ack=2500` → 305/305 batches: attempt 1 timed out at 1.5 s, attempt 2 acked as a replay. `error;attempts=2` → 280 batches ok on attempt 3, 3 on attempt 4, 1 hit the 5 s deadline (recorded `dropped`), yet the DO log shows its attempt 3 applied and attempt 4 replayed: counted once. Across 13,978 applies, no id applied twice. - **After promotion (lat5, 78ddd5c3 at 100%, unpinned, default vs `x-ply-rate-mode: legacy`, 10,800 requests/arm):** all 200, 0 wrong-version. Client TTFB p50 / p99 / p99.9 45 / 935 / 4505 ms default vs 59 / 1035 / 4717 legacy. Read cache HIT 74.7%, UPDATING 25.1%, LATE 0.25% (27; 9 `fail_open`). 10,563 flushes, all ok on attempt 1 (0 timeouts, 0 replays, 0 deadline); RPC p50 / p99 / p99.9 150 / 290 / 453 ms, a much tighter tail than lat4's 762 / 1363. Real (non-load) traffic on batched in the window: 58 checks, p99 5 ms. DO `rows_written` p50 6, p99.9 519, max 3983: the large ones are the retention prunes (`rate_hits`, `rate_batches` older than 10 min), not hit writes. - **Memo mode (2a4741c0, 2026-11-07; ply-load ad3db19e).** lat6: batched vs memo interleaved per request, pinned, 27,200 requests (13,600/arm), all 200. The memo check is a map lookup: `rate.check_ms` 0 at every percentile (Workers clocks only advance on I/O, so the page wall from `ply.request` joined by trace is the comparison). Page wall memo − batched, bootstrap 95% CI (400 resamples): p50 −3 ms [−3, −3], p90 −3 [−4, −3], p99 −12 [−39, +12]. So a 3 ms win at the median and no measurable change at p99, which is git work. Batched reads: ok 13,542, LATE 29 (7 `fail_open`). The interleaved memo arm shares the isolate's document with the batched arm (it is module state), which keeps it fresh: memo_age p50 200 ms, 26 refreshes. So refresh and lag were measured memo-only. lat7, memo only, 13,583 checks: memo_fresh 9,046, memo_stale_refresh 4,495, cold 42 (`fail_open`; batched lat6: 7). memo_age p50/p99/p99.9 400 / 602 / 653 ms. 4,517 refreshes (one per 3 requests), refresh ms p50/p99/p99.9/max 4 / 8 / 50 / 301, RateCache HIT 2,944, UPDATING 1,569, EXPIRED 4. 20 single-flight joins (max 5 on one flight). - **Enforcement, memo vs batched (2a4741c0, 12 clients at 2× the limit, 100 s, client-side lag quantised by the 500 ms request gap):** abuse3 interleaved (memo kept fresh by batched): batched lag p50 500 / max 1500 ms, overshoot p50 1 / max 3; memo the same. abuse5 batched only: lag p50 500 / max 1500 ms, overshoot max 3, DO `peak − limit` 3. abuse4 memo only: **lag p50 1500 / max 2000 ms, overshoot p50 3 / max 4**, DO `peak − limit` 4. Memo adds about one request gap plus ≤ 500 ms of memo age on top of the cache's max-age and SWR. - **Decision (superseded 2026-11-07, below):** memo stays opt-in (`x-ply-rate-mode: memo` / `RATE_MODE=memo`) and the default stays batched. Its page p99 gain is not measurable and its lag max (2.0 s) is over the ~1.5 s bar. 2a4741c0 was promoted with the default unchanged, which ships the bounded prune. - **Counter prune bound (PRUNE_BATCH_ROWS 128).** `ply.rate.apply` rows_written max before: 12,892 (78ddd5c3), 7,915 (a020b1b1). After: lat8, 2a4741c0 at 100% unpinned, default arm, 13,600 requests all 200, starting on the backlog that lat6/lat7 left: 6,550 applies, rows_written median 9 / p99 140 / max 268. - **Bursts (2026-11-07, canary 9d33f400 pinned, ply-load de344fdb, runs burst2-c{1,5}-r{50,100,200}).** One client (shared label, 4 regions × 2 generators) at 50 / 100 / 200 req/s for 60 s: admitted in a 60 s window 116 / 125 / 140 (overshoot 56 / 65 / 80; the DO's `peak` equals the client's count every time). Five clients at 10 / 20 / 40 req/s each: overshoot 6-10 / 14-20 / 28-32. Overshoot ≈ client rate × 0.4-1.1 s. DO crossing → first deny per colo p50 0.5-1.8 s, max 4.5 s. The counter stayed healthy: every flush one attempt, 0 timeouts / replays / drops, apply RPC p99 ≤ 294 ms, CPU p99 ≤ 4 ms, rows_written p99 ≤ 51; denied requests never flush. The most a client's requests landed in one colo before it was refused was 59, so a per-isolate pre-count at the limit would not have bitten here. - **Memo becomes the default (2026-11-07; Ivar: "Overshoot is not horrible, we budget that into the limit. I care most about hot path perf.").** Commits 53a8e47 (RATE_MODE memo, gate expects memo) + e6baadd (memory bounds). Canary dbabbca6-b606-4c97-856b-22423a4a65da at 0% beside 935e8eb1-bdb5-4572-b7de-058d4c51492e (batched; it already ran REFS_POLICY=watched, so the arms differ only in the limiter). ply-load de344fdb, both versions pinned, interleaved per request, GET /r/ply, load checks joined to `ply.request` by trace: | run · arm | checks | check p50 / p99 / p99.9 / max ms | pages | wall p50 / p90 / p99 / p99.9 ms | |---|---|---|---|---| | memoab1 · 935e8eb1 batched | 14,138 | 3 / 8 / 25 / 25 | 13,874 | 11 / 15 / 34 / 755 | | memoab1 · dbabbca6 memo | 14,142 | **0 / 0 / 0 / 0** | 13,976 | **9 / 14 / 30 / 562** | | mem1m2 · 935e8eb1 batched | 531,656 | 3 / 19 / 25 / 25 | 531,764 | 14 / – / 154 / 295 | | mem1m2 · dbabbca6 memo | 531,728 | **0 / 0 / 0 / 0** | 531,598 | **11 / – / 139 / 256** | memoab1: 6 regions × 4 generators × 4 req/s × 300 s, 16 colos. Memo − batched, bootstrap 95%: check −3 [−3, −3] / −4 [−4, −4] (p90) / −8 [−8, −8] / −25 [−25, −25]; wall p50 −2 [−2, −2], p90 −1 [−2, −1], p99 −4 [−8, −1], p99.9 −193 [−720, +409] (the tail is git work). mem1m2 (48 generators × 25 req/s × 900 s, a new client per request, 1,063,558 distinct): every percentile lower on memo (Workers Observability calculations). Memo reads (memoab1): memo_fresh 3,200, memo_stale_refresh 10,918, cold 24 (all start-up, then stamped `rate.cold_isolate`, now `rate.cold = isolate_start`); warm fail_open 0 vs 1 on batched. Refreshes 10,977 (one per 1.3 requests), all ok, HIT 80% / UPDATING 20%, ms p50 / p99 / p99.9 / max 3 / 8 / 40 / 173; memo age p50 / p99 500 / 1001 ms. Promoted with `npm run cd` (canary + live rate gate: memo on every default check, warm fail_open 0/26 and 0/37, 0 drops; visual smoke 32/32). Previous: 935e8eb1. - **Enforcement with memo (the overshoot budget).** abuse6 (dbabbca6 pinned, 12 clients at 2 req/s = 2× the limit, 4 regions, 100 s): lag p50 / p90 / max 1000 / 2000 / 2500 ms (client clock, quantised by the 500 ms gap), overshoot p50 / p90 / max 2 / 4 / 5 requests per 60 s window; DO `peak` max 65. blk2 (1,536 clients at 1.2× the limit, each on one version, under counter load): batched lag p50 / p99 / max 833 / 1667 / 2500 ms, overshoot max 3; memo 833 / 2500 / 3334 ms, overshoot max 4. Budget about +5 per 60/min client at 2× and ≈ rate × 1–3 s for bursts. - **A million distinct clients (mem1m1, mem1m2, 2026-11-07).** keys held per isolate p50 / p99 / max 6 / 18 / 40 (mem1m2), max 81 (mem1m1): far under MAX_KEYS_HELD; 0 budget or overflow drops. Heap profiles (12 captures of dbabbca6 under mem1m2, 10 s each; the profile carries allocation samples, not a live peak): rate code is 0.10% of allocation (4.5 of 4,626 MiB); the page render dominates. **The one counter is the ceiling:** at ~1,180 admitted req/s from ~50 colos (mem1m2) flush p50 / p99 / p99.9 321 / 1766 / 4167 ms, 4,692 attempt timeouts and 195 batches dropped at the 5 s deadline (under-count; requests unaffected). At ~470 req/s (mem1m1) it was 212 / 948 / 1360 ms with 0 drops. So ≥ 5k concurrently blocked clients (≥ 5k admitted req/s) is beyond one RateCounter; the largest measured document was 1,488 entries / 75,937 bytes (blk2), refresh p99 36 ms. A 0% canary's Durable Object calls run the 100% version's code, so the counter changes in e6baadd (index, adaptive prune, stats) are measured after promotion (below). - **Counter after promotion (do1, dbabbca6 at 100%, 48 generators × 20 req/s × 300 s, a new client per request, 283,913 sent).** 17,167 apply logs: rows_written p50 / p99 / max 18 / 151 / 296; pruned per apply max 138 rate_hits, 129 rate_batches (the budgets). The mem runs had left 30,662 expired `rate_batches` rows (one id per apply at ~330 applies/s outran the old 128-row prune); `ply.rate.stats` showed 30,662 → 1,037 → 377 → 152 within 3.5 min while the inflow continued; expired rate_hits 0; databaseSize 7.8–10.2 MB. Flushes 64,289 ok + 66 dropped, flush p50 / p99 214 / 2051 ms at ~950 admitted req/s: the throughput ceiling is the one object's commits, not its prune.