Artifactspublic-v0

ply

main
ply4 git calls334 ms40 KiB2 cachedprotocol v2
59 lines5.1 KiBRaw

worker-mem: ply's streaming paths inside a real Worker

A manual harness that proves ply's streaming paths stay inside real Cloudflare Workers limits (128 MB isolate memory, the configured CPU limit) on inputs far larger than isolate memory. It deploys a scratch Worker, ply-memlab, bundled from this checkout's dist/, and drives it from the laptop. Nothing here ships: the package publishes dist/ only, the harness has no dependencies beyond the repo's dev tooling, and CI never runs it (it needs network and an account).

Cells

cellwhat runs in the Workerpass means
scanpackStream of the seed repo's whole history (one ~1 GiB pack) into parsePackInit/Feed/Pull with WORK_BUDGET, verify onthe trailer is reached, every object hashed, nothing retained; no exceededMemory / exceededCpu
readopenBlob of the seed's 30 MiB blob, streamed and countedbytes = declared size
bomba 1 GiB-of-zeros blob (~1 MB deflated) built in the Worker, scanned with WORK_BUDGET and default limitsrefused with limit_exceeded before work passes the budget
bombworkerthe same bomb under WORKER_LIMITSrefused, earlier
indexscanPackIndex (#236) over the same ~1 GiB history, oids on, WORK_BUDGET, verify oncount and trailer reached, every whole object hashed; reports the index's held column bytes per entry and entries whose type stayed unknown
resolvespool writes the whole history once to the scratch R2 bucket ply-memlab-spool (multipart, 16 MiB parts); each resolve run is scanPackIndex over the spooled object, then resolvePackRange over R2 ranged gets (a harness-side RangeSource: ply stays storage-neutral); spooldel deletes the object, the bucket stays0 unknown types and 0 unknown oids, the head commit found; reports both indexes' held bytes, the 16 MiB base-cache bound and the resolver's stats
scanprofiledthe scan cell with a 10 s heap capture taken 4 s inreported apart from scan (profiling adds overhead); the report records which captures overlapped their scan
pushpackone request per seed commit: packStream of that commit (previous commit as a have) piped as a ReadableStream into pushPack (#234), under a 64 MiB − 64 KiB maxPackBytes, into a throwaway ply-loadgen repoevery report line ok; afterwards a stock git clone --mirror of the destination passes git fsck --strict with the seed's head and object count; the destination and its tokens are then removed

Each request's CPU time, wall time and outcome come from Workers Observability, joined to the cell by the request id of the Worker's memlab log line. In-Worker Date.now() only advances on I/O, so in-Worker wall times are I/O time, not CPU. Heap profiles (cf workers versions profile --profile-type heap) are captured while a scan runs. They are allocation sampling (alloc_objects, alloc_space over the capture window), not in-use memory, so they show where bytes are allocated, not the peak; the evidence for bounded memory is the outcome (no exceededMemory) on a pack many times the isolate's 128 MB. Each report records the Worker version, the ply commit it was built from (/health x-memlab-ply) and the configured CPU limit (x-memlab-cpu-limit-ms, 300,000 ms).

Run

sh
node bench/worker-mem/seed.mjs          # once: creates + pushes the seed repo (~1.1 GiB, ~12 min on a home link)
npm run bench:worker-mem -- --cells bomb,bombworker,read --profiles 0          # build, deploy, short cells
npm run bench:worker-mem -- --no-deploy --url https://ply-memlab.<sub>.workers.dev --cells scan --profiles 10  # ~25 min
npm run bench:worker-mem -- --cells pushpack --runs 19 --profiles 0            # relay all 19 seed commits + verify
npm run bench:worker-mem -- --cells resolve --runs 10 --profiles 0             # spool to R2, resolve ×10, delete the spool
node bench/worker-mem/seed.mjs --delete # remove the seed repo

Reports land in bench/worker-mem/reports/<run>.{json,md} (gitignored), profiles in reports/<run>-profiles/. Every number in a report carries the Worker version id it was measured on.

Credentials and cleanup

  • cf runs with CLOUDFLARE_API_TOKEN unset (OAuth), never wrangler.
  • The Worker's bearer is generated once into ~/.config/ply/memlab-token (0600) and uploaded as the MEMLAB_TOKEN secret through a 0600 secrets file that is deleted after the deploy. It is never printed.
  • The seed repo lives in the ply-loadgen Artifacts namespace. seed.mjs revokes the repo-create token, mints one 2-hour write token for the pushes, and revokes it at the end (and checks it is refused). The Worker reads through its Artifacts binding: each request mints a 15-minute read token and revokes it when done.
  • seed.mjs keeps the one seed repo (named in .data/seed.json) for reruns; --delete removes it.

Files

  • worker.mjs: the Worker (cells, NDJSON responses, memlab log line).
  • build.mjs: esbuild bundle + cf Build Output (.cloudflare/output/v0), so cf deploy --prebuilt uploads exactly it.
  • seed.mjs: seed repo create / push / delete.
  • run.mjs: the one command.
  • lib.mjs: cf wrapper, redaction, secrets files, quantiles.
1# worker-mem: ply's streaming paths inside a real Worker
3A manual harness that proves ply's streaming paths stay inside real Cloudflare Workers limits (128 MB isolate memory,
4the configured CPU limit) on inputs far larger than isolate memory. It deploys a scratch Worker, `ply-memlab`, bundled
5from this checkout's `dist/`, and drives it from the laptop. Nothing here ships: the package publishes `dist/` only,
6the harness has no dependencies beyond the repo's dev tooling, and CI never runs it (it needs network and an account).
8## Cells
10| cell | what runs in the Worker | pass means |
11|---|---|---|
12| `scan` | `packStream` of the seed repo's whole history (one ~1 GiB pack) into `parsePackInit/Feed/Pull` with `WORK_BUDGET`, verify on | the trailer is reached, every object hashed, nothing retained; no `exceededMemory` / `exceededCpu` |
13| `read` | `openBlob` of the seed's 30 MiB blob, streamed and counted | bytes = declared size |
14| `bomb` | a 1 GiB-of-zeros blob (~1 MB deflated) built in the Worker, scanned with `WORK_BUDGET` and default limits | refused with `limit_exceeded` before work passes the budget |
15| `bombworker` | the same bomb under `WORKER_LIMITS` | refused, earlier |
16| `index` | `scanPackIndex` (#236) over the same ~1 GiB history, oids on, `WORK_BUDGET`, verify on | count and trailer reached, every whole object hashed; reports the index's held column bytes per entry and entries whose type stayed unknown |
17| `resolve` | `spool` writes the whole history once to the scratch R2 bucket `ply-memlab-spool` (multipart, 16 MiB parts); each `resolve` run is `scanPackIndex` over the spooled object, then `resolvePackRange` over R2 ranged gets (a harness-side `RangeSource`: ply stays storage-neutral); `spooldel` deletes the object, the bucket stays | 0 unknown types and 0 unknown oids, the head commit found; reports both indexes' held bytes, the 16 MiB base-cache bound and the resolver's stats |
18| `scanprofiled` | the `scan` cell with a 10 s heap capture taken 4 s in | reported apart from `scan` (profiling adds overhead); the report records which captures overlapped their scan |
19| `pushpack` | one request per seed commit: `packStream` of that commit (previous commit as a have) piped as a `ReadableStream` into `pushPack` (#234), under a 64 MiB − 64 KiB `maxPackBytes`, into a throwaway `ply-loadgen` repo | every report line `ok`; afterwards a stock `git clone --mirror` of the destination passes `git fsck --strict` with the seed's head and object count; the destination and its tokens are then removed |
21Each request's CPU time, wall time and outcome come from Workers Observability, joined to the cell by the request id
22of the Worker's `memlab` log line. In-Worker `Date.now()` only advances on I/O, so in-Worker wall times are I/O time,
23not CPU. Heap profiles (`cf workers versions profile --profile-type heap`) are captured while a scan runs. They are
24allocation sampling (`alloc_objects`, `alloc_space` over the capture window), not in-use memory, so they show where
25bytes are allocated, not the peak; the evidence for bounded memory is the outcome (no `exceededMemory`) on a pack many
26times the isolate's 128 MB. Each report records the Worker version, the ply commit it was built from (`/health`
27`x-memlab-ply`) and the configured CPU limit (`x-memlab-cpu-limit-ms`, 300,000 ms).
29## Run
31```sh
32node bench/worker-mem/seed.mjs # once: creates + pushes the seed repo (~1.1 GiB, ~12 min on a home link)
33npm run bench:worker-mem -- --cells bomb,bombworker,read --profiles 0 # build, deploy, short cells
34npm run bench:worker-mem -- --no-deploy --url https://ply-memlab.<sub>.workers.dev --cells scan --profiles 10 # ~25 min
35npm run bench:worker-mem -- --cells pushpack --runs 19 --profiles 0 # relay all 19 seed commits + verify
36npm run bench:worker-mem -- --cells resolve --runs 10 --profiles 0 # spool to R2, resolve ×10, delete the spool
37node bench/worker-mem/seed.mjs --delete # remove the seed repo
38```
40Reports land in `bench/worker-mem/reports/<run>.{json,md}` (gitignored), profiles in `reports/<run>-profiles/`.
41Every number in a report carries the Worker version id it was measured on.
43## Credentials and cleanup
45- `cf` runs with `CLOUDFLARE_API_TOKEN` unset (OAuth), never wrangler.
46- The Worker's bearer is generated once into `~/.config/ply/memlab-token` (0600) and uploaded as the `MEMLAB_TOKEN`
47 secret through a 0600 secrets file that is deleted after the deploy. It is never printed.
48- The seed repo lives in the `ply-loadgen` Artifacts namespace. `seed.mjs` revokes the repo-create token, mints one
49 2-hour write token for the pushes, and revokes it at the end (and checks it is refused). The Worker reads through
50 its Artifacts binding: each request mints a 15-minute read token and revokes it when done.
51- `seed.mjs` keeps the one seed repo (named in `.data/seed.json`) for reruns; `--delete` removes it.
53## Files
55- `worker.mjs`: the Worker (cells, NDJSON responses, `memlab` log line).
56- `build.mjs`: esbuild bundle + cf Build Output (`.cloudflare/output/v0`), so `cf deploy --prebuilt` uploads exactly it.
57- `seed.mjs`: seed repo create / push / delete.
58- `run.mjs`: the one command.
59- `lib.mjs`: cf wrapper, redaction, secrets files, quantiles.
Diagnostics4 git calls · 334 ms
CallStatusSizeTime0334 ms
discover200152 B3 ms
git.upload-pack discover
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=ls-refs
git.upstream.dur_ms=3
git.response.bytes=152
git.server.agent=gitty/1.0
git.server.object_format=sha1
git.server.caps=agent ls-refs fetch object-format
git.server.fetch=shallow filter sideband-all
git.protocol.version=2
git.pkt.frames=8
git.pkt.data=6
git.pkt.flush=2
git.pkt.delim=0
git.pkt.bytes=152
git.pkt.payload.max=34
ls-refs200145 B · hit3 ms
git.upload-pack ls-refs
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=ls-refs
git.upstream.dur_ms=3
git.response.bytes=145
git.response.cache=HIT
git.response.duration_ms=3
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=145
git.pkt.payload.max=76
git.ls_refs.count=2
git.ls_refs.head=refs/heads/main
fetchblob:none · deepen 1020038 KiB · hit2 ms
git.upload-pack fetch
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=2
git.response.bytes=38434
git.response.cache=HIT
git.response.duration_ms=2
git.fetch.filter=blob:none
git.fetch.filter.requested=tree:0
git.fetch.filter.fallback=blob:none
git.fetch.filter.learned=false
git.fetch.wants=19912563f540228593df0c489082e5e2508e21e4
git.fetch.haves.count=0
git.fetch.deepen=10
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=6
git.pkt.data=4
git.pkt.flush=1
git.pkt.delim=1
git.pkt.bytes=38434
git.pkt.payload.max=38339
git.sideband.band1.bytes=38338
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0
git.pack.objects=63
git.pack.bytes=38338
git.pack.ofs_delta=0
git.pack.ref_delta=8
git.pack.chain.max=8
git.pack.trailer=0bc36ab7adf82d3109177f11f1ea93f1ce071f77
git.pack.full=55
Open pack map
fetch2002.4 KiB329 ms
git.upload-pack fetch
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=328
git.response.bytes=2495
git.response.cache=MISS
git.response.duration_ms=329
git.fetch.wants=365d9e6cc5b684409956521edcae9c954017d937
git.fetch.haves.count=0
git.fetch.deepen=0
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=3
git.pkt.data=2
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=2495
git.pkt.payload.max=2474
git.sideband.band1.bytes=2473
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0

Each row is one HTTP round trip ply made to the git server for this page, on one timeline. discover asks what the server speaks, ls-refs lists branch tips, fetch brings objects by id. A hit was served by Workers Cache. Tap a row for its wire details.

Remote
https://5ad2236b39de4c8bd8f6270ea63f3ea9.artifacts.cloudflare.net/git/public-v0/ply.git
Ref
refs/heads/main
Commit
19912563f540228593df0c489082e5e2508e21e4
cf-ray
a5e33f920bdbbe76
Request
fafa1259…
Protocol
v2 · sha1
Agent
gitty/1.0
Filter
advertised
Fetch
shallow filter sideband-all
Capabilities
agent ls-refs fetch object-format