GitHubivarvong

ply

main
ply7 git calls126 ms20 KiB4 cachedprotocol v2
59 lines5.1 KiBRaw

worker-mem: ply's streaming paths inside a real Worker

A manual harness that proves ply's streaming paths stay inside real Cloudflare Workers limits (128 MB isolate memory, the configured CPU limit) on inputs far larger than isolate memory. It deploys a scratch Worker, ply-memlab, bundled from this checkout's dist/, and drives it from the laptop. Nothing here ships: the package publishes dist/ only, the harness has no dependencies beyond the repo's dev tooling, and CI never runs it (it needs network and an account).

Cells

cellwhat runs in the Workerpass means
scanpackStream of the seed repo's whole history (one ~1 GiB pack) into parsePackInit/Feed/Pull with WORK_BUDGET, verify onthe trailer is reached, every object hashed, nothing retained; no exceededMemory / exceededCpu
readopenBlob of the seed's 30 MiB blob, streamed and countedbytes = declared size
bomba 1 GiB-of-zeros blob (~1 MB deflated) built in the Worker, scanned with WORK_BUDGET and default limitsrefused with limit_exceeded before work passes the budget
bombworkerthe same bomb under WORKER_LIMITSrefused, earlier
indexscanPackIndex (#236) over the same ~1 GiB history, oids on, WORK_BUDGET, verify oncount and trailer reached, every whole object hashed; reports the index's held column bytes per entry and entries whose type stayed unknown
resolvespool writes the whole history once to the scratch R2 bucket ply-memlab-spool (multipart, 16 MiB parts); each resolve run is scanPackIndex over the spooled object, then resolvePackRange over R2 ranged gets (a harness-side RangeSource: ply stays storage-neutral); spooldel deletes the object, the bucket stays0 unknown types and 0 unknown oids, the head commit found; reports both indexes' held bytes, the 16 MiB base-cache bound and the resolver's stats
scanprofiledthe scan cell with a 10 s heap capture taken 4 s inreported apart from scan (profiling adds overhead); the report records which captures overlapped their scan
pushpackone request per seed commit: packStream of that commit (previous commit as a have) piped as a ReadableStream into pushPack (#234), under a 64 MiB − 64 KiB maxPackBytes, into a throwaway ply-loadgen repoevery report line ok; afterwards a stock git clone --mirror of the destination passes git fsck --strict with the seed's head and object count; the destination and its tokens are then removed

Each request's CPU time, wall time and outcome come from Workers Observability, joined to the cell by the request id of the Worker's memlab log line. In-Worker Date.now() only advances on I/O, so in-Worker wall times are I/O time, not CPU. Heap profiles (cf workers versions profile --profile-type heap) are captured while a scan runs. They are allocation sampling (alloc_objects, alloc_space over the capture window), not in-use memory, so they show where bytes are allocated, not the peak; the evidence for bounded memory is the outcome (no exceededMemory) on a pack many times the isolate's 128 MB. Each report records the Worker version, the ply commit it was built from (/health x-memlab-ply) and the configured CPU limit (x-memlab-cpu-limit-ms, 300,000 ms).

Run

sh
node bench/worker-mem/seed.mjs          # once: creates + pushes the seed repo (~1.1 GiB, ~12 min on a home link)
npm run bench:worker-mem -- --cells bomb,bombworker,read --profiles 0          # build, deploy, short cells
npm run bench:worker-mem -- --no-deploy --url https://ply-memlab.<sub>.workers.dev --cells scan --profiles 10  # ~25 min
npm run bench:worker-mem -- --cells pushpack --runs 19 --profiles 0            # relay all 19 seed commits + verify
npm run bench:worker-mem -- --cells resolve --runs 10 --profiles 0             # spool to R2, resolve ×10, delete the spool
node bench/worker-mem/seed.mjs --delete # remove the seed repo

Reports land in bench/worker-mem/reports/<run>.{json,md} (gitignored), profiles in reports/<run>-profiles/. Every number in a report carries the Worker version id it was measured on.

Credentials and cleanup

  • cf runs with CLOUDFLARE_API_TOKEN unset (OAuth), never wrangler.
  • The Worker's bearer is generated once into ~/.config/ply/memlab-token (0600) and uploaded as the MEMLAB_TOKEN secret through a 0600 secrets file that is deleted after the deploy. It is never printed.
  • The seed repo lives in the ply-loadgen Artifacts namespace. seed.mjs revokes the repo-create token, mints one 2-hour write token for the pushes, and revokes it at the end (and checks it is refused). The Worker reads through its Artifacts binding: each request mints a 15-minute read token and revokes it when done.
  • seed.mjs keeps the one seed repo (named in .data/seed.json) for reruns; --delete removes it.

Files

  • worker.mjs: the Worker (cells, NDJSON responses, memlab log line).
  • build.mjs: esbuild bundle + cf Build Output (.cloudflare/output/v0), so cf deploy --prebuilt uploads exactly it.
  • seed.mjs: seed repo create / push / delete.
  • run.mjs: the one command.
  • lib.mjs: cf wrapper, redaction, secrets files, quantiles.
1# worker-mem: ply's streaming paths inside a real Worker
3A manual harness that proves ply's streaming paths stay inside real Cloudflare Workers limits (128 MB isolate memory,
4the configured CPU limit) on inputs far larger than isolate memory. It deploys a scratch Worker, `ply-memlab`, bundled
5from this checkout's `dist/`, and drives it from the laptop. Nothing here ships: the package publishes `dist/` only,
6the harness has no dependencies beyond the repo's dev tooling, and CI never runs it (it needs network and an account).
8## Cells
10| cell | what runs in the Worker | pass means |
11|---|---|---|
12| `scan` | `packStream` of the seed repo's whole history (one ~1 GiB pack) into `parsePackInit/Feed/Pull` with `WORK_BUDGET`, verify on | the trailer is reached, every object hashed, nothing retained; no `exceededMemory` / `exceededCpu` |
13| `read` | `openBlob` of the seed's 30 MiB blob, streamed and counted | bytes = declared size |
14| `bomb` | a 1 GiB-of-zeros blob (~1 MB deflated) built in the Worker, scanned with `WORK_BUDGET` and default limits | refused with `limit_exceeded` before work passes the budget |
15| `bombworker` | the same bomb under `WORKER_LIMITS` | refused, earlier |
16| `index` | `scanPackIndex` (#236) over the same ~1 GiB history, oids on, `WORK_BUDGET`, verify on | count and trailer reached, every whole object hashed; reports the index's held column bytes per entry and entries whose type stayed unknown |
17| `resolve` | `spool` writes the whole history once to the scratch R2 bucket `ply-memlab-spool` (multipart, 16 MiB parts); each `resolve` run is `scanPackIndex` over the spooled object, then `resolvePackRange` over R2 ranged gets (a harness-side `RangeSource`: ply stays storage-neutral); `spooldel` deletes the object, the bucket stays | 0 unknown types and 0 unknown oids, the head commit found; reports both indexes' held bytes, the 16 MiB base-cache bound and the resolver's stats |
18| `scanprofiled` | the `scan` cell with a 10 s heap capture taken 4 s in | reported apart from `scan` (profiling adds overhead); the report records which captures overlapped their scan |
19| `pushpack` | one request per seed commit: `packStream` of that commit (previous commit as a have) piped as a `ReadableStream` into `pushPack` (#234), under a 64 MiB − 64 KiB `maxPackBytes`, into a throwaway `ply-loadgen` repo | every report line `ok`; afterwards a stock `git clone --mirror` of the destination passes `git fsck --strict` with the seed's head and object count; the destination and its tokens are then removed |
21Each request's CPU time, wall time and outcome come from Workers Observability, joined to the cell by the request id
22of the Worker's `memlab` log line. In-Worker `Date.now()` only advances on I/O, so in-Worker wall times are I/O time,
23not CPU. Heap profiles (`cf workers versions profile --profile-type heap`) are captured while a scan runs. They are
24allocation sampling (`alloc_objects`, `alloc_space` over the capture window), not in-use memory, so they show where
25bytes are allocated, not the peak; the evidence for bounded memory is the outcome (no `exceededMemory`) on a pack many
26times the isolate's 128 MB. Each report records the Worker version, the ply commit it was built from (`/health`
27`x-memlab-ply`) and the configured CPU limit (`x-memlab-cpu-limit-ms`, 300,000 ms).
29## Run
31```sh
32node bench/worker-mem/seed.mjs # once: creates + pushes the seed repo (~1.1 GiB, ~12 min on a home link)
33npm run bench:worker-mem -- --cells bomb,bombworker,read --profiles 0 # build, deploy, short cells
34npm run bench:worker-mem -- --no-deploy --url https://ply-memlab.<sub>.workers.dev --cells scan --profiles 10 # ~25 min
35npm run bench:worker-mem -- --cells pushpack --runs 19 --profiles 0 # relay all 19 seed commits + verify
36npm run bench:worker-mem -- --cells resolve --runs 10 --profiles 0 # spool to R2, resolve ×10, delete the spool
37node bench/worker-mem/seed.mjs --delete # remove the seed repo
38```
40Reports land in `bench/worker-mem/reports/<run>.{json,md}` (gitignored), profiles in `reports/<run>-profiles/`.
41Every number in a report carries the Worker version id it was measured on.
43## Credentials and cleanup
45- `cf` runs with `CLOUDFLARE_API_TOKEN` unset (OAuth), never wrangler.
46- The Worker's bearer is generated once into `~/.config/ply/memlab-token` (0600) and uploaded as the `MEMLAB_TOKEN`
47 secret through a 0600 secrets file that is deleted after the deploy. It is never printed.
48- The seed repo lives in the `ply-loadgen` Artifacts namespace. `seed.mjs` revokes the repo-create token, mints one
49 2-hour write token for the pushes, and revokes it at the end (and checks it is refused). The Worker reads through
50 its Artifacts binding: each request mints a 15-minute read token and revokes it when done.
51- `seed.mjs` keeps the one seed repo (named in `.data/seed.json`) for reruns; `--delete` removes it.
53## Files
55- `worker.mjs`: the Worker (cells, NDJSON responses, `memlab` log line).
56- `build.mjs`: esbuild bundle + cf Build Output (`.cloudflare/output/v0`), so `cf deploy --prebuilt` uploads exactly it.
57- `seed.mjs`: seed repo create / push / delete.
58- `run.mjs`: the one command.
59- `lib.mjs`: cf wrapper, redaction, secrets files, quantiles.
Diagnostics7 git calls · 126 ms
CallStatusSizeTime0126 ms
discover200191 B2 ms
git.upload-pack discover
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=2
git.response.bytes=191
git.server.agent=git/github-9922ec520325-Linux
git.server.object_format=sha1
git.server.caps=agent ls-refs fetch server-option object-format
git.server.fetch=shallow wait-for-done filter
git.protocol.version=2
git.pkt.frames=9
git.pkt.data=7
git.pkt.flush=2
git.pkt.delim=0
git.pkt.bytes=191
git.pkt.payload.max=36
ls-refs20012 KiB2 ms
git.upload-pack ls-refs
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=ls-refs
git.upstream.dur_ms=2
git.response.bytes=11990
git.response.cache=UPDATING
git.response.duration_ms=2
git.protocol.version=2
git.pkt.frames=152
git.pkt.data=151
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=11990
git.pkt.payload.max=122
git.ls_refs.count=151
git.ls_refs.head=refs/heads/main
fetchtree:0 · deepen 102004.2 KiB · hit2 ms
git.upload-pack fetch
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=3
git.response.bytes=4347
git.response.cache=HIT
git.response.duration_ms=2
git.fetch.filter=tree:0
git.fetch.wants=19912563f540228593df0c489082e5e2508e21e4
git.fetch.haves.count=0
git.fetch.deepen=10
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=7
git.pkt.data=5
git.pkt.flush=1
git.pkt.delim=1
git.pkt.bytes=4347
git.pkt.payload.max=4247
git.sideband.band1.bytes=4247
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0
git.pack.objects=10
git.pack.bytes=4247
git.pack.ofs_delta=0
git.pack.ref_delta=0
git.pack.chain.max=0
git.pack.trailer=a6caf8e129bea82c9bae4cbc7c130c9205add400
git.pack.full=10
Open pack map
fetchtree:0200919 B · hit3 ms
git.upload-pack fetch
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=2
git.response.bytes=919
git.response.cache=HIT
git.response.duration_ms=3
git.fetch.filter=tree:0
git.fetch.wants=800d30ba8923860bfc460ca884b7f9ba22083588
git.fetch.haves.count=0
git.fetch.deepen=0
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=4
git.pkt.data=3
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=919
git.pkt.payload.max=892
git.sideband.band1.bytes=892
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0
git.pack.objects=1
git.pack.bytes=892
git.pack.trailer=5323b9266eea5ded637285d1c5320699f12be76e
Open pack map
fetchtree:0200106 B · hit2 ms
git.upload-pack fetch
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=3
git.response.bytes=106
git.response.cache=HIT
git.response.duration_ms=2
git.fetch.filter=tree:0
git.fetch.wants=72bbbdb37dc951e0de5615d1376ca1b23588856c
git.fetch.haves.count=0
git.fetch.deepen=0
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=4
git.pkt.data=3
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=106
git.pkt.payload.max=79
git.sideband.band1.bytes=79
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0
git.pack.objects=1
git.pack.bytes=79
git.pack.trailer=1cca45a06ebd0c08bccef52d1cdb92cae1868075
Open pack map
fetchtree:0200291 B · hit3 ms
git.upload-pack fetch
http.response.status_code=200
git.upstream.cf_ray=-
git.upstream.phase=fetch
git.upstream.dur_ms=114
git.response.bytes=291
git.response.cache=HIT
git.response.duration_ms=3
git.fetch.filter=tree:0
git.fetch.wants=d3ab99f804f647a2fe02e623e69e5ccc7a17b935
git.fetch.haves.count=0
git.fetch.deepen=0
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=4
git.pkt.data=3
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=291
git.pkt.payload.max=264
git.sideband.band1.bytes=264
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0
git.pack.objects=1
git.pack.bytes=264
git.pack.trailer=1fec5d654a53d2752ea2b9c5bd06569c4565d5b6
Open pack map
fetch2002.5 KiB114 ms
git.upload-pack fetch
http.response.status_code=200
git.response.bytes=2524
git.response.cache=MISS
git.response.duration_ms=114
git.fetch.wants=365d9e6cc5b684409956521edcae9c954017d937
git.fetch.haves.count=0
git.fetch.deepen=0
git.fetch.thin_pack=false
git.fetch.ofs_delta=true
git.protocol.version=2
git.pkt.frames=4
git.pkt.data=3
git.pkt.flush=1
git.pkt.delim=0
git.pkt.bytes=2524
git.pkt.payload.max=2497
git.sideband.band1.bytes=2497
git.sideband.band2.bytes=0
git.sideband.band3.bytes=0

Each row is one HTTP round trip ply made to the git server for this page, on one timeline. discover asks what the server speaks, ls-refs lists branch tips, fetch brings objects by id. A hit was served by Workers Cache. Tap a row for its wire details.

Remote
https://github.com/ivarvong/ply
Ref
refs/heads/main
Commit
19912563f540228593df0c489082e5e2508e21e4
cf-ray
a5e302324b817929
Request
fca60fde…
Protocol
v2 · sha1
Agent
git/github-9922ec520325-Linux
Filter
advertised
Fetch
shallow wait-for-done filter
Capabilities
agent ls-refs fetch server-option object-format