Skip to content

Benchmarks

Measured throughput and text coverage for the real backends over the pinned local corpus. Two runs from 2026-10-31, on different machines and commits — read them as separate measurements, not as a series:

  • CPU, a546d73 (2026.10.13.post53.dev0+a546d73), 17 rows over 9 documents, PyMuPDF not installed — Results.
  • GPU, 9cbb645 (2026.10.16.post8.dev0+9cbb645), 3 fixtures × 8 backends = 24 rows, the five-way OCR pass — five-way results. Its finding is an 8 GB raster limit, not a backend ranking. The stamp is the install-time label: the pass imported the tree at 97fc8ab, which is the commit to reproduce from — see Provenance in that section.

The 2026-09-27 snapshot (11 rows, with PyMuPDF measured) is retained under 2026-09-27 snapshot with its provenance; no historical number is presented as a new-run result.

What was measured

Documents (all local, never fetched at run time):

document format pages
…-pg1342.txt (Gutenberg, Pride & Prejudice, 754 KiB) text/plain 1
…-rfc5322.txt (119 KiB) text/plain 1
…-commonmark-spec.md (201 KiB) text/markdown 1
exoplanets.csv (7.5 KiB) text/csv 1
whatwg-html.html (151 KiB) text/html 1
wikipedia-calculus.html (792 KiB) text/html 1
itu-t-p863.pdf (licensed, 2.3 MiB) application/pdf 80
etsi-ts-103558.pdf (licensed, 2.7 MiB) application/pdf 68
…-nist-sp-800-53r5.pdf (5.8 MiB) application/pdf 492

wikidata-q42.json (316.1 KiB) was staged but unsupported in this run — recorded as a skip, never silently dropped.

Backends (2026-10-31 run): native-text, native-markdown, native-html, native-csv (CPU), plus liteparse, pdf-inspector, and trafilatura where eligible. Measured component versions: trafilatura 2.3.1, liteparse 2.17.0, pdf-inspector 1.25.2, openpyxl 3.1.5. Not installed and therefore not measured: pymupdf, docling, marker-pdf, and mineru; OCR was disabled. The runner tries each registered backend eligible under this run's routing constraints and the document's media type. PyMuPDF's registered backend had no installed implementation, so its three PDF rows record typed dependency_missing; the 2026-09-27 snapshot below measured it while installed.

Environment (2026-10-31 run): Windows laptop, CPython 3.14.2, CPU only; the light set installed (pypdf and huggingface-hub are core; liteparse, trafilatura, pdf-inspector, openpyxl as needed). No GPU work is included in this snapshot — see Gaps.

Reproduce

From an isolated checkout of a546d73, with the pinned corpus staged in tests/downloads/ (including the manually supplied PDFs):

uv sync --no-progress --reinstall-package parsecraft
uv pip install 'trafilatura==2.3.1'
uv run --no-sync python -c "from importlib.metadata import version; assert version('parsecraft') == '2026.10.13.post53.dev0+a546d73'"
uv run --no-sync parsecraft benchmark 'tests/downloads/*' --no-ocr -o benchmark-trial/

The metadata assertion prevents a stale editable-install label from misrepresenting the measured checkout. Review the scratch outputs before copying them into docs/benchmarks/; elapsed and memory values vary by run.

The report writers are deterministic: stable key and row ordering, fixed rounding, LF newlines, no wall-clock fields. Re-running on the same inputs yields the same schema and row set — the measured values (elapsed, pages/s, peak bytes) are run data and will vary. Benchmarking goes straight to the backend's convert and never reads the conversion cache, which would report ~0 s and falsify the measurement. Mind the asymmetry when comparing surfaces: a plain parsecraft convert does cache, so a repeated convert can look impossibly fast beside the same row here.

pymupdf extraction needs the pymupdf extra (AGPL); without it the report records a typed dependency_missing row instead of crashing.

Launching a multi-hour run (Windows)

The harness writes its report only when the run finishes, so an OCR leg that takes hours has to outlive the shell that started it. Three launch shapes were measured against the same five-way GPU run (pc-6ev, 2026-10-31), and only the third one finished:

  • Child of the calling process tree — dies with that call. Measured as a 0.1-minute run with 9 rows and zero OCR rows.
  • Console attached, window closed — aborts inside the torch/MKL Fortran runtime: forrtl: error (200): program aborting due to window-CLOSE event. That cost a 65-minute run.
  • Detached, no console — the run that completed (5 h 54 m, 24 rows).
subprocess.Popen(
    argv,
    cwd=checkout,  # a detached child inherits the spawner's cwd and PATH
    stdin=subprocess.DEVNULL,
    stdout=log_file,
    stderr=err_file,
    creationflags=subprocess.DETACHED_PROCESS | subprocess.CREATE_NO_WINDOW,
)

DETACHED_PROCESS alone is not enough: without CREATE_NO_WINDOW the child still owns a console window, and closing it is the forrtl abort above.

Because the child inherits the spawner's PATH, the environment probe reads that PATH — and tesseract and pandoc are binaries, not extras. Launch from a shell whose PATH contains every engine you expect the plan to consider, then check the selected column of the first rows before trusting a long run.

Weights downloaded during a run are silent: a pre-cached backend and an uncached one differ only in elapsed_s, which includes the download. Warm the cache first (parsecraft models install <backend>) and record which backends were pre-cached, or the first rows carry a cost that is not the model's.

Results (2026-10-31, a546d73)

The full harness output is committed alongside this page: benchmark.md and benchmark.json (package 2026.10.13.post53.dev0+a546d73, 17 rows over 9 documents, 8 backends: 14 successful conversions + 3 typed dependency_missing rows, no other failures).

Headlines (CPU, 2026-10-31; values as committed):

document backend elapsed pages/s coverage
pg1342.txt (754 KiB) native-text 0.005 s 194.3 0.988
rfc5322.txt native-text 0.013 s 77.6 0.976
commonmark-spec.md (201 KiB) native-markdown 0.683 s 1.5* 0.717
exoplanets.csv (7.5 KiB) native-csv 0.002 s 466.6 1.212†
whatwg-html.html native-html 0.166 s 6.0* 0.271
wikipedia-calculus.html native-html 0.474 s 2.1* 0.107
etsi-ts-103558.pdf (68 pp) liteparse 2.701 s 25.2 0.925
itu-t-p863.pdf (80 pp) liteparse 3.014 s 26.5 0.978
nist-sp-800-53r5.pdf (492 pp) liteparse 65.343 s 7.5 0.877

Also measured but not planner-selected: pdf-inspector on all three PDFs (0.369 s / 184.3 p/s / 0.884; 4.300 s / 18.6 / 0.982; 6.863 s / 71.7 / 1.010†) and trafilatura on both HTML pages (coverage 0.021 / 0.073). PyMuPDF contributes three dependency_missing rows — listed, not measured.

* Single-page formats: "pages/s" is one page over elapsed time — read the elapsed figure instead. † Coverage above 1.0 needs the ratio reading — see How to read coverage.

How to read text_coverage

text_coverage is the ratio of emitted chunk characters to the analyzer's source character count — an unbounded emitted/source ratio, not an accuracy score. Two rows exceed 1.0 in this run: exoplanets.csv at 1.212 and the NIST row via pdf-inspector at 1.010. Segmentation and projection markup are possible influences; fidelity is unjudged here — the ratio is disclosed, not explained away. Magnitude ordering across backends is comparable (one definition).

Measurement notes

  • Timing and memory come from the benchmark runner: it starts tracemalloc before timing, so peak bytes cover traced Python allocations only — not total RSS and not native/GPU allocations — and elapsed times include the tracing overhead. Compare backends within one run, not against wall-clock expectations.
  • Three rows carry typed dependency_missing failures (PyMuPDF not installed); every other row loaded, so no selected: true + failure combination appears.

Skips in this run: wikidata-q42.json only (unsupported source suffix). No historical PyMuPDF number is presented as a new-run result anywhere on this page.

2026-09-27 snapshot

The 2026-09-27 harness output is preserved verbatim under benchmark-2026-09-27.md and benchmark-2026-09-27.json (package 2026.9.11.post9.dev0+917dbdb, 11 rows; exoplanets.csv was then an honest skip — unsupported suffix — and is measured in the 2026-10-31 run above).

Headlines (CPU pass, 2026-09-27; values as committed in benchmark-2026-09-27.json):

document backend elapsed pages/s coverage
itu-t-p863.pdf (80 pp) pymupdf 0.612 s 130.8 0.999
nist-sp-800-53r5.pdf (492 pp) pymupdf 4.002 s 122.9 0.995
etsi-ts-103558.pdf (68 pp, image-heavy) pymupdf 6.468 s 10.5 0.963
itu-t-p863.pdf (80 pp) liteparse 5.672 s 14.1 1.017
nist-sp-800-53r5.pdf (492 pp) liteparse 115.647 s 4.3 0.904
etsi-ts-103558.pdf (68 pp) liteparse 6.011 s 11.3 1.012
pg1342.txt (754 KiB) native-text 0.002 s 409.9 0.988
rfc5322.txt native-text 0.009 s 116.6 0.976
commonmark-spec.md (201 KiB) native-markdown 0.739 s 1.4* 0.717
whatwg-html.html native-html 0.873 s 1.1* 0.271
wikipedia-calculus.html native-html 2.460 s 0.4* 0.106

* Single-page formats: "pages/s" is one page over elapsed time — read the elapsed figure instead. Coverage may exceed 1.0 (different segmentation is one possible influence; fidelity is unjudged).

GPU OCR legs (bounded fixtures)

The four OCR models ran on an RTX A2000 8 GB over 2-page fixtures of the three corpus PDFs — full-corpus OCR would take hours. At run time the four models pinned two different transformers majors, so the snapshot is two passes over the same fixtures. That split was a pinning fact, not a model requirement: the extras now share one transformers>=5.17,<6 window and the TeleOCR/Unlimited models load from local vendored modeling code, so a re-run needs one environment and one pass (see Deferred runs).

fixture pass A: ovis (5.17) pass A: qianfan (5.17) pass B: tele (4.57.1) pass B: unlimited (4.57.1)
etsi-ts-103558-p1-2 197.6 s 713.7 s 220.2 s 751.7 s
nist-…-p1-2 212.6 s backend_error 81.7 s 128.0 s
itu-t-p863-p1-2 413.8 s 1024.0 s 84.0 s 133.6 s

Cross-version rows behave exactly as typed: the wrong-major pair records backend_error rows whose detail names the cause (e.g. qwen3_5/ qianfan_ocr unknown to transformers 4.57.1), never a silent pass. liteparse rows (0.07–0.16 s for the fixtures) appear in pass B after an entry-point refresh.

Reading rules:

  • elapsed_s includes per-document model load — the runner constructs the backend per document; these are 2-page fixtures, not per-page rates you can multiply to the full corpus.
  • The two passes are different virtualenvs: merge rows per backend, never sum across passes.
  • Full-corpus qianfan is a dated deferral, not a gap — see Deferred runs.
  • selected is recorded before load success/failure: a selected: true + failures: [backend_error] row means “planner picked it, then it could not load in this transformers” — not a planner bug.

GPU OCR five-way with ocr-surya (2026-10-31, design first)

Definition written before the run (pc-6ev item 2; the three decisions are binding):

  • Corpus: 2-page fixtures cut from the same three staged corpus PDFs (pages 1–2) — identical bounded inputs to the 2026-09-28 GPU legs above; full-corpus OCR stays a dated deferral. Fixtures are cut from the sha-pinned corpus files into the repo's gitignored .tmp/ scratch space and are never committed.
  • Comparison set: the five GPU OCR backends — ocr-ovis, ocr-surya, ocr-tele, ocr-qianfan, ocr-unlimited — on the same fixtures in one environment (.venv-gpu: one transformers>=5.17,<6 window; tele and unlimited load their vendored modeling, so no second pass is needed). CPU backends are out of the comparison (they measure a different task); native/pymupdf rows may appear in the artifact because the runner measures every eligible backend and are read as sanity anchors, not comparators.
  • Headlines: pages/s per (fixture, backend) from the harness elapsed_s (model load included — same reading rule as above). Not a peak-VRAM headline — see the WDDM paragraph below for the measurement that killed it.
  • text_coverage is not a headline: these pages have no ground truth; coverage stays recorded in the artifact only.
  • Host scope: this run is on an RTX 2070 8 GB — a different host from the 2026-09-28 A2000 8 GB table above, so absolute times are not comparable across those tables. The same-host five-way is the deliverable and carries ocr-surya's first measured rows.
  • Generation budgets: the harness defaults (the test tier's tighter _MAX_NEW_TOKENS caps do not apply to benchmark runs).

Why there is no peak-VRAM column (measured 2026-10-31). A first draft of this definition promised one, sampled outside the harness with nvidia-smi --query-gpu=memory.used. Two measurements killed it. First, the harness cannot carry it: peak_memory_bytes is tracemalloc, host Python only, so it never sees device memory. Second, neither can a sampler on this host: surya's 2-page ETSI cell reports torch.cuda.max_memory_allocated() of 9155 MiB on an 8191 MiB card, and a synthetic torch.empty loop reached 12288 MiB allocated while mem_get_info() reported free = 0. Windows/WDDM backs past physical VRAM with shared system memory, so any number here measures what PyTorch asked for — it can exceed the card, is not comparable across hosts, and cannot support a "fits in 8 GB" claim. VRAM therefore stays in Deferred runs below with that caveat; measuring it honestly needs per-cell reset_peak_memory_stats() plus an nvidia-smi cross-check on a Linux host.

Results (2026-10-31, 9cbb645)

One environment, one pass: .venv-gpu reporting 2026.10.16.post8.dev0+9cbb645, three 2-page fixtures × eight backends = 24 rows, zero skips, 5 h 54 m, clean exit. That figure is wall clock (launch.log 23:54:22 → stdout.log last write 05:48:50) and it agrees with the artifact, whose elapsed_s values sum to 21 261.9 s = 5 h 54.4 m. Artifacts: benchmark-gpu-fiveway-2026-10-31.json and .md — the committed tables, not a summary of them.

Provenance: reproduce with 97fc8ab, not the stamped version. The harness records package_version from the installed distribution metadata, and this venv was installed editable, so the stamp names the commit it was installed from — not the source the run imported. The working tree at launch was 97fc8ab, which is two commits ahead of 9cbb645 (9d93d71 derives allow_ocr from the OCR backends a host can run; 97fc8ab stops a judge leading an OCR page with a parser). That difference is load-bearing here: 9cbb645 predates the allow_ocr fix and would plan zero OCR rows, which is exactly how the first attempt at this run failed. Anyone re-running from this page must check out 97fc8ab.

The result this run actually establishes is not a speed ranking. At the shipped RASTER_DPI = 300, four of the five OCR models cannot process a single A4 page on an 8 GB card: the harness recorded typed backend_error failures instead of output, and only ocr-qianfan completed all three fixtures.

fixture qianfan surya ovis tele unlimited
etsi-ts-103558-p1-2 2 pages, 939.4 s OOM OOM OOM 1 page, then error
itu-t-p863-p1-2 2 pages, 762.0 s OOM OOM OOM 1 page, then error
nist-sp-800-53r5-p1-2 2 pages, 689.4 s OOM OOM OOM 1 page, then error

For the single model that ran, pages/s is the headline the definition promised: 0.0021 / 0.0026 / 0.0029 pages per second (roughly 345–470 s per page, model load included; all five weight sets were already on disk when the pass started, so no cell pays for a download — see caveat 2). The other four rows have no pages/s because they produced no pages; their failures column is the result.

The failures are the same allocation, not four bugs: ovis and surya both ask for 52.66 GiB in one Tried to allocate (two different model families sharing one qwen3.5-class vision tower), tele asks for 118.03 GiB, and unlimited returns infer_multi returned 0 page segments for 2 pages plus Cannot copy out of meta tensor after offloading. A 300 dpi A4 page is 2480×3509 = 8.7 Mpx, which becomes 34 320 vision patches (≈ 8 580 tokens) for ovis/surya and 44 500 (≈ 11 125 tokens) for tele; the attention itself is small, and the blow-up is the image-token forward. Measured bracket on tele: 100 dpi fine (351 s), 150 dpi fine (408 s), 200 dpi OOM at 5.86 GiB, 300 dpi OOM at 118.03 GiB. Two of these numbers are worth carrying away: the cost is super-linear in pixels (2.25× the pixels, ~20× the allocation), and the only surviving model priced the reduction experimentally — qianfan at 150 dpi took 813 s and produced 3 401 characters where 300 dpi took 2 717 s and produced 3 376, i.e. 3.34× the wall clock for 0.7 % fewer characters.

Four caveats, all of which constrain what may be quoted from that table:

  1. elapsed_s includes model load. _measure builds a fresh backend per (document, backend), so no row is pure inference time.
  2. No cell includes a download. A mid-run weight fetch inflates a cell silently — no failure row, no skip row, only a downloading … line on stderr — so the caches were checked rather than assumed. The recorded pass (started 2026-10-31T23:54:22) logged zero downloading lines on either stream, and every weight file predates it: ovis newest 22:49 (1648.9 MB / 22 files), qianfan 23:07 (9059.2 MB / 33), unlimited 21:58 (6373.6 MB / 27), surya 18:03 (1318.9 MB / 21), tele 21:55 (2714.3 MB / 29). The fetch happened in the two aborted earlier attempts, which is why run.log from that attempt carries the downloading ATH-MaaS/OvisOCR2 line and stdout.log does not. The elapsed_s values are therefore free of download cost; what they still include is model load (caveat 1).
  3. VRAM spilled to host (Some parameters are on the meta device because they were offloaded to the cpu), so even the rows that completed are not clean GPU numbers.
  4. coverage is emitted/source length, not accuracy. It says nothing about whether the text is right, and one artifact row shows how far it can drift: unlimited on ITU records coverage 39.05 off a single partial page. There is no accuracy gate anywhere yet, which is why changing the raster default is a tracked follow-up rather than something this run licenses.

Cross-host reading is forbidden by construction: this host is an RTX 2070 8 GB and the 2026-09-28 table is an A2000 8 GB, in two different virtual environments. Compare rows within this table only.

Deferred runs

The legs below were deliberately deferred on 2026-09-28, with the reason recorded so a later run is a scheduling decision rather than a missing result. None of them failed: the measured fixtures already prove the adapters work, and the full-corpus runs only cost machine time.

Run Deferred on Reason Command
Full-corpus OCR, all four models 2026-09-28 Cost/time only: the 2-page fixtures already take 82–1024 s per document, so the corpus is many hours of exclusive GPU time. Not an install conflict — the extras are jointly installable on one transformers window Full-corpus command
Full-corpus ocr-qianfan 2026-09-28 Cost/time. The original ≈ 145 s/page estimate is revised upward by the 2026-10-31 five-way run: 689–939 s per 2-page document at the shipped 300 dpi raster is 345–470 s/page, i.e. ≈ 47–64 h for NIST SP 800-53r5 alone, not the ≈ 20 h first written here. Weights are staged and sha-verified (20 GB cache), so the run is offline-ready same command (qianfan is one of the measured backends)
Full-corpus OCR, all four parsecraft[ocr] models (re-affirmed) 2026-10-31 Cost/time only, and now priced by measurement rather than estimate: the five-way run on this date produced 345–470 s/page for the one model that survives 300 dpi, and hours of exclusive GPU time per corpus file. A GPU environment (.venv-gpu) did exist that day — the full corpus was deferred for cost, not for lack of hardware same OCR command above
Full-corpus docling (re-affirmed) 2026-10-31 Runtime budget — unchanged from 2026-09-28; not an automated check same docling command above
PyMuPDF extraction on the 2026-10-31 corpus 2026-10-31 Scope/env: PyMuPDF was absent from the measured env; historical numbers stay in the 2026-09-27 snapshot In an isolated checkout, uv sync --extra web --extra pymupdf --reinstall-package parsecraft, then run the benchmark command from the recipe
GPU VRAM measurement 2026-10-31 Out of scope on this host: peak bytes are tracemalloc host-Python allocations, and WDDM lets torch.cuda.max_memory_allocated() exceed the card (9155 MiB reported on 8191 MiB; a synthetic loop reached 12288 MiB with free = 0), so a number here would not be a VRAM measurement On Linux: per-cell reset_peak_memory_stats() plus an nvidia-smi cross-check
Full-corpus docling 2026-09-28 Runtime budget: docling is ≈ 8–15 s/page warm on this laptop → ≈ 1.8 h for NIST SP 800-53r5 alone, and the incident shape (>1 h on 113 pp) says hours. Not an automated check Full-corpus docling
Quantised variants (int8 / 4-bit) 2026-09-28 Out of scope by design: quantisation is a backend-installation choice, not a harness dimension —

Full-corpus OCR command

All four OCR extras share one transformers>=5.17,<6 window, so the full run is a single pass in a single environment. The command prints JSON on stdout and never overwrites the committed CPU report; PDF rasterisation uses base pypdfium2, so no extra is needed.

parsecraft benchmark itu-t-p863.pdf etsi-ts-103558.pdf nist-sp-800-53r5.pdf \
  --json > benchmark-ocr-full.json

Full-corpus docling command

parsecraft benchmark *.pdf --json > benchmark-docling-full.json

The committed bounded runs stay the evidence for adapter correctness: benchmark-ocr-pass-a.json and benchmark-ocr-pass-b.json.

Cross-check: a 113-page PDF on Docling

Docling ran for >1 hour on a 113-page ITU-T P.863 PDF at full CPU/RAM and was cancelled (≈ >31.9 s/page, never finished). All figures in this section are the 2026-09-27 snapshot (PyMuPDF measured while installed), not the 2026-10-31 run.

Measured here on the staged ITU-T P.863 artifact (80 pages, same document family — the staged file is not byte-identical to the incident's 113-page bundle): pymupdf extracts 80 pages in 0.612 s (≈ 7.7 ms/page), and the 492-page NIST SP 800-53r5 in 4.00 s (122.9 pages/s). That is more than three orders of magnitude below the incident's per-page cost — with honest caveats: this is text-layer extraction into typed chunks, not Docling's full layout/table pipeline. The routing design leans exactly on this gap: run the cheap native pass first, escalate to OCR/layout backends only for pages whose signals say native extraction cannot do the job (the cold-cache budget for a whole document, escalations included, is 15 minutes).

Docling vs pymupdf (bounded head-to-head, 2026-09-28)

The gap the incident cross-check only inferred is now measured on 2-page fixtures of the three corpus PDFs (docling 2.130.0, 2026-09-28). Environment: Windows laptop, CPython 3.13, CPU only. Docling ran its default pipeline untuned (layout, tables, and its bundled RapidOCR); pymupdf used PyMuPDF. Both were measured with parsecraft benchmark.

fixture (2 pp) docling pymupdf ratio
itu-t-p863 (of 80) 20.6 s 0.014 s ≈ 1470×
etsi-ts-103558 (of 68) 30.5 s 2.98 s ≈ 10×
nist-sp-800-53r5 (of 492) 15.5 s 0.024 s ≈ 650×
  • Docling's first conversion in a process is far slower — 191.9 s for the p863 fixture — because it loads its layout/OCR models. The table is warm; its peak Python memory is 158–405 MB against 0.03–15 MB for pymupdf.
  • Docling emits more structure (headings, tables) at comparable text coverage; the cost of that is runtime and memory, not correctness.
  • Ratios vary per document (the ETSI fixture is the most docling-favourable of the set) — read the per-row numbers, not a single ratio.

This is an on-demand measurement, not an automated check. Cut two-page fixtures from the corpus PDFs and benchmark them:

parsecraft benchmark fixtures/*.pdf --json > benchmark-docling-vs-native.json

Honest gaps

  • Full-corpus OCR was deferred, not skipped. The GPU legs cover 2-page fixtures only (see GPU OCR legs); scaling any OCR number here to a whole document would be fabricated. The dated deferral and exact commands are in Deferred runs. The pinned licensed PDFs are hand-staged and never downloaded.
  • Full-corpus qianfan is a dated deferral — see Deferred runs, which carries the current 345–470 s/page rate from the five-way run (≈ 47–64 h for NIST alone). All four models are staged and sha-verified (20 GB cache), so the deferred run passes offline asset checks.
  • VRAM is not measured here, and timings carry tracing overhead. Peak bytes are tracemalloc (host Python allocations); GPU VRAM and native allocations are out of scope. Elapsed times include the runner's tracing overhead — compare backends within one run, not against wall-clock expectations.
  • No quantisation variants were benchmarked — model quantisation stays a backend-installation concern, not a harness dimension.
  • wikidata-q42.json was skipped in this run (unsupported source suffix, recorded in the report, not hidden). exoplanets.csv was a skip in the 2026-09-27 snapshot and has been measured since 2026-10-31 (native-csv, one TABLE chunk, coverage 1.212 — an unbounded ratio, see How to read coverage).
  • HTML coverage is a structure proxy. native-html emits headings, paragraphs, lists, tables, code, and quotes; raw inline markup dominates the analyzer's character count, so 0.11–0.27 coverage is typical for markup-heavy pages; whether anything is lost is unjudged.
  • itu-t-p863.pdf showed selected: - in the 2026-09-27 report: its page signals routed some pages to OCR intents while no OCR backend was installed in this environment, so auto-mode planning correctly declined (selected is null) — the direct per-backend measurement still ran.

Artifacts

file contents
benchmark.md 2026-10-31 harness Markdown (a546d73, 17 rows)
benchmark.json 2026-10-31 harness JSON, schema-guarded
benchmark-2026-09-27.md 2026-09-27 harness Markdown, preserved verbatim
benchmark-2026-09-27.json 2026-09-27 harness JSON, preserved verbatim
benchmark-ocr-pass-a.json GPU pass A (ovis + qianfan, transformers 5.17.0), 12 rows — schema-guarded
benchmark-ocr-pass-b.json GPU pass B (tele + unlimited, transformers 4.57.1), 15 rows — schema-guarded
benchmark-docling-vs-native.json docling vs pymupdf, 2-page fixtures, 6 rows
benchmark-gpu-fiveway-2026-10-31.md 2026-10-31 five-way GPU Markdown (9cbb645, 24 rows), schema-guarded with a writer round-trip
benchmark-gpu-fiveway-2026-10-31.json the same five-way JSON