Utility Commands¶
File system access, spec downloading, and database diagnostics.
TDoc Access¶
open¶
Download (if needed), extract, and open the main document of a TDoc with the
system default application. The main file is picked from the archive by the
configured selector (heuristic by default; see checkout.selector in
Configuration) and can be overridden per TDoc with
main-file.
3gpp-crawler tdoc open S4-250638
3gpp-crawler tdoc open S4-250638 --selector decision_model # one-off strategy
--selector accepts heuristic, decision_model, or manual and overrides
checkout.selector for that run only. Decision-model selection additionally
requires an API key (checkout.systemone_api_key / TDC_CHECKOUT_SYSTEMONE_API_KEY
or TYPESAFE_API_KEY); without one, tdoc open logs and falls back to the
heuristic. The resulting source is printed (main file: heuristic|decision_model|manual_override).
checkout¶
Download and extract TDoc(s) to the checkout folder for offline access.
3gpp-crawler tdoc checkout S4-250638
3gpp-crawler tdoc checkout S4-250638 S4-250639
3gpp-crawler tdoc checkout S4-250638 --keep-zip # keep the downloaded zip next to the extracted files
A TDoc that is not in the local database is looked up by ID first through the
public portal download endpoint: that yields the file URL without credentials,
so the checkout (or open) works on a cold database. Credentialed portal
metadata and WhatTheSpec remain the fallbacks. The parent meeting is resolved
from the download URL, and the meetings of the TDoc's subgroup are fetched
once when they are missing locally, so fetched records are stored with their
meeting linkage instead of as orphans.
A TDoc without any downloadable document (WhatTheSpec knows the ID but returns
an empty link, e.g. a numbering gap or a meeting that is not published yet)
fails immediately with no download URL; no checkout folder is created.
The command exits with code 1 when any TDoc fails, after printing every failure reason. Withdrawn TDocs are reported as skipped and do not fail the run.
Extraction is configurable per config file ([extract], no CLI flag or env
var): flatten_single_root = true (default) moves the contents of a single
top-level wrapper folder into the TDoc folder root, so an archive containing
S4-262108/S4-262108.docx is extracted as <tdoc-folder>/S4-262108.docx.
exclude holds gitignore-style
patterns that are skipped during extraction (defaults: .git/, .hg/,
.svn/); excluded members are never written, and the built-in clutter rules
(__MACOSX/, .DS_Store, …) keep applying. See
Configuration.
files¶
List the files inside a TDoc's zip archive. The first call reads the listing from the server (HTTP range reads, no full download) and stores it; later calls use the stored listing. --refresh re-reads it.
3gpp-crawler tdoc files S4-250638
3gpp-crawler tdoc files S4-250638 --refresh
main-file¶
Manage manual main-document overrides. An override is a stored archive member
path and takes precedence over the configured selector when opening a TDoc.
Use the exact member path printed by tdoc files; the value must be relative
to the archive root (.. and absolute paths are rejected), and tdoc open
fails loudly if the stored member is no longer inside the archive.
tdoc open also records the resulting classification in the database —
member path, source (manual_override, decision_model, heuristic),
selector, confidence — so a later run can replay it. show displays both
the override and that stored record.
3gpp-crawler tdoc main-file set S4-250638 "Cover sheet.pdf" --note "real contribution"
3gpp-crawler tdoc main-file show S4-250638 # override + stored classification
3gpp-crawler tdoc main-file list # every override
3gpp-crawler tdoc main-file clear S4-250638 # remove the override
Setting checkout.selector = "manual" (or TDC_CHECKOUT_SELECTOR=manual)
disables the heuristic and the decision model, so only overrides are used.
main-file reclassify¶
Re-run classification for TDocs that are already checked out, without downloading anything. Use this after changing the selector, configuring the decision model, or upgrading the heuristic.
3gpp-crawler tdoc main-file reclassify S4-250638 S4-250639
3gpp-crawler tdoc main-file reclassify --all
3gpp-crawler tdoc main-file reclassify --all --selector decision_model
# Overwrite manual choices too (deletes those overrides):
3gpp-crawler tdoc main-file reclassify --all --include-manual --confirm-overwrite
TDocs without a stored classification or without a checkout are reported as
missing; manual choices are skipped unless --include-manual is given.
The decision model works best when configured once in the global config
(~/.config/3gpp-crawler/config.toml); see
Configuration.
Specifications¶
open¶
Download and open the latest document for a specification.
3gpp-crawler spec open 23.501
3gpp-crawler spec open 23.501 --release 17
checkout¶
Batch download specification documents to the checkout folder.
Features:
- Progress bar for multi-spec operations.
- Range syntax support (e.g.
26130-26140). --releaseselector (latestor e.g.18.0).
3gpp-crawler spec checkout 26130-26140
3gpp-crawler spec checkout 26260 --release 18.0
files¶
List the files inside a specification's zip archive. Stored from checkout; if nothing is stored yet, the listing is read from the server (HTTP range reads) and kept.
3gpp-crawler spec files 26260 --release 18.0
3gpp-crawler spec files 26260 --refresh
Diagnostics¶
stats¶
Display high-level database statistics.
3gpp-crawler tdoc stats
config¶
Manage configuration — see the Configuration guide for
init, show, validate, and docs subcommands.