The Python API mirrors the CLI: the same core-library functions back both.
Package overview¶
tdoc_crawler ¶
TDoc Crawler - CLI tool for querying structured 3GPP TDoc data.
TDoc operations¶
Main-file selection, open, and re-classification are shared by CLI and MCP.
operations ¶
TDoc operations.
CheckoutResult
dataclass
¶
CheckoutResult(success_count: int, error_count: int, errors: list[str], archives: list[ArchiveRecord] = list(), skipped_count: int = 0)
FetchMethod ¶
FetchMissingResult ¶
FetchMissingResult(refreshed: list[TDocMetadata], fetch_result: TDocCrawlResult | None = None, used_whatthespec_fallback: bool = False)
Result from fetch_missing_tdocs orchestration.
Parameters:
-
(refreshed¶list[TDocMetadata]) –Updated list of TDoc metadata after fetching
-
(fetch_result¶TDocCrawlResult | None, default:None) –Result of the batch fetch operation
-
(used_whatthespec_fallback¶bool, default:False) –Whether WhatTheSpec was used as fallback
Source code in src/tdoc_crawler/tdocs/operations/fetch.py
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 | |
FieldChange
dataclass
¶
FieldChange(field: str, old: str, new: str)
One metadata field that changed during a refresh.
MeetingPatternError ¶
Bases: ValueError
Raised when a meeting filter pattern matches no locally known meeting.
MeetingRefreshSummary
dataclass
¶
MeetingRefreshSummary(meeting_id: int, short_name: str, tdocs_before: int, tdocs_after: int, added: tuple[str, ...], changed: tuple[TDocChange, ...], unchanged: int, absent_from_list: tuple[str, ...], errors: tuple[str, ...])
Changes found for one meeting in the refresh window.
ReclassifyOutcome
dataclass
¶
ReclassifyOutcome(tdoc_id: str, status: ReclassifyStatus, message: str = '', selection: MainFileSelection | None = None)
RefreshResult
dataclass
¶
RefreshResult(lookback_days: int, start_date: date, meetings_scanned: int, meetings_with_changes: int, tdocs_processed: int, tdocs_added: int, tdocs_changed: int, tdocs_unchanged: int, urls_repaired: int, meetings: tuple[MeetingRefreshSummary, ...], errors: tuple[str, ...])
TDocChange
dataclass
¶
TDocChange(tdoc_id: str, url_repaired: bool = False, field_changes: tuple[FieldChange, ...] = ())
Change report for a single TDoc.
TDocCrawlResult
dataclass
¶
TDocCrawlResult(processed: int, inserted: int, updated: int, errors: list[str])
TDocCrawler ¶
TDocCrawler(database: TDocDatabase)
Configuration-driven crawler orchestrating subinterpreter workers.
Source code in src/tdoc_crawler/tdocs/operations/crawl.py
43 44 45 46 47 | |
crawl
async
¶
crawl(config: TDocCrawlConfig, progress_callback: Callable[[float, float], None] | None = None) -> TDocCrawlResult
Execute a crawl using the provided configuration.
Parameters:
-
(config¶TDocCrawlConfig) –Configuration for the TDoc crawl
-
(progress_callback¶Callable[[float, float], None] | None, default:None) –Optional callback for progress updates
Returns:
-
TDocCrawlResult–Summary of crawl results
Source code in src/tdoc_crawler/tdocs/operations/crawl.py
49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 | |
get_meetings_to_crawl
async
¶
get_meetings_to_crawl(config: TDocCrawlConfig) -> list[MeetingMetadata]
Query meetings from the database based on configuration filters.
Source code in src/tdoc_crawler/tdocs/operations/crawl.py
101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 | |
TDocNotFoundError ¶
TDocOpenResult
dataclass
¶
TDocOpenResult(metadata: TDocMetadata, selection: MainFileSelection, fetch_errors: tuple[str, ...] = ())
checkout_meeting_tdocs ¶
checkout_meeting_tdocs(meetings: list[MeetingMetadata], checkout_dir: Path, http_cache_file: Path, session: Session | None = None, cache_manager_name: str | None = None, extract: ExtractConfig | None = None) -> CheckoutResult
Checkout TDoc files from a list of meetings.
Parameters:
-
(meetings¶list[MeetingMetadata]) –List of MeetingMetadata to checkout TDocs from
-
(checkout_dir¶Path) –Base checkout directory
-
(http_cache_file¶Path) –Path to HTTP cache database
-
(session¶Session | None, default:None) –Optional requests.Session to reuse for downloads
-
(cache_manager_name¶str | None, default:None) –Optional cache manager name for HTTP caching
-
(extract¶ExtractConfig | None, default:None) –Extraction settings; defaults to
ExtractConfig()
Returns:
-
CheckoutResult–CheckoutResult with success/error counts
Source code in src/tdoc_crawler/tdocs/operations/checkout.py
295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 | |
checkout_tdoc ¶
checkout_tdoc(metadata: TDocMetadata, checkout_dir: Path, *, force: bool = False, session: Session | None = None, keep_zip: bool = False, archive_sink: list[ArchiveRecord] | None = None, extract: ExtractConfig | None = None) -> Path
Download and extract a TDoc to the checkout folder.
Parameters:
-
(metadata¶TDocMetadata) –TDoc metadata containing URL and ID
-
(checkout_dir¶Path) –Base checkout directory
-
(force¶bool, default:False) –If True, re-download even if already exists
-
(session¶Session | None, default:None) –Optional requests.Session to reuse for downloads
-
(keep_zip¶bool, default:False) –If True, keep the downloaded zip in the checkout folder
-
(archive_sink¶list[ArchiveRecord] | None, default:None) –If given, the zip member listing is appended to it when a download happens (an already checked-out TDoc records nothing)
-
(extract¶ExtractConfig | None, default:None) –Extraction settings (flattening, skip patterns); defaults to
ExtractConfig()
Returns:
-
Path–Path to the extracted TDoc directory
Raises:
-
FileNotFoundError–If download fails or zip is empty
Source code in src/tdoc_crawler/tdocs/operations/checkout.py
481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 | |
checkout_tdocs ¶
checkout_tdocs(results: list[TDocMetadata], checkout_dir: Path, force: bool = False, session: Session | None = None, cache_manager_name: str | None = None, keep_zip: bool = False, extract: ExtractConfig | None = None, workers: int = 1) -> CheckoutResult
Checkout multiple TDoc files to the checkout directory.
Parameters:
-
(results¶list[TDocMetadata]) –List of TDocMetadata to checkout
-
(checkout_dir¶Path) –Base checkout directory
-
(force¶bool, default:False) –If True, re-download even if already exists
-
(session¶Session | None, default:None) –Optional requests.Session to reuse for downloads (sequential mode only; parallel mode uses per-download sessions for thread safety)
-
(cache_manager_name¶str | None, default:None) –Optional cache manager name for HTTP caching
-
(keep_zip¶bool, default:False) –If True, keep each downloaded zip next to its extracted files
-
(extract¶ExtractConfig | None, default:None) –Extraction settings; defaults to
ExtractConfig() -
(workers¶int, default:1) –Number of parallel downloads. 1 (default) keeps sequential behavior; >1 downloads concurrently while preserving input order.
Returns:
-
CheckoutResult–CheckoutResult with success/error counts and the archive listings
-
CheckoutResult–recorded during this run (persist them with
save_archive_records)
Source code in src/tdoc_crawler/tdocs/operations/checkout.py
379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 | |
clear_checkout_tdocs ¶
clear_checkout_tdocs(checkout_dir: Path, *, keep_specs: bool = True) -> int
Delete downloaded TDoc files from the checkout directory.
Destructive by design and never called automatically: only the explicit
--purge-checkouts switch on database clear-* reaches this helper.
Parameters:
-
(checkout_dir¶Path) –Base checkout directory
-
(keep_specs¶bool, default:True) –Preserve the
Specssubtree (TDoc-only clears)
Returns:
-
int–Number of top-level entries removed
Source code in src/tdoc_crawler/tdocs/operations/checkout.py
268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 | |
create_source ¶
create_source(method: FetchMethod, config: TDocSourceConfig, session: Session | None = None) -> WhatTheSpecSource | PortalSource | DocumentListSource
Create TDoc source based on fetch method.
Parameters:
-
(method¶FetchMethod) –Fetch method to use
-
(config¶TDocSourceConfig) –Source configuration
-
(session¶Session | None, default:None) –Optional requests.Session to reuse
Returns:
-
WhatTheSpecSource | PortalSource | DocumentListSource–Configured TDoc source instance
Raises:
-
ValueError–If method requires credentials but none provided
Source code in src/tdoc_crawler/tdocs/operations/fetch.py
257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 | |
diff_tdoc ¶
diff_tdoc(before: TDocMetadata, after: TDocMetadata) -> tuple[FieldChange, ...]
Return the meaningful metadata fields that differ between two records.
Source code in src/tdoc_crawler/tdocs/operations/refresh.py
170 171 172 173 174 175 176 177 178 | |
fetch_missing_tdocs
async
¶
fetch_missing_tdocs(database: TDocDatabase, config: TDocQueryConfig, results: list[TDocMetadata], full_metadata: bool = False, use_whatthespec: bool = False, session: Session | None = None, http_cache_file: Path | None = None, db_file: Path | None = None, url_only_first: bool = False) -> FetchMissingResult
Fetch missing TDocs if any are requested but not found in database.
Orchestrates fetching based on flags, uses WhatTheSpec fallback when needed.
Parameters:
-
(database¶TDocDatabase) –Database instance
-
(config¶TDocQueryConfig) –TDocQueryConfig with requested TDoc IDs
-
(results¶list[TDocMetadata]) –Current query results
-
(full_metadata¶bool, default:False) –If True, fetch full metadata from portal
-
(use_whatthespec¶bool, default:False) –If True, always use WhatTheSpec
-
(session¶Session | None, default:None) –Optional requests.Session to reuse
-
(http_cache_file¶Path | None, default:None) –Optional path to HTTP cache database file
-
(db_file¶Path | None, default:None) –Optional path to metadata database file
-
(url_only_first¶bool, default:False) –If True, try the public URL-only portal lookup first (checkout and open need a download URL, not full metadata)
Returns:
-
FetchMissingResult–FetchMissingResult with updated results and fetch statistics
Source code in src/tdoc_crawler/tdocs/operations/fetch.py
289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 | |
fetch_missing_tdocs_batch
async
¶
fetch_missing_tdocs_batch(database: TDocDatabase, missing_ids: list[str], session: Session | None = None, timeout: int = 30, http_config: HttpConfig | None = None, http_cache_file: Path | None = None) -> TDocCrawlResult
Fetch multiple missing TDocs using portal authentication.
Parameters:
-
(database¶TDocDatabase) –Database instance for storing results
-
(missing_ids¶list[str]) –List of TDoc IDs to fetch
-
(http_config¶HttpConfig | None, default:None) –HTTP configuration from ThreeGPPConfig
-
(timeout¶int, default:30) –Request timeout in seconds
-
(session¶Session | None, default:None) –Optional requests.Session to reuse
-
(http_cache_file¶Path | None, default:None) –Optional path to HTTP cache database file
Returns:
-
TDocCrawlResult–TDocCrawlResult with fetch statistics
Source code in src/tdoc_crawler/tdocs/operations/fetch.py
542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 | |
fetch_via_portal_url_batch
async
¶
fetch_via_portal_url_batch(database: TDocDatabase, missing_ids: list[str], *, session: Session | None = None, timeout: int = 30, http_cache_file: Path | None = None) -> TDocCrawlResult
Resolve download URLs for missing TDocs without credentials.
The portal's DownloadTDoc endpoint is public: it redirects to the
direct file URL, which is all a checkout needs. Records created this way
carry the URL and no bibliographic metadata; a later tdoc crawl (or a
credentialed fetch) fills the rest in.
Parameters:
-
(database¶TDocDatabase) –Database instance for storing results
-
(missing_ids¶list[str]) –TDoc IDs to resolve
-
(session¶Session | None, default:None) –Optional requests.Session to reuse (left open)
-
(timeout¶int, default:30) –Request timeout in seconds
-
(http_cache_file¶Path | None, default:None) –Optional path to HTTP cache database file
Returns:
-
TDocCrawlResult–TDocCrawlResult with fetch statistics
Source code in src/tdoc_crawler/tdocs/operations/fetch.py
469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 | |
fetch_via_whatthespec_batch
async
¶
fetch_via_whatthespec_batch(database: TDocDatabase, missing_ids: list[str], http_config: HttpConfig | None = None, session: Session | None = None, http_cache_file: Path | None = None, db_file: Path | None = None) -> TDocCrawlResult
Fetch multiple missing TDocs using WhatTheSpec.net API.
Parameters:
-
(database¶TDocDatabase) –Database instance for storing results
-
(missing_ids¶list[str]) –List of TDoc IDs to fetch
-
(http_config¶HttpConfig | None, default:None) –HTTP configuration from ThreeGPPConfig
-
(session¶Session | None, default:None) –Optional requests.Session to reuse
-
(http_cache_file¶Path | None, default:None) –Optional path to HTTP cache database file
-
(db_file¶Path | None, default:None) –Optional path to metadata database file
Returns:
-
TDocCrawlResult–TDocCrawlResult with fetch statistics
Source code in src/tdoc_crawler/tdocs/operations/fetch.py
620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 | |
filter_meetings_by_patterns ¶
filter_meetings_by_patterns(meetings: list[MeetingMetadata], patterns: list[str] | None) -> list[MeetingMetadata]
Filter meetings by short-name patterns, normalized names, or numeric IDs.
Patterns accept the portal spelling (SA4#138), the normalized database
spelling (S4-138), fnmatch wildcards (SA4#13*), and plain
numeric meeting IDs. A leading 3GPP on either side is ignored, so
SA4#134 matches a meeting stored as 3GPPSA4#134. Every pattern
must match at least one meeting — silently crawling nothing is worse
than failing loudly.
Source code in src/tdoc_crawler/tdocs/operations/crawl.py
489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 | |
get_checked_out_tdocs ¶
get_checked_out_tdocs(checkout_dir: Path) -> list[str]
Get list of TDoc IDs that are already checked out.
Parameters:
Returns:
Source code in src/tdoc_crawler/tdocs/operations/checkout.py
244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 | |
get_checkout_path ¶
get_checkout_path(metadata: TDocMetadata, checkout_dir: Path) -> Path
Calculate the checkout path for a TDoc based on its URL.
The checkout path mirrors the 3GPP server directory structure.
Parameters:
-
(metadata¶TDocMetadata) –TDoc metadata containing the URL
-
(checkout_dir¶Path) –Base checkout directory
Returns:
-
Path–Path to the checkout directory for this TDoc
Source code in src/tdoc_crawler/tdocs/operations/checkout.py
577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 | |
get_meeting_checkout_folder ¶
get_meeting_checkout_folder(meeting: MeetingMetadata, checkout_dir: Path) -> Path
Calculate the checkout folder for a meeting based on its files_url.
The folder mirrors the 3GPP server directory structure.
Parameters:
-
(meeting¶MeetingMetadata) –MeetingMetadata containing the files_url
-
(checkout_dir¶Path) –Base checkout directory
Returns:
-
Path–Path to the meeting checkout folder
Source code in src/tdoc_crawler/tdocs/operations/checkout.py
668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 | |
has_revision_marks ¶
has_revision_marks(path: Path) -> bool
True when an OOXML document carries tracked changes (revision marks).
Scans the word/ XML parts for insert/delete/move revision elements.
Non-OOXML files (PDF, legacy .doc/.xls) cannot be checked
cheaply and count as unmarked; unreadable files likewise.
This is the only signal used to prefer a tracked-changes version
over a clean one — filename markers (_clean, _rm, ...) are
inconsistent across authors and are never consulted.
Source code in src/tdoc_crawler/tdocs/operations/main_file.py
192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 | |
member_candidates ¶
member_candidates(extract_dir: Path) -> list[Path]
Non-clutter files of an extracted TDoc directory, sorted by path.
Source code in src/tdoc_crawler/tdocs/operations/main_file.py
278 279 280 281 282 283 | |
prepare_tdoc_file ¶
prepare_tdoc_file(metadata: TDocMetadata, checkout_dir: Path, return_dir: bool = False, session: Session | None = None, *, override_member: str | None = None, config: CheckoutConfig | None = None, extract: ExtractConfig | None = None) -> Path
Prepare the local file for a TDoc, downloading/extracting if needed.
Routes through :func:checkout_tdoc so that tdoc open and
tdoc checkout share a single business-logic path: the on-disk
layout mirrors the 3GPP server (under
tsg_<tsg>/<wg>/<meeting>/Docs/<tdoc_id>); archive clutter is
filtered by the user-editable clutter-ignore patterns.
Main-file choice is delegated to
:func:tdoc_crawler.tdocs.operations.main_file.resolve_main_file:
manual override first, then the optional decision model, then the
offline heuristic. Call :func:prepare_tdoc_selection instead when
the source of the choice matters to the caller.
Parameters:
-
(metadata¶TDocMetadata) –TDoc metadata with download URL.
-
(checkout_dir¶Path) –Checkout directory (base of the mirror tree).
-
(return_dir¶bool, default:False) –When True, return the extract directory itself instead of the picked "main file" inside it.
-
(session¶Session | None, default:None) –Optional requests.Session to reuse for downloads.
-
(override_member¶str | None, default:None) –Stored manual override (archive-relative path).
-
(config¶CheckoutConfig | None, default:None) –Checkout settings; defaults to
CheckoutConfig(). -
(extract¶ExtractConfig | None, default:None) –Extraction settings; defaults to
ExtractConfig().
Returns:
-
Path–Path to the picked main file, or the extract directory when
-
Path–return_diris True. For non-zip URLs, returns the downloaded -
Path–file path.
Raises:
-
FileNotFoundError–If the TDoc is withdrawn, the download fails, or the archive contains no usable file.
-
ValueError–If
metadata.urlis missing.
Source code in src/tdoc_crawler/tdocs/operations/checkout.py
68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 | |
prepare_tdoc_selection ¶
prepare_tdoc_selection(metadata: TDocMetadata, checkout_dir: Path, *, override_member: str | None = None, config: CheckoutConfig | None = None, session: Session | None = None, extract: ExtractConfig | None = None) -> MainFileSelection
Prepare a TDoc and report which file was chosen as its main file.
Same download/extract path as :func:prepare_tdoc_file; use this
variant when the caller must show or log the selection source
(manual override, decision model, or heuristic).
Parameters:
-
(metadata¶TDocMetadata) –TDoc metadata with download URL.
-
(checkout_dir¶Path) –Checkout directory (base of the mirror tree).
-
(override_member¶str | None, default:None) –Stored manual override (archive-relative path).
-
(config¶CheckoutConfig | None, default:None) –Checkout settings; defaults to
CheckoutConfig(). -
(session¶Session | None, default:None) –Optional requests.Session to reuse for downloads.
-
(extract¶ExtractConfig | None, default:None) –Extraction settings; defaults to
ExtractConfig().
Returns:
-
MainFileSelection–The resolved main file plus its source.
Raises:
-
FileNotFoundError–If the TDoc is withdrawn, the download fails, or the archive contains no usable (or no overridden) file.
-
ValueError–If
metadata.urlis missing.
Source code in src/tdoc_crawler/tdocs/operations/checkout.py
128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 | |
reclassify_tdocs
async
¶
reclassify_tdocs(*, db_file: Path, checkout_dir: Path, checkout_config: CheckoutConfig, tdoc_ids: list[str] | None = None, include_manual: bool = False) -> list[ReclassifyOutcome]
Re-run main-file classification on stored TDocs with a checkout.
Parameters:
-
(db_file¶Path) –Path to the metadata database.
-
(checkout_dir¶Path) –Root directory containing extracted TDoc archives.
-
(checkout_config¶CheckoutConfig) –Main-document selection settings.
-
(tdoc_ids¶list[str] | None, default:None) –Specific TDocs to re-classify; None means every stored classification.
-
(include_manual¶bool, default:False) –Also re-classify TDocs with a manual selector or a stored override, deleting that override. Destructive.
Returns:
-
list[ReclassifyOutcome]–One outcome per requested TDoc, in request order.
Source code in src/tdoc_crawler/tdocs/operations/reclassify.py
43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 | |
refresh_tdocs
async
¶
refresh_tdocs(database: TDocDatabase, *, lookback_days: int = 14, working_groups: list[WorkingGroup] | None = None, subgroups: list[str] | None = None, workers: int = 4, timeout: int = 30, max_retries: int = 3, repair_urls: bool = True, resolve_url: UrlResolver = _resolve_corrected_url, progress_callback: Callable[[float, float], None] | None = None, today: date | None = None) -> RefreshResult
Re-fetch complete metadata for TDocs of meetings in the lookback window.
Parameters:
-
(database¶TDocDatabase) –Open database used for snapshots and upserts.
-
(lookback_days¶int, default:14) –Include meetings that started within this many days.
-
(working_groups¶list[WorkingGroup] | None, default:None) –Restrict to these groups (default: all RAN/SA/CT).
-
(subgroups¶list[str] | None, default:None) –Optional sub-working-group filter.
-
(workers¶int, default:4) –Parallel workers for the document-list crawl and URL repair.
-
(timeout¶int, default:30) –Per-request HTTP timeout in seconds.
-
(max_retries¶int, default:3) –Retry attempts for failed meetings.
-
(repair_urls¶bool, default:True) –Resolve and persist corrected download URLs.
-
(resolve_url¶UrlResolver, default:_resolve_corrected_url) –URL resolver (injectable for tests).
-
(progress_callback¶Callable[[float, float], None] | None, default:None) –Optional (completed, total) crawl progress callback.
-
(today¶date | None, default:None) –Override for
date.today()in tests.
Returns:
-
A(RefreshResult) –class:
RefreshResultwith per-meeting change details.
Source code in src/tdoc_crawler/tdocs/operations/refresh.py
207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 | |
resolve_main_file ¶
resolve_main_file(extract_dir: Path, tdoc_id: str, *, title: str | None = None, override_member: str | None = None, config: CheckoutConfig | None = None, session: Session | None = None) -> MainFileSelection
Resolve the main file of an extracted TDoc archive.
Precedence: manual override > decision model (when configured and confident) > heuristic.
Parameters:
-
(extract_dir¶Path) –Directory holding the extracted TDoc contents.
-
(tdoc_id¶str) –TDoc identifier, used by the heuristic and the model.
-
(title¶str | None, default:None) –Optional portal title, used by the heuristic and the model.
-
(override_member¶str | None, default:None) –Stored manual override (archive-relative path).
-
(config¶CheckoutConfig | None, default:None) –Checkout settings; defaults to
CheckoutConfig(). -
(session¶Session | None, default:None) –Optional HTTP session reused for the decision-model call.
Returns:
-
MainFileSelection–The resolved main file together with its source.
Raises:
-
FileNotFoundError–If no usable file is present, or the override no longer exists in the archive.
Source code in src/tdoc_crawler/tdocs/operations/main_file.py
59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 | |
resolve_tdoc_main_file
async
¶
resolve_tdoc_main_file(tdoc_id: str, *, db_file: Path, checkout_dir: Path, checkout_config: CheckoutConfig, full_metadata: bool = False, use_whatthespec: bool = False, session: Session | None = None, extract_config: ExtractConfig | None = None) -> TDocOpenResult
Ensure a TDoc is stored, extract it, and resolve its main document.
Parameters:
-
(tdoc_id¶str) –TDoc identifier (normalized before lookup).
-
(db_file¶Path) –Path to the metadata database.
-
(checkout_dir¶Path) –Root directory for TDoc archives.
-
(checkout_config¶CheckoutConfig) –Main-document selection settings.
-
(full_metadata¶bool, default:False) –If True, fetch full metadata from the portal.
-
(use_whatthespec¶bool, default:False) –If True, always use WhatTheSpec for missing TDocs.
-
(session¶Session | None, default:None) –Optional HTTP session to reuse.
-
(extract_config¶ExtractConfig | None, default:None) –Extraction settings; defaults to
ExtractConfig().
Returns:
-
TDocOpenResult–The TDoc metadata and the resolved main-document selection.
Raises:
-
TDocNotFoundError–If the TDoc cannot be found or fetched.
-
FileNotFoundError–If the stored override or the archive is invalid.
-
OSError–If the archive cannot be written.
-
ValueError–If the TDoc metadata is unusable.
Source code in src/tdoc_crawler/tdocs/operations/open.py
40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 | |
select_heuristic_main_file ¶
select_heuristic_main_file(extract_dir: Path, tdoc_id: str, *, title: str | None = None) -> Path
Pick the most likely "main" file inside an extracted TDoc directory.
Candidates are filtered by the gitignore-style clutter patterns
(<cache-dir>/clutter-ignore, see :mod:tdoc_crawler.utils.clutter)
so that OS metadata (__MACOSX/, ._*, ~$* ...) is never
picked — even from checkout directories created before extraction
filtering existed.
When an archive holds two versions of the same document (one with,
one without revision marks), the tracked-changes version wins —
decided by inspecting the document content via
:func:has_revision_marks, never by filename conventions.
Among the remaining document-suffix files the best-ranked candidate wins, by these criteria in order:
1. Name is exactly the TDoc ID (e.g. ``S4-261599.docx``).
2. Highest similarity to ``<tdoc_id> <title>`` (only when *title*
is given; scores below the threshold count as zero).
3. Name starts with the TDoc ID (case-insensitive).
4. Document carries revision marks (content probe, ``.docx`` only).
5. Document type, in ``DocumentType`` declaration order.
6. Most recent file timestamp, then name (deterministic tie-break).
Fallbacks: largest remaining document-suffix file, then largest non-clutter file of any type.
Parameters:
-
(extract_dir¶Path) –Directory holding the extracted TDoc contents.
-
(tdoc_id¶str) –TDoc identifier (e.g.
S4-261599). -
(title¶str | None, default:None) –Optional portal title of the TDoc, used for fuzzy matching.
Returns:
-
Path–Path to the selected main file.
Raises:
-
FileNotFoundError–If no usable file is present.
Source code in src/tdoc_crawler/tdocs/operations/main_file.py
120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 | |
Main-file overrides¶
Stored overrides (user-owned) and classification records (computed memory).
main_files ¶
Per-TDoc manual main-file overrides.
An override records which archive member the user considers the main document of a TDoc. It always wins over automatic selection (heuristic ranking or the decision model) and survives re-download and re-extraction because it is keyed by the archive member path.
clear_main_file_override
async
¶
clear_main_file_override(tdoc_id: str) -> bool
Delete the override of one TDoc. Returns True when one existed.
Source code in src/tdoc_crawler/database/main_files.py
41 42 43 44 45 46 47 | |
clear_main_file_record
async
¶
clear_main_file_record(tdoc_id: str) -> bool
Delete the stored classification of one TDoc. Returns True when one existed.
Source code in src/tdoc_crawler/database/main_files.py
85 86 87 88 89 90 91 | |
get_main_file_override
async
¶
get_main_file_override(tdoc_id: str) -> MainFileOverride | None
Return the stored override for one TDoc, or None.
Source code in src/tdoc_crawler/database/main_files.py
50 51 52 | |
get_main_file_record
async
¶
get_main_file_record(tdoc_id: str) -> MainFileRecord | None
Return the last stored classification for one TDoc, or None.
Source code in src/tdoc_crawler/database/main_files.py
94 95 96 | |
list_main_file_overrides
async
¶
list_main_file_overrides() -> list[MainFileOverride]
All stored overrides, sorted by TDoc ID.
Source code in src/tdoc_crawler/database/main_files.py
55 56 57 58 | |
list_main_file_records
async
¶
list_main_file_records() -> list[MainFileRecord]
All stored classifications, sorted by TDoc ID.
Source code in src/tdoc_crawler/database/main_files.py
99 100 101 102 | |
save_main_file_record
async
¶
save_main_file_record(tdoc_id: str, *, member_path: str, source: str, selector: str, confidence: float | None = None) -> MainFileRecord
Create or replace the stored classification of one TDoc.
Source code in src/tdoc_crawler/database/main_files.py
61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 | |
set_main_file_override
async
¶
set_main_file_override(tdoc_id: str, member_path: str, *, note: str | None = None) -> MainFileOverride
Create or replace the manual main-file override of one TDoc.
Parameters:
-
(tdoc_id¶str) –TDoc identifier (normalized by the caller).
-
(member_path¶str) –Archive-relative POSIX path of the main file.
-
(note¶str | None, default:None) –Optional free-text note explaining the choice.
Raises:
-
ValueError–If
member_pathis empty, absolute, or escapes the archive directory.
Source code in src/tdoc_crawler/database/main_files.py
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 | |
Tag operations¶
Shared by CLI and MCP — AND-of-ORs tag expressions.
tags ¶
Tag operations for TDocs, specs, and other items.
Tags are the lightweight grouping mechanism (replacing the removed
workspace subsystem). A tag row points at an item via kind +
item_id — no foreign key; integrity is app-layer.
Filter semantics shared by CLI and MCP: a tag group is a set of tags
combined with OR (--tag a,b); repeated groups combine with AND
(--tag a,b --tag c matches items having (a or b) and c).
add_tags
async
¶
add_tags(kind: TagKind, item_id: str, tags: list[str]) -> int
Add tags to an item. Returns number of new tag rows (existing skipped).
Source code in src/tdoc_crawler/database/tags.py
26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 | |
item_ids_for_tag_groups
async
¶
item_ids_for_tag_groups(groups: list[list[str]], kind: TagKind) -> set[str]
Item ids matching the AND-of-ORs tag expression for kind.
Empty groups returns an empty set (no filter applied by callers).
Source code in src/tdoc_crawler/database/tags.py
88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 | |
list_tags
async
¶
list_tags(kind: TagKind | None = None, item_id: str | None = None, tag: str | None = None) -> list[TagEntry]
List tag rows, optionally filtered by kind, item, and/or tag.
Source code in src/tdoc_crawler/database/tags.py
73 74 75 76 77 78 79 80 81 82 83 84 85 | |
normalize_tags ¶
normalize_tags(tags: list[str]) -> list[str]
Strip, drop empties, dedupe (order-preserving).
Source code in src/tdoc_crawler/database/tags.py
61 62 63 64 65 66 67 68 69 70 | |
parse_tag_groups ¶
parse_tag_groups(values: list[str]) -> list[list[str]]
Parse CLI --tag values into OR-groups: comma-split each value.
Source code in src/tdoc_crawler/database/tags.py
21 22 23 | |
remove_tags
async
¶
remove_tags(kind: TagKind, item_id: str, tags: list[str] | None = None) -> int
Remove tags from an item (all tags if tags is None). Returns removed count.
Source code in src/tdoc_crawler/database/tags.py
48 49 50 51 52 53 54 55 56 57 58 | |
Configuration¶
settings ¶
Configuration settings for 3GPP Crawler using pydantic-settings.
This module provides the ThreeGPPConfig class with nested models for: - PathConfig: File system paths (cache, checkout) - HttpConfig: HTTP caching and request behavior - CredentialsConfig: Portal authentication credentials - CrawlConfig: Crawling behavior and filters
All paths default to sensible defaults but can be overridden via environment variables with the TDC_ prefix.
CheckoutConfig ¶
Bases: BaseSettings
Checkout behaviour: how the main file of a TDoc archive is picked.
A stored manual override always wins; selector only controls the
automatic fallback chain (heuristic ranking, or the optional System One
decision model before it).
CrawlConfig ¶
Bases: BaseSettings
Crawling behavior and filter configuration.
Controls which documents are crawled based on working groups, date ranges, and other filters.
CredentialsConfig ¶
Bases: BaseSettings
Portal authentication credentials.
Stores credentials for accessing protected resources like the 3GPP portal.
ExtractConfig ¶
Bases: BaseModel
Archive extraction behaviour for downloaded TDoc zips.
Config-file only by design: no CLI option and no environment variable is generated for these settings, because they describe how files land on disk rather than how a single run behaves.
Exclusion patterns use gitignore syntax and layer on top of the
<cache-dir>/clutter-ignore file (macOS/Windows/Office clutter).
HttpConfig ¶
Bases: BaseSettings
HTTP client and caching configuration.
Controls HTTP request behavior including timeouts, retries, and caching behavior for web requests.
PathConfig ¶
Bases: BaseSettings
File system path configuration.
Provides paths for cache directory, database, and checkout. All paths support ~ expansion and are resolved to absolute paths.
ThreeGPPConfig ¶
Bases: BaseSettings
Main configuration class for 3GPP Crawler.
Aggregates all configuration sub-models: - path: File system paths - http: HTTP client and caching behavior - credentials: Portal authentication - crawl: Crawling filters and limits - checkout: Main-file selection strategy and decision-model settings - extract: Archive extraction (flattening and skip patterns), config file only
Configuration can be overridden via environment variables with the appropriate prefixes (TDC_, TDC_EOL_, TDC_CRAWL_, TDC_CHECKOUT_).
ensure_paths ¶
ensure_paths() -> None
Ensure all configured paths exist on the filesystem.
Creates parent directories as needed.
Source code in src/tdoc_crawler/config/settings.py
341 342 343 344 345 346 347 | |
from_settings
classmethod
¶
from_settings(config_file: Path | str | None = None, cwd: Path | str | None = None) -> ThreeGPPConfig
Create a ThreeGPPConfig instance with config file and environment values.
Config precedence (highest to lowest): 1. CLI explicit config_file parameter 2. Discovered config files (later override earlier) 3. Environment variables (TDC_*) 4. Hard-coded defaults
Parameters:
-
(config_file¶Path | str | None, default:None) –Explicit path to a config file. If provided, only this file is loaded (no discovery).
-
(cwd¶Path | str | None, default:None) –Working directory for project-level config discovery. Defaults to current working directory.
Returns:
-
ThreeGPPConfig–A validated ThreeGPPConfig instance.
Source code in src/tdoc_crawler/config/settings.py
349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 | |
HTTP client¶
http_client ¶
HTTP client factory with hishel caching support.
Re-exports from session module for backward-compatible imports
from tdoc_crawler.http_client import create_cached_session from tdoc_crawler.http_client import download_to_file from tdoc_crawler.http_client import PoolConfig
HttpCacheStats
dataclass
¶
HttpCacheStats(entries_before: int, entries_after: int, expired_removed: int, evicted: int, bytes_before: int, bytes_after: int, vacuumed: bool)
PoolConfig
dataclass
¶
PoolConfig(max_connections: int = 10, max_per_host: int = 5, connection_timeout: float = 30.0, enable_retry: bool = True, retry_attempts: int = 3)
HTTP connection pool configuration.
Attributes:
-
max_connections(int) –Maximum number of connection pools to cache
-
max_per_host(int) –Maximum number of connections per host
-
connection_timeout(float) –Connection timeout in seconds
-
enable_retry(bool) –Whether to enable retry logic
-
retry_attempts(int) –Number of retry attempts for failed requests
create_cached_session ¶
create_cached_session(http_cache_file: Path | None = None, http_cache_enabled: bool | None = None, pool_config: PoolConfig | None = None, verify: bool | str | None = None, http_config: HttpConfig | None = None) -> requests.Session
Create a requests.Session with hishel caching enabled.
Parameters:
-
(http_cache_file¶Path | None, default:None) –Optional explicit path to the HTTP cache database. Falls back to PathConfig().http_cache_file when caching is enabled.
-
(http_cache_enabled¶bool | None, default:None) –Whether to enable HTTP caching. If None, uses http_config.cache_enabled or True.
-
(pool_config¶PoolConfig | None, default:None) –Optional connection pool configuration. When provided, pool settings are applied to the active adapter (cache adapter when caching is enabled, HTTPAdapter otherwise).
-
(verify¶bool | str | None, default:None) –SSL certificate verification mode. Can be bool or CA bundle path.
-
(http_config¶HttpConfig | None, default:None) –HttpConfig from ThreeGPPConfig. Controls caching TTL, retries, SSL.
Returns:
-
Session–Configured requests.Session with caching enabled (unless disabled)
Source code in src/tdoc_crawler/http_client/session.py
292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 | |
download_to_file ¶
download_to_file(url: str, target_file: Path, session: Session | None = None, http_cache_file: Path | None = None, http_cache_enabled: bool | None = None, pool_config: PoolConfig | None = None, verify: bool | str | None = None, http_config: HttpConfig | None = None) -> None
Download a file from URL to destination path.
Parameters:
-
(url¶str) –Source URL
-
(target_file¶Path) –Destination file path
-
(session¶Session | None, default:None) –Optional requests.Session to reuse. If None, a temporary cached session is created and closed.
-
(http_cache_file¶Path | None, default:None) –Optional explicit path to the HTTP cache database. Falls back to PathConfig default.
-
(http_cache_enabled¶bool | None, default:None) –Whether to enable HTTP caching. If None, defaults to http_config.cache_enabled or True.
-
(pool_config¶PoolConfig | None, default:None) –Optional connection pool configuration.
-
(verify¶bool | str | None, default:None) –SSL certificate verification mode. Can be bool or CA bundle path.
-
(http_config¶HttpConfig | None, default:None) –HttpConfig from ThreeGPPConfig. Controls caching, retries, SSL.
Raises:
-
ValueError–If URL scheme is not supported
-
FileNotFoundError–If download fails
Source code in src/tdoc_crawler/http_client/session.py
230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 | |
prune_http_cache ¶
prune_http_cache(cache_file: str | Path, *, max_bytes: int | None = None, now: float | None = None) -> HttpCacheStats
Remove expired entries, optionally evict to max_bytes, and vacuum.
Safe to call at any time: a missing or uninitialised database is a no-op,
and a concurrent writer that holds a lock only prevents the VACUUM
(logged at debug level) — row deletions still succeed.
Parameters:
-
(cache_file¶str | Path) –Path to the hishel SQLite cache database.
-
(max_bytes¶int | None, default:None) –When set, evict the least-recently-used entries until the live database size is at or below this many bytes.
-
(now¶float | None, default:None) –Overridable clock for deterministic tests.
Returns:
-
HttpCacheStats–class:
HttpCacheStatsdescribing what was reclaimed.
Source code in src/tdoc_crawler/http_client/cache_maintenance.py
48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 | |
resolve_ssl_verify ¶
resolve_ssl_verify(verify: bool | str | None = None, http_config: HttpConfig | None = None) -> bool | str
Resolve SSL verification behavior from explicit argument or HttpConfig.
Resolution order:
1. Explicit verify parameter
2. http_config.verify_ssl if provided
3. Default (True)
Parameters:
-
(verify¶bool | str | None, default:None) –Optional explicit SSL verification mode.
-
(http_config¶HttpConfig | None, default:None) –Optional HttpConfig. When provided, verify_ssl is used as fallback.
Returns:
Source code in src/tdoc_crawler/http_client/session.py
412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 | |