Skip to content

The Python API mirrors the CLI: the same core-library functions back both.

Package overview

tdoc_crawler

TDoc Crawler - CLI tool for querying structured 3GPP TDoc data.

TDoc operations

Main-file selection, open, and re-classification are shared by CLI and MCP.

operations

TDoc operations.

CheckoutResult dataclass

CheckoutResult(success_count: int, error_count: int, errors: list[str], archives: list[ArchiveRecord] = list(), skipped_count: int = 0)

Result of a checkout operation.

FetchMethod

Bases: Enum

Enumeration of available TDoc fetching methods.

FetchMissingResult

FetchMissingResult(refreshed: list[TDocMetadata], fetch_result: TDocCrawlResult | None = None, used_whatthespec_fallback: bool = False)

Result from fetch_missing_tdocs orchestration.

Parameters:

  • refreshed

    (list[TDocMetadata]) –

    Updated list of TDoc metadata after fetching

  • fetch_result

    (TDocCrawlResult | None, default: None ) –

    Result of the batch fetch operation

  • used_whatthespec_fallback

    (bool, default: False ) –

    Whether WhatTheSpec was used as fallback

Source code in src/tdoc_crawler/tdocs/operations/fetch.py
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
def __init__(
    self,
    refreshed: list[TDocMetadata],
    fetch_result: TDocCrawlResult | None = None,
    used_whatthespec_fallback: bool = False,
) -> None:
    """Initialize fetch result.

    Args:
        refreshed: Updated list of TDoc metadata after fetching
        fetch_result: Result of the batch fetch operation
        used_whatthespec_fallback: Whether WhatTheSpec was used as fallback
    """
    self.refreshed = refreshed
    self.fetch_result = fetch_result
    self.used_whatthespec_fallback = used_whatthespec_fallback

FieldChange dataclass

FieldChange(field: str, old: str, new: str)

One metadata field that changed during a refresh.

MeetingPatternError

Bases: ValueError

Raised when a meeting filter pattern matches no locally known meeting.

MeetingRefreshSummary dataclass

MeetingRefreshSummary(meeting_id: int, short_name: str, tdocs_before: int, tdocs_after: int, added: tuple[str, ...], changed: tuple[TDocChange, ...], unchanged: int, absent_from_list: tuple[str, ...], errors: tuple[str, ...])

Changes found for one meeting in the refresh window.

ReclassifyOutcome dataclass

ReclassifyOutcome(tdoc_id: str, status: ReclassifyStatus, message: str = '', selection: MainFileSelection | None = None)

Result of re-classifying one TDoc.

RefreshResult dataclass

RefreshResult(lookback_days: int, start_date: date, meetings_scanned: int, meetings_with_changes: int, tdocs_processed: int, tdocs_added: int, tdocs_changed: int, tdocs_unchanged: int, urls_repaired: int, meetings: tuple[MeetingRefreshSummary, ...], errors: tuple[str, ...])

Summary of a complete metadata refresh run.

TDocChange dataclass

TDocChange(tdoc_id: str, url_repaired: bool = False, field_changes: tuple[FieldChange, ...] = ())

Change report for a single TDoc.

TDocCrawlResult dataclass

TDocCrawlResult(processed: int, inserted: int, updated: int, errors: list[str])

TDocCrawler

TDocCrawler(database: TDocDatabase)

Configuration-driven crawler orchestrating subinterpreter workers.

Source code in src/tdoc_crawler/tdocs/operations/crawl.py
43
44
45
46
47
def __init__(self, database: TDocDatabase) -> None:
    if not isinstance(database, TDocDatabase):
        msg = f"Expected TDocDatabase, got {type(database)}"
        raise TypeError(msg)
    self.database = database

crawl async

crawl(config: TDocCrawlConfig, progress_callback: Callable[[float, float], None] | None = None) -> TDocCrawlResult

Execute a crawl using the provided configuration.

Parameters:

  • config

    (TDocCrawlConfig) –

    Configuration for the TDoc crawl

  • progress_callback

    (Callable[[float, float], None] | None, default: None ) –

    Optional callback for progress updates

Returns:

Source code in src/tdoc_crawler/tdocs/operations/crawl.py
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
async def crawl(self, config: TDocCrawlConfig, progress_callback: Callable[[float, float], None] | None = None) -> TDocCrawlResult:
    """Execute a crawl using the provided configuration.

    Args:
        config: Configuration for the TDoc crawl
        progress_callback: Optional callback for progress updates

    Returns:
        Summary of crawl results
    """
    meetings = await self.get_meetings_to_crawl(config)
    if not meetings:
        logger.warning("No meetings found matching criteria")
        return TDocCrawlResult(processed=0, inserted=0, updated=0, errors=["No meetings found"])

    meetings = self._apply_meeting_limits(
        meetings,
        config.limit_meetings,
        config.limit_meetings_per_subwg,
        config.limit_subwgs,
    )
    if not meetings:
        logger.warning("Meetings filtered out by limits configuration")
        return TDocCrawlResult(processed=0, inserted=0, updated=0, errors=["No meetings found after applying limits"])

    existing_ids: set[str] = set()
    if config.incremental:
        existing_ids = {tdoc_id.upper() for tdoc_id in await self.database.get_existing_tdoc_ids(config.working_groups)}

    targets = set(config.target_ids) if config.target_ids else None

    collected: list[TDocMetadata] = []
    work_items: dict[str, list[str]] = {}
    agenda_items: dict[int, dict[str, str]] = {}
    errors = self._crawl_meetings(meetings, config, collected, work_items, agenda_items, existing_ids, targets)

    inserted = 0
    updated = 0
    if collected:
        inserted, updated = await self.database.bulk_upsert_tdocs(collected, progress_callback=progress_callback)
    if work_items:
        await self.database.store_tdoc_work_items(work_items)
    if agenda_items:
        await self.database.store_meeting_agenda_items(agenda_items)

    return TDocCrawlResult(
        processed=len(collected),
        inserted=inserted,
        updated=updated,
        errors=errors,
    )

get_meetings_to_crawl async

get_meetings_to_crawl(config: TDocCrawlConfig) -> list[MeetingMetadata]

Query meetings from the database based on configuration filters.

Source code in src/tdoc_crawler/tdocs/operations/crawl.py
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
async def get_meetings_to_crawl(self, config: TDocCrawlConfig) -> list[MeetingMetadata]:
    """Query meetings from the database based on configuration filters."""
    meeting_config = MeetingQueryConfig(
        working_groups=config.working_groups,
        subgroups=config.subgroups,
        limit=config.limit_meetings,
        order=SortOrder.DESC,
        include_without_files=False,
    )

    meetings = await self.database.query_meetings(meeting_config)

    filtered = self._filter_meetings_by_ids(meetings, config.meeting_ids)
    filtered = filter_meetings_by_patterns(filtered, config.meeting_patterns)
    filtered = self._filter_meetings_by_start_date(filtered, config.start_date)
    return self._filter_meetings_by_end_date(filtered, config.end_date)

TDocNotFoundError

Bases: LookupError

Raised when a TDoc is neither stored nor retrievable from the portal.

TDocOpenResult dataclass

TDocOpenResult(metadata: TDocMetadata, selection: MainFileSelection, fetch_errors: tuple[str, ...] = ())

Outcome of resolving a TDoc archive's main document.

checkout_meeting_tdocs

checkout_meeting_tdocs(meetings: list[MeetingMetadata], checkout_dir: Path, http_cache_file: Path, session: Session | None = None, cache_manager_name: str | None = None, extract: ExtractConfig | None = None) -> CheckoutResult

Checkout TDoc files from a list of meetings.

Parameters:

  • meetings

    (list[MeetingMetadata]) –

    List of MeetingMetadata to checkout TDocs from

  • checkout_dir

    (Path) –

    Base checkout directory

  • http_cache_file

    (Path) –

    Path to HTTP cache database

  • session

    (Session | None, default: None ) –

    Optional requests.Session to reuse for downloads

  • cache_manager_name

    (str | None, default: None ) –

    Optional cache manager name for HTTP caching

  • extract

    (ExtractConfig | None, default: None ) –

    Extraction settings; defaults to ExtractConfig()

Returns:

Source code in src/tdoc_crawler/tdocs/operations/checkout.py
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
def checkout_meeting_tdocs(
    meetings: list[MeetingMetadata],
    checkout_dir: Path,
    http_cache_file: Path,
    session: requests.Session | None = None,
    cache_manager_name: str | None = None,
    extract: ExtractConfig | None = None,
) -> CheckoutResult:
    """Checkout TDoc files from a list of meetings.

    Args:
        meetings: List of MeetingMetadata to checkout TDocs from
        checkout_dir: Base checkout directory
        http_cache_file: Path to HTTP cache database
        session: Optional requests.Session to reuse for downloads
        cache_manager_name: Optional cache manager name for HTTP caching
        extract: Extraction settings; defaults to ``ExtractConfig()``

    Returns:
        CheckoutResult with success/error counts
    """
    if not meetings:
        return CheckoutResult(success_count=0, error_count=0, errors=[])

    unique: dict[str, TDocMetadata] = {}
    errors: list[str] = []

    for meeting in meetings:
        if not meeting.files_url:
            errors.append(f"{meeting.short_name}: no files URL")
            continue
        try:
            meeting_folder = get_meeting_checkout_folder(meeting, checkout_dir)
            doclist_path = meeting_folder / f"meeting_{meeting.meeting_id}_doclist.xlsx"
            tdocs = fetch_meeting_document_list(
                meeting.meeting_id,
                http_cache_file=http_cache_file,
                save_doclist_to=doclist_path,
            )
        except DocumentListError as exc:
            errors.append(f"{meeting.short_name}: {exc}")
            continue
        for metadata in tdocs:
            if metadata.tdoc_id not in unique:
                unique[metadata.tdoc_id] = metadata

    return checkout_tdocs(
        list(unique.values()),
        checkout_dir,
        force=False,
        session=session,
        cache_manager_name=cache_manager_name,
        extract=extract,
    )

checkout_tdoc

checkout_tdoc(metadata: TDocMetadata, checkout_dir: Path, *, force: bool = False, session: Session | None = None, keep_zip: bool = False, archive_sink: list[ArchiveRecord] | None = None, extract: ExtractConfig | None = None) -> Path

Download and extract a TDoc to the checkout folder.

Parameters:

  • metadata

    (TDocMetadata) –

    TDoc metadata containing URL and ID

  • checkout_dir

    (Path) –

    Base checkout directory

  • force

    (bool, default: False ) –

    If True, re-download even if already exists

  • session

    (Session | None, default: None ) –

    Optional requests.Session to reuse for downloads

  • keep_zip

    (bool, default: False ) –

    If True, keep the downloaded zip in the checkout folder

  • archive_sink

    (list[ArchiveRecord] | None, default: None ) –

    If given, the zip member listing is appended to it when a download happens (an already checked-out TDoc records nothing)

  • extract

    (ExtractConfig | None, default: None ) –

    Extraction settings (flattening, skip patterns); defaults to ExtractConfig()

Returns:

  • Path –

    Path to the extracted TDoc directory

Raises:

Source code in src/tdoc_crawler/tdocs/operations/checkout.py
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
def checkout_tdoc(
    metadata: TDocMetadata,
    checkout_dir: Path,
    *,
    force: bool = False,
    session: requests.Session | None = None,
    keep_zip: bool = False,
    archive_sink: list[ArchiveRecord] | None = None,
    extract: ExtractConfig | None = None,
) -> Path:
    """Download and extract a TDoc to the checkout folder.

    Args:
        metadata: TDoc metadata containing URL and ID
        checkout_dir: Base checkout directory
        force: If True, re-download even if already exists
        session: Optional requests.Session to reuse for downloads
        keep_zip: If True, keep the downloaded zip in the checkout folder
        archive_sink: If given, the zip member listing is appended to it when
            a download happens (an already checked-out TDoc records nothing)
        extract: Extraction settings (flattening, skip patterns); defaults to
            ``ExtractConfig()``

    Returns:
        Path to the extracted TDoc directory

    Raises:
        FileNotFoundError: If download fails or zip is empty
    """
    # Check if TDoc is withdrawn - if so, skip download and log debug message
    if metadata.status == TDocStatus.WITHDRAWN:
        logger.debug(f"Skipping checkout for withdrawn TDoc {metadata.tdoc_id}")
        msg = "withdrawn"
        raise FileNotFoundError(msg)

    # Validate the URL *before* touching the filesystem: WhatTheSpec records
    # for TDocs without a downloadable document carry an empty link, and
    # creating the directory first used to leave empty ``<checkout>/<id>``
    # folders behind (the path mirrors the URL, so an empty URL maps to the
    # root of the checkout tree).
    if not metadata.url or not metadata.url.strip():
        msg = f"TDoc {metadata.tdoc_id} has no download URL (the metadata source provided no file link)"
        raise ValueError(msg)

    checkout_path = get_checkout_path(metadata, checkout_dir)

    if checkout_path.exists() and not force:
        has_files = any(f.is_file() for f in checkout_path.rglob("*") if not f.name.startswith("."))
        if has_files:
            logger.debug(f"TDoc {metadata.tdoc_id} already checked out at {checkout_path}")
            return checkout_path
        logger.debug(f"TDoc {metadata.tdoc_id} checkout dir exists but is empty, re-downloading")

    checkout_path.mkdir(parents=True, exist_ok=True)
    temp_zip_file = checkout_path / f"{metadata.tdoc_id}.zip"

    try:
        try:
            archive_members = _download_and_list_members(metadata, metadata.url, temp_zip_file, session)
        except _EmptyDownloadError:
            # A 0-byte response (HTTP 200, no body) is not raised by
            # download_to_file, so a corrected URL must be tried for it too —
            # not just for HTTPError/BadZipFile.
            corrected_url = _resolve_corrected_url(metadata)
            if corrected_url and corrected_url != metadata.url:
                logger.info("Retrying empty archive %s with corrected URL: %s", metadata.tdoc_id, corrected_url)
                with requests.Session(disable_http3=True) as plain_session:
                    archive_members = _download_and_list_members(metadata, corrected_url, temp_zip_file, plain_session)
            else:
                msg = f"TDoc {metadata.tdoc_id}: downloaded archive is empty (0 bytes)"
                raise ValueError(msg)
        except (requests.HTTPError, zipfile.BadZipFile):
            corrected_url = _resolve_corrected_url(metadata)
            if corrected_url and corrected_url != metadata.url:
                logger.info("Retrying %s with corrected URL: %s", metadata.tdoc_id, corrected_url)
                with requests.Session(disable_http3=True) as plain_session:
                    archive_members = _download_and_list_members(metadata, corrected_url, temp_zip_file, plain_session)
            else:
                raise
        extraction = extract or ExtractConfig()
        safe_extract_zip(
            temp_zip_file,
            checkout_path,
            exclude=extraction.exclude,
            flatten_single_root=extraction.flatten_single_root,
        )
        if archive_sink is not None:
            archive_sink.append(ArchiveRecord(kind=ArchiveKind.TDOC, item_id=metadata.tdoc_id, archive_name=temp_zip_file.name, members=archive_members))
        logger.info("Checked out %s to %s", metadata.tdoc_id, checkout_path)
    finally:
        if not keep_zip and temp_zip_file.exists():
            temp_zip_file.unlink()

    return checkout_path

checkout_tdocs

checkout_tdocs(results: list[TDocMetadata], checkout_dir: Path, force: bool = False, session: Session | None = None, cache_manager_name: str | None = None, keep_zip: bool = False, extract: ExtractConfig | None = None, workers: int = 1) -> CheckoutResult

Checkout multiple TDoc files to the checkout directory.

Parameters:

  • results

    (list[TDocMetadata]) –

    List of TDocMetadata to checkout

  • checkout_dir

    (Path) –

    Base checkout directory

  • force

    (bool, default: False ) –

    If True, re-download even if already exists

  • session

    (Session | None, default: None ) –

    Optional requests.Session to reuse for downloads (sequential mode only; parallel mode uses per-download sessions for thread safety)

  • cache_manager_name

    (str | None, default: None ) –

    Optional cache manager name for HTTP caching

  • keep_zip

    (bool, default: False ) –

    If True, keep each downloaded zip next to its extracted files

  • extract

    (ExtractConfig | None, default: None ) –

    Extraction settings; defaults to ExtractConfig()

  • workers

    (int, default: 1 ) –

    Number of parallel downloads. 1 (default) keeps sequential behavior; >1 downloads concurrently while preserving input order.

Returns:

  • CheckoutResult –

    CheckoutResult with success/error counts and the archive listings

  • CheckoutResult –

    recorded during this run (persist them with save_archive_records)

Source code in src/tdoc_crawler/tdocs/operations/checkout.py
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
def checkout_tdocs(
    results: list[TDocMetadata],
    checkout_dir: Path,
    force: bool = False,
    session: requests.Session | None = None,
    cache_manager_name: str | None = None,
    keep_zip: bool = False,
    extract: ExtractConfig | None = None,
    workers: int = 1,
) -> CheckoutResult:
    """Checkout multiple TDoc files to the checkout directory.

    Args:
        results: List of TDocMetadata to checkout
        checkout_dir: Base checkout directory
        force: If True, re-download even if already exists
        session: Optional requests.Session to reuse for downloads (sequential
            mode only; parallel mode uses per-download sessions for thread safety)
        cache_manager_name: Optional cache manager name for HTTP caching
        keep_zip: If True, keep each downloaded zip next to its extracted files
        extract: Extraction settings; defaults to ``ExtractConfig()``
        workers: Number of parallel downloads. 1 (default) keeps sequential
            behavior; >1 downloads concurrently while preserving input order.

    Returns:
        CheckoutResult with success/error counts and the archive listings
        recorded during this run (persist them with ``save_archive_records``)
    """
    if not results:
        return CheckoutResult(success_count=0, error_count=0, errors=[])

    success_count = 0
    error_count = 0
    skipped_count = 0
    errors: list[str] = []
    archives: list[ArchiveRecord] = []

    def _record(outcome: tuple[bool, bool, str | None, list[ArchiveRecord]]) -> None:
        nonlocal success_count, error_count, skipped_count
        ok, skipped, error, item_archives = outcome
        archives.extend(item_archives)
        if ok:
            success_count += 1
        elif skipped:
            skipped_count += 1
        else:
            errors.append(error or "unknown error")
            error_count += 1

    if workers <= 1:
        for metadata in results:
            _record(_checkout_one(metadata, checkout_dir, force=force, session=session, keep_zip=keep_zip, extract=extract))
    else:
        # Do not share the niquests Session across threads; per-download sessions
        # (session=None) are created and closed inside download_to_file.
        with ThreadPoolExecutor(max_workers=workers) as executor:
            futures = [
                executor.submit(
                    _checkout_one,
                    metadata,
                    checkout_dir,
                    force=force,
                    session=None,
                    keep_zip=keep_zip,
                    extract=extract,
                )
                for metadata in results
            ]
            # Consume in input order so results/errors are deterministic.
            for future in futures:
                _record(future.result())

    return CheckoutResult(success_count=success_count, error_count=error_count, errors=errors, archives=archives, skipped_count=skipped_count)

clear_checkout_tdocs

clear_checkout_tdocs(checkout_dir: Path, *, keep_specs: bool = True) -> int

Delete downloaded TDoc files from the checkout directory.

Destructive by design and never called automatically: only the explicit --purge-checkouts switch on database clear-* reaches this helper.

Parameters:

  • checkout_dir

    (Path) –

    Base checkout directory

  • keep_specs

    (bool, default: True ) –

    Preserve the Specs subtree (TDoc-only clears)

Returns:

  • int –

    Number of top-level entries removed

Source code in src/tdoc_crawler/tdocs/operations/checkout.py
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
def clear_checkout_tdocs(checkout_dir: Path, *, keep_specs: bool = True) -> int:
    """Delete downloaded TDoc files from the checkout directory.

    Destructive by design and never called automatically: only the explicit
    ``--purge-checkouts`` switch on ``database clear-*`` reaches this helper.

    Args:
        checkout_dir: Base checkout directory
        keep_specs: Preserve the ``Specs`` subtree (TDoc-only clears)

    Returns:
        Number of top-level entries removed
    """
    if not checkout_dir.exists():
        return 0
    removed = 0
    for entry in checkout_dir.iterdir():
        if keep_specs and entry.name == "Specs":
            continue
        if entry.is_dir():
            shutil.rmtree(entry)
        else:
            entry.unlink()
        removed += 1
    return removed

create_source

create_source(method: FetchMethod, config: TDocSourceConfig, session: Session | None = None) -> WhatTheSpecSource | PortalSource | DocumentListSource

Create TDoc source based on fetch method.

Parameters:

  • method

    (FetchMethod) –

    Fetch method to use

  • config

    (TDocSourceConfig) –

    Source configuration

  • session

    (Session | None, default: None ) –

    Optional requests.Session to reuse

Returns:

  • WhatTheSpecSource | PortalSource | DocumentListSource –

    Configured TDoc source instance

Raises:

  • ValueError –

    If method requires credentials but none provided

Source code in src/tdoc_crawler/tdocs/operations/fetch.py
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
def create_source(
    method: FetchMethod,
    config: TDocSourceConfig,
    session: requests.Session | None = None,
) -> WhatTheSpecSource | PortalSource | DocumentListSource:
    """Create TDoc source based on fetch method.

    Args:
        method: Fetch method to use
        config: Source configuration
        session: Optional requests.Session to reuse

    Returns:
        Configured TDoc source instance

    Raises:
        ValueError: If method requires credentials but none provided
    """
    if method == FetchMethod.WHATTHESPEC:
        return WhatTheSpecSource(config=config, session=session)
    if method == FetchMethod.PORTAL_FULL_METADATA:
        return PortalSource(config=config)
    if method == FetchMethod.DOCUMENT_LIST:
        return DocumentListSource(config=config)
    if method == FetchMethod.PORTAL_URL_ONLY:
        # For URL-only method, we'll handle it separately as it doesn't fit the source pattern
        msg = "PORTAL_URL_ONLY method should be handled separately"
        raise ValueError(msg)
    msg = f"Unknown fetch method: {method}"
    raise ValueError(msg)

diff_tdoc

diff_tdoc(before: TDocMetadata, after: TDocMetadata) -> tuple[FieldChange, ...]

Return the meaningful metadata fields that differ between two records.

Source code in src/tdoc_crawler/tdocs/operations/refresh.py
170
171
172
173
174
175
176
177
178
def diff_tdoc(before: TDocMetadata, after: TDocMetadata) -> tuple[FieldChange, ...]:
    """Return the meaningful metadata fields that differ between two records."""
    changes: list[FieldChange] = []
    for name in COMPARED_FIELDS:
        old_value = _as_text(getattr(before, name, None))
        new_value = _as_text(getattr(after, name, None))
        if old_value != new_value:
            changes.append(FieldChange(field=name, old=old_value, new=new_value))
    return tuple(changes)

fetch_missing_tdocs async

fetch_missing_tdocs(database: TDocDatabase, config: TDocQueryConfig, results: list[TDocMetadata], full_metadata: bool = False, use_whatthespec: bool = False, session: Session | None = None, http_cache_file: Path | None = None, db_file: Path | None = None, url_only_first: bool = False) -> FetchMissingResult

Fetch missing TDocs if any are requested but not found in database.

Orchestrates fetching based on flags, uses WhatTheSpec fallback when needed.

Parameters:

  • database

    (TDocDatabase) –

    Database instance

  • config

    (TDocQueryConfig) –

    TDocQueryConfig with requested TDoc IDs

  • results

    (list[TDocMetadata]) –

    Current query results

  • full_metadata

    (bool, default: False ) –

    If True, fetch full metadata from portal

  • use_whatthespec

    (bool, default: False ) –

    If True, always use WhatTheSpec

  • session

    (Session | None, default: None ) –

    Optional requests.Session to reuse

  • http_cache_file

    (Path | None, default: None ) –

    Optional path to HTTP cache database file

  • db_file

    (Path | None, default: None ) –

    Optional path to metadata database file

  • url_only_first

    (bool, default: False ) –

    If True, try the public URL-only portal lookup first (checkout and open need a download URL, not full metadata)

Returns:

Source code in src/tdoc_crawler/tdocs/operations/fetch.py
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
async def fetch_missing_tdocs(
    database: TDocDatabase,
    config: TDocQueryConfig,
    results: list[TDocMetadata],
    full_metadata: bool = False,
    use_whatthespec: bool = False,
    session: requests.Session | None = None,
    http_cache_file: Path | None = None,
    db_file: Path | None = None,
    url_only_first: bool = False,
) -> FetchMissingResult:
    """Fetch missing TDocs if any are requested but not found in database.

    Orchestrates fetching based on flags, uses WhatTheSpec fallback when needed.

    Args:
        database: Database instance
        config: TDocQueryConfig with requested TDoc IDs
        results: Current query results
        full_metadata: If True, fetch full metadata from portal
        use_whatthespec: If True, always use WhatTheSpec
        session: Optional requests.Session to reuse
        http_cache_file: Optional path to HTTP cache database file
        db_file: Optional path to metadata database file
        url_only_first: If True, try the public URL-only portal lookup first
            (checkout and open need a download URL, not full metadata)

    Returns:
        FetchMissingResult with updated results and fetch statistics
    """
    if not config.tdoc_ids:
        return FetchMissingResult(refreshed=results)

    requested = [value.upper() for value in config.tdoc_ids]
    found = {item.tdoc_id for item in results}
    missing = [value for value in requested if value not in found]
    if not missing:
        return FetchMissingResult(refreshed=results)

    # Log what we're about to do
    if use_whatthespec:
        logger.info(f"Fetching missing TDocs via WhatTheSpec: {', '.join(missing)}")
    elif full_metadata:
        logger.info(f"Fetching missing TDocs with full metadata: {', '.join(missing)}")
    else:
        logger.info(f"Fetching missing TDocs: {', '.join(missing)}")

    # A by-ID fetch bypasses the meeting crawl; make sure the parent meetings
    # are known so stored records link to them instead of becoming orphans.
    await ensure_meeting_links(database, missing)

    # Handle use_whatthespec flag - always use WhatTheSpec method
    if use_whatthespec:
        fetch_result = await fetch_via_whatthespec_batch(database, missing, session=session, http_cache_file=http_cache_file, db_file=db_file)
        refreshed = await database.query_tdocs(config)
        return FetchMissingResult(
            refreshed=refreshed,
            fetch_result=fetch_result,
            used_whatthespec_fallback=True,
        )

    # Handle full_metadata flag - use authenticated portal method
    credentials = resolve_credentials(None, None)
    credentials_unavailable = credentials is None

    if full_metadata:
        if credentials_unavailable:
            fetch_result = TDocCrawlResult(
                processed=len(missing),
                inserted=0,
                updated=0,
                errors=["Portal credentials required for full metadata fetching"],
            )
            logger.warning("Portal credentials required for full metadata fetching")
            return FetchMissingResult(refreshed=results, fetch_result=fetch_result)

        fetch_result = await fetch_missing_tdocs_batch(
            database=database,
            missing_ids=missing,
            session=session,
            http_cache_file=http_cache_file,
        )

        refreshed = await database.query_tdocs(config)
        return FetchMissingResult(refreshed=refreshed, fetch_result=fetch_result)

    url_result: TDocCrawlResult | None = None
    remaining = missing
    if url_only_first:
        logger.info("Resolving portal download URLs: %s", ", ".join(missing))
        url_result = await fetch_via_portal_url_batch(
            database,
            missing,
            session=session,
            http_cache_file=http_cache_file,
        )
        resolved = {item.tdoc_id for item in await database.query_tdocs(config)}
        remaining = [tdoc_id for tdoc_id in missing if tdoc_id not in resolved]
        if not remaining:
            return FetchMissingResult(
                refreshed=await database.query_tdocs(config),
                fetch_result=url_result,
            )

    used_fallback = False

    # Without credentials the portal lookup can only fail; go straight to
    # WhatTheSpec instead of logging an expected failure first.
    if credentials_unavailable:
        logger.info("No portal credentials; fetching missing TDocs via WhatTheSpec")
        fetch_result = await fetch_via_whatthespec_batch(database, remaining, session=session, http_cache_file=http_cache_file, db_file=db_file)
        used_fallback = True
    else:
        fetch_result = await fetch_missing_tdocs_batch(
            database=database,
            missing_ids=remaining,
            session=session,
            http_cache_file=http_cache_file,
        )

        # Authentication can still fail (expired account, network); fall back.
        has_credentials_error = any("credentials required" in str(err).lower() for err in fetch_result.errors)
        if has_credentials_error:
            logger.info("Falling back to WhatTheSpec for missing TDocs due to credential issues")
            fetch_result = await fetch_via_whatthespec_batch(database, remaining, session=session, http_cache_file=http_cache_file, db_file=db_file)
            used_fallback = True
        else:
            used_fallback = False

    if url_result is not None:
        fetch_result = _merge_fetch_results(url_result, fetch_result)

    refreshed = await database.query_tdocs(config)
    return FetchMissingResult(
        refreshed=refreshed,
        fetch_result=fetch_result,
        used_whatthespec_fallback=used_fallback,
    )

fetch_missing_tdocs_batch async

fetch_missing_tdocs_batch(database: TDocDatabase, missing_ids: list[str], session: Session | None = None, timeout: int = 30, http_config: HttpConfig | None = None, http_cache_file: Path | None = None) -> TDocCrawlResult

Fetch multiple missing TDocs using portal authentication.

Parameters:

  • database

    (TDocDatabase) –

    Database instance for storing results

  • missing_ids

    (list[str]) –

    List of TDoc IDs to fetch

  • http_config

    (HttpConfig | None, default: None ) –

    HTTP configuration from ThreeGPPConfig

  • timeout

    (int, default: 30 ) –

    Request timeout in seconds

  • session

    (Session | None, default: None ) –

    Optional requests.Session to reuse

  • http_cache_file

    (Path | None, default: None ) –

    Optional path to HTTP cache database file

Returns:

Source code in src/tdoc_crawler/tdocs/operations/fetch.py
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
async def fetch_missing_tdocs_batch(
    database: TDocDatabase,
    missing_ids: list[str],
    session: requests.Session | None = None,
    timeout: int = 30,
    http_config: HttpConfig | None = None,
    http_cache_file: Path | None = None,
) -> TDocCrawlResult:
    """Fetch multiple missing TDocs using portal authentication.

    Args:
        database: Database instance for storing results
        missing_ids: List of TDoc IDs to fetch
        http_config: HTTP configuration from ThreeGPPConfig
        timeout: Request timeout in seconds
        session: Optional requests.Session to reuse
        http_cache_file: Optional path to HTTP cache database file

    Returns:
        TDocCrawlResult with fetch statistics
    """
    errors = []
    inserted_count = 0
    updated_count = 0

    # Create source configuration and portal source
    config = TDocSourceConfig(
        http_config=http_config,
        http_cache_file=http_cache_file,
        timeout=timeout,
    )
    source = PortalSource(config=config)

    for tdoc_id in missing_ids:
        try:
            metadata = source.fetch_by_id(tdoc_id, timeout=timeout)
            if metadata is None:
                error_msg = f"Portal returned no data for {tdoc_id}"
                logger.warning(error_msg)
                errors.append(error_msg)
                continue

            # meeting_id resolution is now done inside source modules

            inserted, updated = await database.upsert_tdoc(metadata)
            if inserted:
                inserted_count += 1
            elif updated:
                updated_count += 1

            logger.info("Successfully fetched and stored %s", tdoc_id)

        except ValidationError as exc:
            validation_errors = []
            for error in exc.errors():
                field = error["loc"][0] if error["loc"] else "unknown"
                message = error["msg"]
                validation_errors.append(f"{field}: {message}")

            error_msg = f"Failed to fetch {tdoc_id}: {'; '.join(validation_errors)}"
            logger.warning(error_msg)
            errors.append(error_msg)
        except Exception as exc:
            error_msg = f"Failed to fetch {tdoc_id}: {exc}"
            logger.exception(error_msg)
            errors.append(error_msg)

    # Clean up source
    source.close()

    return TDocCrawlResult(
        processed=len(missing_ids),
        inserted=inserted_count,
        updated=updated_count,
        errors=errors,
    )

fetch_via_portal_url_batch async

fetch_via_portal_url_batch(database: TDocDatabase, missing_ids: list[str], *, session: Session | None = None, timeout: int = 30, http_cache_file: Path | None = None) -> TDocCrawlResult

Resolve download URLs for missing TDocs without credentials.

The portal's DownloadTDoc endpoint is public: it redirects to the direct file URL, which is all a checkout needs. Records created this way carry the URL and no bibliographic metadata; a later tdoc crawl (or a credentialed fetch) fills the rest in.

Parameters:

  • database

    (TDocDatabase) –

    Database instance for storing results

  • missing_ids

    (list[str]) –

    TDoc IDs to resolve

  • session

    (Session | None, default: None ) –

    Optional requests.Session to reuse (left open)

  • timeout

    (int, default: 30 ) –

    Request timeout in seconds

  • http_cache_file

    (Path | None, default: None ) –

    Optional path to HTTP cache database file

Returns:

Source code in src/tdoc_crawler/tdocs/operations/fetch.py
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
async def fetch_via_portal_url_batch(
    database: TDocDatabase,
    missing_ids: list[str],
    *,
    session: requests.Session | None = None,
    timeout: int = 30,
    http_cache_file: Path | None = None,
) -> TDocCrawlResult:
    """Resolve download URLs for missing TDocs without credentials.

    The portal's ``DownloadTDoc`` endpoint is public: it redirects to the
    direct file URL, which is all a checkout needs. Records created this way
    carry the URL and no bibliographic metadata; a later ``tdoc crawl`` (or a
    credentialed fetch) fills the rest in.

    Args:
        database: Database instance for storing results
        missing_ids: TDoc IDs to resolve
        session: Optional requests.Session to reuse (left open)
        timeout: Request timeout in seconds
        http_cache_file: Optional path to HTTP cache database file

    Returns:
        TDocCrawlResult with fetch statistics
    """
    client = PortalClient(session=session, timeout=timeout, http_cache_file=http_cache_file)
    inserted_count = 0
    updated_count = 0
    errors: list[str] = []

    try:
        for tdoc_id in missing_ids:
            try:
                url = client.extract_tdoc_url(tdoc_id)
            except (PortalParsingError, requests.RequestException) as exc:
                logger.info("Portal URL lookup failed for %s: %s", tdoc_id, exc)
                continue

            metadata = TDocMetadata(
                tdoc_id=tdoc_id.upper(),
                meeting_id=0,
                title="",
                source="",
                contact="",
                agenda_item_nbr=parse_agenda_item_nbr("0"),
                url=url,
            )
            inserted, updated = await database.upsert_tdoc(metadata)
            inserted_count += int(inserted)
            updated_count += int(updated)
            logger.info("Resolved portal URL for %s", tdoc_id)
    finally:
        if session is None:
            client.close()

    return TDocCrawlResult(
        processed=len(missing_ids),
        inserted=inserted_count,
        updated=updated_count,
        errors=errors,
    )

fetch_via_whatthespec_batch async

fetch_via_whatthespec_batch(database: TDocDatabase, missing_ids: list[str], http_config: HttpConfig | None = None, session: Session | None = None, http_cache_file: Path | None = None, db_file: Path | None = None) -> TDocCrawlResult

Fetch multiple missing TDocs using WhatTheSpec.net API.

Parameters:

  • database

    (TDocDatabase) –

    Database instance for storing results

  • missing_ids

    (list[str]) –

    List of TDoc IDs to fetch

  • http_config

    (HttpConfig | None, default: None ) –

    HTTP configuration from ThreeGPPConfig

  • session

    (Session | None, default: None ) –

    Optional requests.Session to reuse

  • http_cache_file

    (Path | None, default: None ) –

    Optional path to HTTP cache database file

  • db_file

    (Path | None, default: None ) –

    Optional path to metadata database file

Returns:

Source code in src/tdoc_crawler/tdocs/operations/fetch.py
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
async def fetch_via_whatthespec_batch(
    database: TDocDatabase,
    missing_ids: list[str],
    http_config: HttpConfig | None = None,
    session: requests.Session | None = None,
    http_cache_file: Path | None = None,
    db_file: Path | None = None,
) -> TDocCrawlResult:
    """Fetch multiple missing TDocs using WhatTheSpec.net API.

    Args:
        database: Database instance for storing results
        missing_ids: List of TDoc IDs to fetch
        http_config: HTTP configuration from ThreeGPPConfig
        session: Optional requests.Session to reuse
        http_cache_file: Optional path to HTTP cache database file
        db_file: Optional path to metadata database file

    Returns:
        TDocCrawlResult with fetch statistics
    """
    errors = []
    inserted_count = 0
    updated_count = 0

    # Create source configuration and WhatTheSpec source
    config = TDocSourceConfig(
        http_config=http_config,
        http_cache_file=http_cache_file,
    )
    source = WhatTheSpecSource(config=config, session=session)
    resolved_db_file = db_file if db_file is not None else PathConfig().db_file

    for tdoc_id in missing_ids:
        try:
            metadata = source.fetch_by_id(tdoc_id, db_file=resolved_db_file)
            if metadata:
                inserted, updated = await database.upsert_tdoc(metadata)
                if inserted:
                    inserted_count += 1
                elif updated:
                    updated_count += 1
                logger.info("Successfully fetched %s via whatthespec", tdoc_id)
            else:
                error_msg = f"WhatTheSpec returned no data for {tdoc_id}"
                logger.warning(error_msg)
                errors.append(error_msg)
        except Exception as exc:
            error_msg = f"Error fetching {tdoc_id} via whatthespec: {exc}"
            logger.warning(error_msg)
            errors.append(error_msg)

    return TDocCrawlResult(
        processed=len(missing_ids),
        inserted=inserted_count,
        updated=updated_count,
        errors=errors,
    )

filter_meetings_by_patterns

filter_meetings_by_patterns(meetings: list[MeetingMetadata], patterns: list[str] | None) -> list[MeetingMetadata]

Filter meetings by short-name patterns, normalized names, or numeric IDs.

Patterns accept the portal spelling (SA4#138), the normalized database spelling (S4-138), fnmatch wildcards (SA4#13*), and plain numeric meeting IDs. A leading 3GPP on either side is ignored, so SA4#134 matches a meeting stored as 3GPPSA4#134. Every pattern must match at least one meeting — silently crawling nothing is worse than failing loudly.

Source code in src/tdoc_crawler/tdocs/operations/crawl.py
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
def filter_meetings_by_patterns(
    meetings: list[MeetingMetadata],
    patterns: list[str] | None,
) -> list[MeetingMetadata]:
    """Filter meetings by short-name patterns, normalized names, or numeric IDs.

    Patterns accept the portal spelling (``SA4#138``), the normalized database
    spelling (``S4-138``), ``fnmatch`` wildcards (``SA4#13*``), and plain
    numeric meeting IDs. A leading ``3GPP`` on either side is ignored, so
    ``SA4#134`` matches a meeting stored as ``3GPPSA4#134``. Every pattern
    must match at least one meeting — silently crawling nothing is worse
    than failing loudly.
    """
    if not patterns:
        return meetings

    matched_ids: set[int] = set()
    unmatched: list[str] = []
    for pattern in patterns:
        text = str(pattern).strip()
        if not text:
            continue
        matches = [meeting for meeting in meetings if _meeting_matches_pattern(meeting, text)]
        if not matches:
            unmatched.append(text)
            continue
        matched_ids.update(meeting.meeting_id for meeting in matches)

    if unmatched:
        known = sorted({meeting.short_name for meeting in meetings if meeting.short_name})[:10]
        hint = f" (known meetings include: {', '.join(known)})" if known else ""
        msg = f"No meetings matched: {', '.join(unmatched)}{hint}"
        raise MeetingPatternError(msg)
    return [meeting for meeting in meetings if meeting.meeting_id in matched_ids]

get_checked_out_tdocs

get_checked_out_tdocs(checkout_dir: Path) -> list[str]

Get list of TDoc IDs that are already checked out.

Parameters:

  • checkout_dir

    (Path) –

    Base checkout directory

Returns:

  • list[str] –

    List of TDoc IDs (directory names)

Source code in src/tdoc_crawler/tdocs/operations/checkout.py
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
def get_checked_out_tdocs(checkout_dir: Path) -> list[str]:
    """Get list of TDoc IDs that are already checked out.

    Args:
        checkout_dir: Base checkout directory

    Returns:
        List of TDoc IDs (directory names)
    """
    if not checkout_dir.exists():
        return []

    tdoc_ids = []
    for path in checkout_dir.rglob("*"):
        if path.is_dir():
            # Check if this is a leaf directory (contains files, not just subdirs)
            # and is not an intermediate directory
            has_files = any(f.is_file() for f in path.iterdir())
            if has_files:
                tdoc_ids.append(path.name)

    return tdoc_ids

get_checkout_path

get_checkout_path(metadata: TDocMetadata, checkout_dir: Path) -> Path

Calculate the checkout path for a TDoc based on its URL.

The checkout path mirrors the 3GPP server directory structure.

Parameters:

  • metadata

    (TDocMetadata) –

    TDoc metadata containing the URL

  • checkout_dir

    (Path) –

    Base checkout directory

Returns:

  • Path –

    Path to the checkout directory for this TDoc

Source code in src/tdoc_crawler/tdocs/operations/checkout.py
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
def get_checkout_path(metadata: TDocMetadata, checkout_dir: Path) -> Path:
    """Calculate the checkout path for a TDoc based on its URL.

    The checkout path mirrors the 3GPP server directory structure.

    Args:
        metadata: TDoc metadata containing the URL
        checkout_dir: Base checkout directory

    Returns:
        Path to the checkout directory for this TDoc
    """
    url_path_str: str = str(urlparse(metadata.url).path)
    url_path_str = url_path_str.lstrip("/")
    path_parts: list[str] = url_path_str.split("/")

    try:
        ftp_index = path_parts.index("ftp")
        relative_parts: list[str] = path_parts[ftp_index + 1 :]
    except ValueError:
        relative_parts = path_parts

    if relative_parts:
        relative_parts = relative_parts[:-1]

    return checkout_dir.joinpath(*relative_parts) / metadata.tdoc_id if relative_parts else checkout_dir / metadata.tdoc_id

get_meeting_checkout_folder

get_meeting_checkout_folder(meeting: MeetingMetadata, checkout_dir: Path) -> Path

Calculate the checkout folder for a meeting based on its files_url.

The folder mirrors the 3GPP server directory structure.

Parameters:

  • meeting

    (MeetingMetadata) –

    MeetingMetadata containing the files_url

  • checkout_dir

    (Path) –

    Base checkout directory

Returns:

  • Path –

    Path to the meeting checkout folder

Source code in src/tdoc_crawler/tdocs/operations/checkout.py
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
def get_meeting_checkout_folder(meeting: MeetingMetadata, checkout_dir: Path) -> Path:
    """Calculate the checkout folder for a meeting based on its files_url.

    The folder mirrors the 3GPP server directory structure.

    Args:
        meeting: MeetingMetadata containing the files_url
        checkout_dir: Base checkout directory

    Returns:
        Path to the meeting checkout folder
    """
    if meeting.files_url is None:
        msg = f"Meeting {meeting.short_name} has no files_url"
        raise ValueError(msg)

    url_path_str: str = str(urlparse(meeting.files_url).path)
    url_path_str = url_path_str.lstrip("/")
    path_parts: list[str] = url_path_str.split("/")

    try:
        ftp_index = path_parts.index("ftp")
        relative_parts: list[str] = path_parts[ftp_index + 1 :]
    except ValueError:
        relative_parts = path_parts

    return checkout_dir.joinpath(*relative_parts) if relative_parts else checkout_dir

has_revision_marks

has_revision_marks(path: Path) -> bool

True when an OOXML document carries tracked changes (revision marks).

Scans the word/ XML parts for insert/delete/move revision elements. Non-OOXML files (PDF, legacy .doc/.xls) cannot be checked cheaply and count as unmarked; unreadable files likewise.

This is the only signal used to prefer a tracked-changes version over a clean one — filename markers (_clean, _rm, ...) are inconsistent across authors and are never consulted.

Source code in src/tdoc_crawler/tdocs/operations/main_file.py
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
def has_revision_marks(path: Path) -> bool:
    """True when an OOXML document carries tracked changes (revision marks).

    Scans the ``word/`` XML parts for insert/delete/move revision elements.
    Non-OOXML files (PDF, legacy ``.doc``/``.xls``) cannot be checked
    cheaply and count as unmarked; unreadable files likewise.

    This is the *only* signal used to prefer a tracked-changes version
    over a clean one — filename markers (``_clean``, ``_rm``, ...) are
    inconsistent across authors and are never consulted.
    """
    if path.suffix.lower() != ".docx":
        return False
    try:
        with zipfile.ZipFile(path) as archive:
            return any(_REVISION_MARK_ELEMENTS.search(archive.read(name)) for name in archive.namelist() if name.startswith("word/") and name.endswith(".xml"))
    except (zipfile.BadZipFile, OSError):
        return False

member_candidates

member_candidates(extract_dir: Path) -> list[Path]

Non-clutter files of an extracted TDoc directory, sorted by path.

Source code in src/tdoc_crawler/tdocs/operations/main_file.py
278
279
280
281
282
283
def member_candidates(extract_dir: Path) -> list[Path]:
    """Non-clutter files of an extracted TDoc directory, sorted by path."""
    return sorted(
        (p for p in extract_dir.rglob("*") if p.is_file() and not is_clutter(p.relative_to(extract_dir).as_posix())),
        key=lambda p: p.relative_to(extract_dir).as_posix(),
    )

prepare_tdoc_file

prepare_tdoc_file(metadata: TDocMetadata, checkout_dir: Path, return_dir: bool = False, session: Session | None = None, *, override_member: str | None = None, config: CheckoutConfig | None = None, extract: ExtractConfig | None = None) -> Path

Prepare the local file for a TDoc, downloading/extracting if needed.

Routes through :func:checkout_tdoc so that tdoc open and tdoc checkout share a single business-logic path: the on-disk layout mirrors the 3GPP server (under tsg_<tsg>/<wg>/<meeting>/Docs/<tdoc_id>); archive clutter is filtered by the user-editable clutter-ignore patterns.

Main-file choice is delegated to :func:tdoc_crawler.tdocs.operations.main_file.resolve_main_file: manual override first, then the optional decision model, then the offline heuristic. Call :func:prepare_tdoc_selection instead when the source of the choice matters to the caller.

Parameters:

  • metadata

    (TDocMetadata) –

    TDoc metadata with download URL.

  • checkout_dir

    (Path) –

    Checkout directory (base of the mirror tree).

  • return_dir

    (bool, default: False ) –

    When True, return the extract directory itself instead of the picked "main file" inside it.

  • session

    (Session | None, default: None ) –

    Optional requests.Session to reuse for downloads.

  • override_member

    (str | None, default: None ) –

    Stored manual override (archive-relative path).

  • config

    (CheckoutConfig | None, default: None ) –

    Checkout settings; defaults to CheckoutConfig().

  • extract

    (ExtractConfig | None, default: None ) –

    Extraction settings; defaults to ExtractConfig().

Returns:

  • Path –

    Path to the picked main file, or the extract directory when

  • Path –

    return_dir is True. For non-zip URLs, returns the downloaded

  • Path –

    file path.

Raises:

  • FileNotFoundError –

    If the TDoc is withdrawn, the download fails, or the archive contains no usable file.

  • ValueError –

    If metadata.url is missing.

Source code in src/tdoc_crawler/tdocs/operations/checkout.py
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
def prepare_tdoc_file(
    metadata: TDocMetadata,
    checkout_dir: Path,
    return_dir: bool = False,
    session: requests.Session | None = None,
    *,
    override_member: str | None = None,
    config: CheckoutConfig | None = None,
    extract: ExtractConfig | None = None,
) -> Path:
    """Prepare the local file for a TDoc, downloading/extracting if needed.

    Routes through :func:`checkout_tdoc` so that ``tdoc open`` and
    ``tdoc checkout`` share a single business-logic path: the on-disk
    layout mirrors the 3GPP server (under
    ``tsg_<tsg>/<wg>/<meeting>/Docs/<tdoc_id>``); archive clutter is
    filtered by the user-editable ``clutter-ignore`` patterns.

    Main-file choice is delegated to
    :func:`tdoc_crawler.tdocs.operations.main_file.resolve_main_file`:
    manual override first, then the optional decision model, then the
    offline heuristic. Call :func:`prepare_tdoc_selection` instead when
    the source of the choice matters to the caller.

    Args:
        metadata: TDoc metadata with download URL.
        checkout_dir: Checkout directory (base of the mirror tree).
        return_dir: When True, return the extract directory itself instead
            of the picked "main file" inside it.
        session: Optional requests.Session to reuse for downloads.
        override_member: Stored manual override (archive-relative path).
        config: Checkout settings; defaults to ``CheckoutConfig()``.
        extract: Extraction settings; defaults to ``ExtractConfig()``.

    Returns:
        Path to the picked main file, or the extract directory when
        ``return_dir`` is True. For non-zip URLs, returns the downloaded
        file path.

    Raises:
        FileNotFoundError: If the TDoc is withdrawn, the download fails,
            or the archive contains no usable file.
        ValueError: If ``metadata.url`` is missing.
    """
    if return_dir:
        filename, extract_dir = _resolve_archive(metadata, checkout_dir, session=session, extract=extract)
        if extract_dir is not None:
            return extract_dir
        return _download_loose_tdoc(metadata, checkout_dir, filename, session=session)

    return prepare_tdoc_selection(
        metadata,
        checkout_dir,
        override_member=override_member,
        config=config,
        session=session,
        extract=extract,
    ).path

prepare_tdoc_selection

prepare_tdoc_selection(metadata: TDocMetadata, checkout_dir: Path, *, override_member: str | None = None, config: CheckoutConfig | None = None, session: Session | None = None, extract: ExtractConfig | None = None) -> MainFileSelection

Prepare a TDoc and report which file was chosen as its main file.

Same download/extract path as :func:prepare_tdoc_file; use this variant when the caller must show or log the selection source (manual override, decision model, or heuristic).

Parameters:

  • metadata

    (TDocMetadata) –

    TDoc metadata with download URL.

  • checkout_dir

    (Path) –

    Checkout directory (base of the mirror tree).

  • override_member

    (str | None, default: None ) –

    Stored manual override (archive-relative path).

  • config

    (CheckoutConfig | None, default: None ) –

    Checkout settings; defaults to CheckoutConfig().

  • session

    (Session | None, default: None ) –

    Optional requests.Session to reuse for downloads.

  • extract

    (ExtractConfig | None, default: None ) –

    Extraction settings; defaults to ExtractConfig().

Returns:

  • MainFileSelection –

    The resolved main file plus its source.

Raises:

  • FileNotFoundError –

    If the TDoc is withdrawn, the download fails, or the archive contains no usable (or no overridden) file.

  • ValueError –

    If metadata.url is missing.

Source code in src/tdoc_crawler/tdocs/operations/checkout.py
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
def prepare_tdoc_selection(
    metadata: TDocMetadata,
    checkout_dir: Path,
    *,
    override_member: str | None = None,
    config: CheckoutConfig | None = None,
    session: requests.Session | None = None,
    extract: ExtractConfig | None = None,
) -> MainFileSelection:
    """Prepare a TDoc and report *which* file was chosen as its main file.

    Same download/extract path as :func:`prepare_tdoc_file`; use this
    variant when the caller must show or log the selection source
    (manual override, decision model, or heuristic).

    Args:
        metadata: TDoc metadata with download URL.
        checkout_dir: Checkout directory (base of the mirror tree).
        override_member: Stored manual override (archive-relative path).
        config: Checkout settings; defaults to ``CheckoutConfig()``.
        session: Optional requests.Session to reuse for downloads.
        extract: Extraction settings; defaults to ``ExtractConfig()``.

    Returns:
        The resolved main file plus its source.

    Raises:
        FileNotFoundError: If the TDoc is withdrawn, the download fails,
            or the archive contains no usable (or no overridden) file.
        ValueError: If ``metadata.url`` is missing.
    """
    filename, extract_dir = _resolve_archive(metadata, checkout_dir, session=session, extract=extract)
    if extract_dir is None:
        # A loose (non-zip) TDoc *is* its own main file; the heuristic
        # source is reported because no choice was made.
        downloaded = _download_loose_tdoc(metadata, checkout_dir, filename, session=session)
        return MainFileSelection(
            path=downloaded,
            relative_path=downloaded.name,
            source=MainFileSource.HEURISTIC,
        )

    return resolve_main_file(
        extract_dir,
        metadata.tdoc_id,
        title=metadata.title,
        override_member=override_member,
        config=config,
        session=session,
    )

reclassify_tdocs async

reclassify_tdocs(*, db_file: Path, checkout_dir: Path, checkout_config: CheckoutConfig, tdoc_ids: list[str] | None = None, include_manual: bool = False) -> list[ReclassifyOutcome]

Re-run main-file classification on stored TDocs with a checkout.

Parameters:

  • db_file

    (Path) –

    Path to the metadata database.

  • checkout_dir

    (Path) –

    Root directory containing extracted TDoc archives.

  • checkout_config

    (CheckoutConfig) –

    Main-document selection settings.

  • tdoc_ids

    (list[str] | None, default: None ) –

    Specific TDocs to re-classify; None means every stored classification.

  • include_manual

    (bool, default: False ) –

    Also re-classify TDocs with a manual selector or a stored override, deleting that override. Destructive.

Returns:

Source code in src/tdoc_crawler/tdocs/operations/reclassify.py
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
async def reclassify_tdocs(
    *,
    db_file: Path,
    checkout_dir: Path,
    checkout_config: CheckoutConfig,
    tdoc_ids: list[str] | None = None,
    include_manual: bool = False,
) -> list[ReclassifyOutcome]:
    """Re-run main-file classification on stored TDocs with a checkout.

    Args:
        db_file: Path to the metadata database.
        checkout_dir: Root directory containing extracted TDoc archives.
        checkout_config: Main-document selection settings.
        tdoc_ids: Specific TDocs to re-classify; None means every stored
            classification.
        include_manual: Also re-classify TDocs with a manual selector or a
            stored override, deleting that override. Destructive.

    Returns:
        One outcome per requested TDoc, in request order.
    """
    async with TDocDatabase(db_file) as database:
        records = {record.tdoc_id: record for record in await list_main_file_records()}
        targets = [normalize_tdoc_id(tdoc_id) for tdoc_id in tdoc_ids] if tdoc_ids else sorted(records)

        if not targets:
            return []

        metadata_rows = await database.query_tdocs(TDocQueryConfig(tdoc_ids=targets))
        metadata_by_id: dict[str, TDocMetadata] = {row.tdoc_id: row for row in metadata_rows}

        outcomes: list[ReclassifyOutcome] = []
        for tdoc_id in targets:
            outcome = await _reclassify_one(
                database,
                tdoc_id,
                records.get(tdoc_id),
                metadata_by_id.get(tdoc_id),
                checkout_dir=checkout_dir,
                checkout_config=checkout_config,
                include_manual=include_manual,
            )
            outcomes.append(outcome)

        return outcomes

refresh_tdocs async

refresh_tdocs(database: TDocDatabase, *, lookback_days: int = 14, working_groups: list[WorkingGroup] | None = None, subgroups: list[str] | None = None, workers: int = 4, timeout: int = 30, max_retries: int = 3, repair_urls: bool = True, resolve_url: UrlResolver = _resolve_corrected_url, progress_callback: Callable[[float, float], None] | None = None, today: date | None = None) -> RefreshResult

Re-fetch complete metadata for TDocs of meetings in the lookback window.

Parameters:

  • database

    (TDocDatabase) –

    Open database used for snapshots and upserts.

  • lookback_days

    (int, default: 14 ) –

    Include meetings that started within this many days.

  • working_groups

    (list[WorkingGroup] | None, default: None ) –

    Restrict to these groups (default: all RAN/SA/CT).

  • subgroups

    (list[str] | None, default: None ) –

    Optional sub-working-group filter.

  • workers

    (int, default: 4 ) –

    Parallel workers for the document-list crawl and URL repair.

  • timeout

    (int, default: 30 ) –

    Per-request HTTP timeout in seconds.

  • max_retries

    (int, default: 3 ) –

    Retry attempts for failed meetings.

  • repair_urls

    (bool, default: True ) –

    Resolve and persist corrected download URLs.

  • resolve_url

    (UrlResolver, default: _resolve_corrected_url ) –

    URL resolver (injectable for tests).

  • progress_callback

    (Callable[[float, float], None] | None, default: None ) –

    Optional (completed, total) crawl progress callback.

  • today

    (date | None, default: None ) –

    Override for date.today() in tests.

Returns:

  • A ( RefreshResult ) –

    class:RefreshResult with per-meeting change details.

Source code in src/tdoc_crawler/tdocs/operations/refresh.py
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
async def refresh_tdocs(
    database: TDocDatabase,
    *,
    lookback_days: int = 14,
    working_groups: list[WorkingGroup] | None = None,
    subgroups: list[str] | None = None,
    workers: int = 4,
    timeout: int = 30,
    max_retries: int = 3,
    repair_urls: bool = True,
    resolve_url: UrlResolver = _resolve_corrected_url,
    progress_callback: Callable[[float, float], None] | None = None,
    today: date | None = None,
) -> RefreshResult:
    """Re-fetch complete metadata for TDocs of meetings in the lookback window.

    Args:
        database: Open database used for snapshots and upserts.
        lookback_days: Include meetings that started within this many days.
        working_groups: Restrict to these groups (default: all RAN/SA/CT).
        subgroups: Optional sub-working-group filter.
        workers: Parallel workers for the document-list crawl and URL repair.
        timeout: Per-request HTTP timeout in seconds.
        max_retries: Retry attempts for failed meetings.
        repair_urls: Resolve and persist corrected download URLs.
        resolve_url: URL resolver (injectable for tests).
        progress_callback: Optional (completed, total) crawl progress callback.
        today: Override for ``date.today()`` in tests.

    Returns:
        A :class:`RefreshResult` with per-meeting change details.
    """
    if lookback_days < 1:
        msg = "lookback_days must be at least 1"
        raise ValueError(msg)

    start_date = (today or date.today()) - timedelta(days=lookback_days)
    config = TDocCrawlConfig(
        working_groups=working_groups,
        subgroups=subgroups,
        incremental=False,
        start_date=start_date,
        workers=workers,
        timeout=timeout,
        max_retries=max_retries,
    )

    crawler = TDocCrawler(database)
    meetings = await crawler.get_meetings_to_crawl(config)
    meeting_ids = [meeting.meeting_id for meeting in meetings]

    before_records = await database.query_tdocs_by_meeting_ids(meeting_ids)
    before = {record.tdoc_id: record for record in before_records}

    crawl_result = await crawler.crawl(config, progress_callback=progress_callback)

    after_records = await database.query_tdocs_by_meeting_ids(meeting_ids)
    after = {record.tdoc_id: record for record in after_records}

    repaired: dict[str, str] = {}
    if repair_urls and after_records:
        repaired = _repair_urls(after_records, resolve_url, workers)
        for tdoc_id, corrected_url in repaired.items():
            await database.update_tdoc_url(tdoc_id, corrected_url)
        if repaired:
            # Reload so the change report reflects persisted URLs.
            after_records = await database.query_tdocs_by_meeting_ids(meeting_ids)
            after = {record.tdoc_id: record for record in after_records}

    meeting_summaries: list[MeetingRefreshSummary] = []
    for meeting in meetings:
        meeting_before = {record.tdoc_id: record for record in before_records if record.meeting_id == meeting.meeting_id}
        meeting_after = {record.tdoc_id: record for record in after_records if record.meeting_id == meeting.meeting_id}

        added = tuple(sorted(set(meeting_after) - set(meeting_before)))
        absent = tuple(sorted(set(meeting_before) - set(meeting_after)))
        changed: list[TDocChange] = []
        unchanged = 0
        for tdoc_id in sorted(set(meeting_before) & set(meeting_after)):
            field_changes = diff_tdoc(meeting_before[tdoc_id], meeting_after[tdoc_id])
            url_repaired = tdoc_id in repaired
            if field_changes or url_repaired:
                changed.append(TDocChange(tdoc_id=tdoc_id, url_repaired=url_repaired, field_changes=field_changes))
            else:
                unchanged += 1

        short_name = meeting.short_name or str(meeting.meeting_id)
        meeting_errors = tuple(error for error in crawl_result.errors if short_name in error)

        meeting_summaries.append(
            MeetingRefreshSummary(
                meeting_id=meeting.meeting_id,
                short_name=short_name,
                tdocs_before=len(meeting_before),
                tdocs_after=len(meeting_after),
                added=added,
                changed=tuple(changed),
                unchanged=unchanged,
                absent_from_list=absent,
                errors=meeting_errors,
            )
        )

    return RefreshResult(
        lookback_days=lookback_days,
        start_date=start_date,
        meetings_scanned=len(meetings),
        meetings_with_changes=sum(1 for summary in meeting_summaries if summary.has_changes),
        tdocs_processed=crawl_result.processed,
        tdocs_added=len(set(after) - set(before)),
        tdocs_changed=sum(len(summary.changed) for summary in meeting_summaries),
        tdocs_unchanged=sum(summary.unchanged for summary in meeting_summaries),
        urls_repaired=len(repaired),
        meetings=tuple(meeting_summaries),
        errors=tuple(crawl_result.errors),
    )

resolve_main_file

resolve_main_file(extract_dir: Path, tdoc_id: str, *, title: str | None = None, override_member: str | None = None, config: CheckoutConfig | None = None, session: Session | None = None) -> MainFileSelection

Resolve the main file of an extracted TDoc archive.

Precedence: manual override > decision model (when configured and confident) > heuristic.

Parameters:

  • extract_dir

    (Path) –

    Directory holding the extracted TDoc contents.

  • tdoc_id

    (str) –

    TDoc identifier, used by the heuristic and the model.

  • title

    (str | None, default: None ) –

    Optional portal title, used by the heuristic and the model.

  • override_member

    (str | None, default: None ) –

    Stored manual override (archive-relative path).

  • config

    (CheckoutConfig | None, default: None ) –

    Checkout settings; defaults to CheckoutConfig().

  • session

    (Session | None, default: None ) –

    Optional HTTP session reused for the decision-model call.

Returns:

  • MainFileSelection –

    The resolved main file together with its source.

Raises:

  • FileNotFoundError –

    If no usable file is present, or the override no longer exists in the archive.

Source code in src/tdoc_crawler/tdocs/operations/main_file.py
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
def resolve_main_file(
    extract_dir: Path,
    tdoc_id: str,
    *,
    title: str | None = None,
    override_member: str | None = None,
    config: CheckoutConfig | None = None,
    session: requests.Session | None = None,
) -> MainFileSelection:
    """Resolve the main file of an extracted TDoc archive.

    Precedence: manual override > decision model (when configured and
    confident) > heuristic.

    Args:
        extract_dir: Directory holding the extracted TDoc contents.
        tdoc_id: TDoc identifier, used by the heuristic and the model.
        title: Optional portal title, used by the heuristic and the model.
        override_member: Stored manual override (archive-relative path).
        config: Checkout settings; defaults to ``CheckoutConfig()``.
        session: Optional HTTP session reused for the decision-model call.

    Returns:
        The resolved main file together with its source.

    Raises:
        FileNotFoundError: If no usable file is present, or the override
            no longer exists in the archive.
    """
    settings = config or CheckoutConfig()

    if override_member is not None:
        overridden = _resolve_override(extract_dir, override_member)
        return MainFileSelection(
            path=overridden,
            relative_path=overridden.relative_to(extract_dir).as_posix(),
            source=MainFileSource.MANUAL_OVERRIDE,
        )

    if settings.selector is MainFileSelector.MANUAL:
        raise FileNotFoundError(f"no main file override stored for {tdoc_id}; selector 'manual' disables heuristic and decision-model selection")

    if settings.selector is MainFileSelector.DECISION_MODEL and settings.systemone_api_key:
        decision = _decide_with_model(extract_dir, tdoc_id, title, settings, session)
        if decision is not None:
            relative, confidence = decision
            return MainFileSelection(
                path=extract_dir / relative,
                relative_path=relative,
                source=MainFileSource.DECISION_MODEL,
                confidence=confidence,
            )

    picked = select_heuristic_main_file(extract_dir, tdoc_id, title=title)
    return MainFileSelection(
        path=picked,
        relative_path=picked.relative_to(extract_dir).as_posix(),
        source=MainFileSource.HEURISTIC,
    )

resolve_tdoc_main_file async

resolve_tdoc_main_file(tdoc_id: str, *, db_file: Path, checkout_dir: Path, checkout_config: CheckoutConfig, full_metadata: bool = False, use_whatthespec: bool = False, session: Session | None = None, extract_config: ExtractConfig | None = None) -> TDocOpenResult

Ensure a TDoc is stored, extract it, and resolve its main document.

Parameters:

  • tdoc_id

    (str) –

    TDoc identifier (normalized before lookup).

  • db_file

    (Path) –

    Path to the metadata database.

  • checkout_dir

    (Path) –

    Root directory for TDoc archives.

  • checkout_config

    (CheckoutConfig) –

    Main-document selection settings.

  • full_metadata

    (bool, default: False ) –

    If True, fetch full metadata from the portal.

  • use_whatthespec

    (bool, default: False ) –

    If True, always use WhatTheSpec for missing TDocs.

  • session

    (Session | None, default: None ) –

    Optional HTTP session to reuse.

  • extract_config

    (ExtractConfig | None, default: None ) –

    Extraction settings; defaults to ExtractConfig().

Returns:

  • TDocOpenResult –

    The TDoc metadata and the resolved main-document selection.

Raises:

Source code in src/tdoc_crawler/tdocs/operations/open.py
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
async def resolve_tdoc_main_file(
    tdoc_id: str,
    *,
    db_file: Path,
    checkout_dir: Path,
    checkout_config: CheckoutConfig,
    full_metadata: bool = False,
    use_whatthespec: bool = False,
    session: requests.Session | None = None,
    extract_config: ExtractConfig | None = None,
) -> TDocOpenResult:
    """Ensure a TDoc is stored, extract it, and resolve its main document.

    Args:
        tdoc_id: TDoc identifier (normalized before lookup).
        db_file: Path to the metadata database.
        checkout_dir: Root directory for TDoc archives.
        checkout_config: Main-document selection settings.
        full_metadata: If True, fetch full metadata from the portal.
        use_whatthespec: If True, always use WhatTheSpec for missing TDocs.
        session: Optional HTTP session to reuse.
        extract_config: Extraction settings; defaults to ``ExtractConfig()``.

    Returns:
        The TDoc metadata and the resolved main-document selection.

    Raises:
        TDocNotFoundError: If the TDoc cannot be found or fetched.
        FileNotFoundError: If the stored override or the archive is invalid.
        OSError: If the archive cannot be written.
        ValueError: If the TDoc metadata is unusable.
    """
    normalized_id = normalize_tdoc_id(tdoc_id)
    query_config = TDocQueryConfig(tdoc_ids=[normalized_id])

    async with TDocDatabase(db_file) as database:
        results = await database.query_tdocs(query_config)
        fetch_result = await fetch_missing_tdocs(
            database,
            query_config,
            results,
            full_metadata=full_metadata,
            use_whatthespec=use_whatthespec,
            session=session,
            url_only_first=True,
        )

        fetch_errors: tuple[str, ...] = ()
        if fetch_result.fetch_result is not None and fetch_result.fetch_result.errors:
            fetch_errors = tuple(str(error) for error in fetch_result.fetch_result.errors)

        refreshed = fetch_result.refreshed
        if not refreshed:
            msg = f"TDoc {normalized_id} not found"
            raise TDocNotFoundError(msg)

        metadata = refreshed[0]
        override = await database.get_main_file_override(normalized_id)

    selection = prepare_tdoc_selection(
        metadata,
        checkout_dir,
        override_member=override.member_path if override is not None else None,
        config=checkout_config,
        session=session,
        extract=extract_config,
    )

    selector = MainFileSelector.MANUAL if override is not None else checkout_config.selector
    async with TDocDatabase(db_file):
        await save_main_file_record(
            metadata.tdoc_id,
            member_path=selection.relative_path,
            source=selection.source.value,
            selector=selector.value,
            confidence=selection.confidence,
        )

    return TDocOpenResult(metadata=metadata, selection=selection, fetch_errors=fetch_errors)

select_heuristic_main_file

select_heuristic_main_file(extract_dir: Path, tdoc_id: str, *, title: str | None = None) -> Path

Pick the most likely "main" file inside an extracted TDoc directory.

Candidates are filtered by the gitignore-style clutter patterns (<cache-dir>/clutter-ignore, see :mod:tdoc_crawler.utils.clutter) so that OS metadata (__MACOSX/, ._*, ~$* ...) is never picked — even from checkout directories created before extraction filtering existed.

When an archive holds two versions of the same document (one with, one without revision marks), the tracked-changes version wins — decided by inspecting the document content via :func:has_revision_marks, never by filename conventions.

Among the remaining document-suffix files the best-ranked candidate wins, by these criteria in order:

1. Name is exactly the TDoc ID (e.g. ``S4-261599.docx``).
2. Highest similarity to ``<tdoc_id> <title>`` (only when *title*
   is given; scores below the threshold count as zero).
3. Name starts with the TDoc ID (case-insensitive).
4. Document carries revision marks (content probe, ``.docx`` only).
5. Document type, in ``DocumentType`` declaration order.
6. Most recent file timestamp, then name (deterministic tie-break).

Fallbacks: largest remaining document-suffix file, then largest non-clutter file of any type.

Parameters:

  • extract_dir

    (Path) –

    Directory holding the extracted TDoc contents.

  • tdoc_id

    (str) –

    TDoc identifier (e.g. S4-261599).

  • title

    (str | None, default: None ) –

    Optional portal title of the TDoc, used for fuzzy matching.

Returns:

  • Path –

    Path to the selected main file.

Raises:

Source code in src/tdoc_crawler/tdocs/operations/main_file.py
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
def select_heuristic_main_file(extract_dir: Path, tdoc_id: str, *, title: str | None = None) -> Path:
    """Pick the most likely "main" file inside an extracted TDoc directory.

    Candidates are filtered by the gitignore-style clutter patterns
    (``<cache-dir>/clutter-ignore``, see :mod:`tdoc_crawler.utils.clutter`)
    so that OS metadata (``__MACOSX/``, ``._*``, ``~$*`` ...) is never
    picked — even from checkout directories created before extraction
    filtering existed.

    When an archive holds two versions of the same document (one with,
    one without revision marks), the tracked-changes version wins —
    decided by inspecting the document *content* via
    :func:`has_revision_marks`, never by filename conventions.

    Among the remaining document-suffix files the best-ranked candidate
    wins, by these criteria in order:

        1. Name is exactly the TDoc ID (e.g. ``S4-261599.docx``).
        2. Highest similarity to ``<tdoc_id> <title>`` (only when *title*
           is given; scores below the threshold count as zero).
        3. Name starts with the TDoc ID (case-insensitive).
        4. Document carries revision marks (content probe, ``.docx`` only).
        5. Document type, in ``DocumentType`` declaration order.
        6. Most recent file timestamp, then name (deterministic tie-break).

    Fallbacks: largest remaining document-suffix file, then largest
    non-clutter file of any type.

    Args:
        extract_dir: Directory holding the extracted TDoc contents.
        tdoc_id: TDoc identifier (e.g. ``S4-261599``).
        title: Optional portal title of the TDoc, used for fuzzy matching.

    Returns:
        Path to the selected main file.

    Raises:
        FileNotFoundError: If no usable file is present.
    """
    documents = [
        p
        for p in extract_dir.rglob("*")
        if p.is_file() and p.suffix.lower() in _TDOC_DOCUMENT_SUFFIXES and not is_clutter(p.relative_to(extract_dir).as_posix())
    ]
    if documents:
        if len(documents) == 1:
            return documents[0]
        id_folded = tdoc_id.casefold()
        title_folded = re.sub(r"\W+", " ", f"{tdoc_id} {title or ''}".casefold()).strip()

        def rank(path: Path) -> tuple[bool, float, bool, bool, int, float, str]:
            stem_folded = path.stem.casefold()
            exact = stem_folded == id_folded
            title_score = 0.0
            if title:
                ratio = difflib.SequenceMatcher(None, stem_folded, title_folded).ratio()
                if ratio >= _TITLE_MATCH_THRESHOLD:
                    title_score = round(ratio, 3)
            prefixed = stem_folded.startswith(id_folded)
            marked = has_revision_marks(path)
            type_rank = -_TDOC_DOCUMENT_TYPE_ORDER.index(path.suffix.lower())
            return (exact, title_score, prefixed, marked, type_rank, path.stat().st_mtime, path.name)

        return max(documents, key=rank)

    all_files = [p for p in extract_dir.rglob("*") if p.is_file() and not is_clutter(p.relative_to(extract_dir).as_posix())]
    if not all_files:
        msg = "no-files-in-archive"
        raise FileNotFoundError(msg)
    return max(all_files, key=lambda p: p.stat().st_size)

Main-file overrides

Stored overrides (user-owned) and classification records (computed memory).

main_files

Per-TDoc manual main-file overrides.

An override records which archive member the user considers the main document of a TDoc. It always wins over automatic selection (heuristic ranking or the decision model) and survives re-download and re-extraction because it is keyed by the archive member path.

clear_main_file_override async

clear_main_file_override(tdoc_id: str) -> bool

Delete the override of one TDoc. Returns True when one existed.

Source code in src/tdoc_crawler/database/main_files.py
41
42
43
44
45
46
47
async def clear_main_file_override(tdoc_id: str) -> bool:
    """Delete the override of one TDoc. Returns True when one existed."""
    existing = await get_main_file_override(tdoc_id)
    if existing is None:
        return False
    await existing.delete()
    return True

clear_main_file_record async

clear_main_file_record(tdoc_id: str) -> bool

Delete the stored classification of one TDoc. Returns True when one existed.

Source code in src/tdoc_crawler/database/main_files.py
85
86
87
88
89
90
91
async def clear_main_file_record(tdoc_id: str) -> bool:
    """Delete the stored classification of one TDoc. Returns True when one existed."""
    existing = await get_main_file_record(tdoc_id)
    if existing is None:
        return False
    await existing.delete()
    return True

get_main_file_override async

get_main_file_override(tdoc_id: str) -> MainFileOverride | None

Return the stored override for one TDoc, or None.

Source code in src/tdoc_crawler/database/main_files.py
50
51
52
async def get_main_file_override(tdoc_id: str) -> MainFileOverride | None:
    """Return the stored override for one TDoc, or None."""
    return await MainFileOverride.objects.filter(tdoc_id=tdoc_id).first()

get_main_file_record async

get_main_file_record(tdoc_id: str) -> MainFileRecord | None

Return the last stored classification for one TDoc, or None.

Source code in src/tdoc_crawler/database/main_files.py
94
95
96
async def get_main_file_record(tdoc_id: str) -> MainFileRecord | None:
    """Return the last stored classification for one TDoc, or None."""
    return await MainFileRecord.objects.filter(tdoc_id=tdoc_id).first()

list_main_file_overrides async

list_main_file_overrides() -> list[MainFileOverride]

All stored overrides, sorted by TDoc ID.

Source code in src/tdoc_crawler/database/main_files.py
55
56
57
58
async def list_main_file_overrides() -> list[MainFileOverride]:
    """All stored overrides, sorted by TDoc ID."""
    rows = await MainFileOverride.objects.all()
    return sorted(rows, key=lambda row: row.tdoc_id)

list_main_file_records async

list_main_file_records() -> list[MainFileRecord]

All stored classifications, sorted by TDoc ID.

Source code in src/tdoc_crawler/database/main_files.py
 99
100
101
102
async def list_main_file_records() -> list[MainFileRecord]:
    """All stored classifications, sorted by TDoc ID."""
    rows = await MainFileRecord.objects.all()
    return sorted(rows, key=lambda row: row.tdoc_id)

save_main_file_record async

save_main_file_record(tdoc_id: str, *, member_path: str, source: str, selector: str, confidence: float | None = None) -> MainFileRecord

Create or replace the stored classification of one TDoc.

Source code in src/tdoc_crawler/database/main_files.py
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
async def save_main_file_record(
    tdoc_id: str,
    *,
    member_path: str,
    source: str,
    selector: str,
    confidence: float | None = None,
) -> MainFileRecord:
    """Create or replace the stored classification of one TDoc."""
    existing = await get_main_file_record(tdoc_id)
    if existing is not None:
        await existing.delete()
    record = MainFileRecord(
        tdoc_id=tdoc_id,
        member_path=member_path,
        source=source,
        selector=selector,
        confidence=confidence,
        updated_at=utc_now(),
    )
    await MainFileRecord.objects.create(instance=record)
    return record

set_main_file_override async

set_main_file_override(tdoc_id: str, member_path: str, *, note: str | None = None) -> MainFileOverride

Create or replace the manual main-file override of one TDoc.

Parameters:

  • tdoc_id

    (str) –

    TDoc identifier (normalized by the caller).

  • member_path

    (str) –

    Archive-relative POSIX path of the main file.

  • note

    (str | None, default: None ) –

    Optional free-text note explaining the choice.

Raises:

  • ValueError –

    If member_path is empty, absolute, or escapes the archive directory.

Source code in src/tdoc_crawler/database/main_files.py
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
async def set_main_file_override(tdoc_id: str, member_path: str, *, note: str | None = None) -> MainFileOverride:
    """Create or replace the manual main-file override of one TDoc.

    Args:
        tdoc_id: TDoc identifier (normalized by the caller).
        member_path: Archive-relative POSIX path of the main file.
        note: Optional free-text note explaining the choice.

    Raises:
        ValueError: If ``member_path`` is empty, absolute, or escapes the
            archive directory.
    """
    candidate = PurePosixPath(member_path)
    if not member_path or candidate.is_absolute() or ".." in candidate.parts:
        msg = f"invalid-member-path: {member_path!r} must be relative to the archive root"
        raise ValueError(msg)
    existing = await get_main_file_override(tdoc_id)
    if existing is not None:
        await existing.delete()
    override = MainFileOverride(tdoc_id=tdoc_id, member_path=member_path, note=note, created_at=utc_now())
    await MainFileOverride.objects.create(instance=override)
    return override

Tag operations

Shared by CLI and MCP — AND-of-ORs tag expressions.

tags

Tag operations for TDocs, specs, and other items.

Tags are the lightweight grouping mechanism (replacing the removed workspace subsystem). A tag row points at an item via kind + item_id — no foreign key; integrity is app-layer.

Filter semantics shared by CLI and MCP: a tag group is a set of tags combined with OR (--tag a,b); repeated groups combine with AND (--tag a,b --tag c matches items having (a or b) and c).

add_tags async

add_tags(kind: TagKind, item_id: str, tags: list[str]) -> int

Add tags to an item. Returns number of new tag rows (existing skipped).

Source code in src/tdoc_crawler/database/tags.py
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
async def add_tags(kind: TagKind, item_id: str, tags: list[str]) -> int:
    """Add tags to an item. Returns number of new tag rows (existing skipped)."""
    existing = {row.tag for row in await TagEntry.objects.filter(kind=kind.value, item_id=item_id).all()}
    added = 0
    for tag in normalize_tags(tags):
        if tag in existing:
            continue
        # oxyde create() excludes unset fields; set defaulted fields explicitly
        await TagEntry.objects.create(
            instance=TagEntry(
                tag_id=str(uuid4()),
                kind=kind.value,
                item_id=item_id,
                tag=tag,
                created_at=utc_now(),
            )
        )
        existing.add(tag)
        added += 1
    return added

item_ids_for_tag_groups async

item_ids_for_tag_groups(groups: list[list[str]], kind: TagKind) -> set[str]

Item ids matching the AND-of-ORs tag expression for kind.

Empty groups returns an empty set (no filter applied by callers).

Source code in src/tdoc_crawler/database/tags.py
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
async def item_ids_for_tag_groups(groups: list[list[str]], kind: TagKind) -> set[str]:
    """Item ids matching the AND-of-ORs tag expression for *kind*.

    Empty *groups* returns an empty set (no filter applied by callers).
    """
    if not groups:
        return set()
    result: set[str] | None = None
    for group in groups:
        group_ids: set[str] = set()
        for tag in group:
            rows = await TagEntry.objects.filter(kind=kind.value, tag=tag).all()
            group_ids.update(row.item_id for row in rows)
        result = group_ids if result is None else result & group_ids
        if not result:
            return set()
    return result or set()

list_tags async

list_tags(kind: TagKind | None = None, item_id: str | None = None, tag: str | None = None) -> list[TagEntry]

List tag rows, optionally filtered by kind, item, and/or tag.

Source code in src/tdoc_crawler/database/tags.py
73
74
75
76
77
78
79
80
81
82
83
84
85
async def list_tags(
    kind: TagKind | None = None,
    item_id: str | None = None,
    tag: str | None = None,
) -> list[TagEntry]:
    """List tag rows, optionally filtered by kind, item, and/or tag."""
    rows = await TagEntry.objects.filter(kind=kind.value).all() if kind is not None else await TagEntry.objects.all()
    if item_id is not None:
        rows = [row for row in rows if row.item_id == item_id]
    if tag is not None:
        wanted = tag.strip().lower()
        rows = [row for row in rows if row.tag == wanted]
    return sorted(rows, key=lambda r: (r.kind, r.item_id, r.tag))

normalize_tags

normalize_tags(tags: list[str]) -> list[str]

Strip, drop empties, dedupe (order-preserving).

Source code in src/tdoc_crawler/database/tags.py
61
62
63
64
65
66
67
68
69
70
def normalize_tags(tags: list[str]) -> list[str]:
    """Strip, drop empties, dedupe (order-preserving)."""
    seen: set[str] = set()
    result: list[str] = []
    for tag in tags:
        cleaned = tag.strip().lower()
        if cleaned and cleaned not in seen:
            seen.add(cleaned)
            result.append(cleaned)
    return result

parse_tag_groups

parse_tag_groups(values: list[str]) -> list[list[str]]

Parse CLI --tag values into OR-groups: comma-split each value.

Source code in src/tdoc_crawler/database/tags.py
21
22
23
def parse_tag_groups(values: list[str]) -> list[list[str]]:
    """Parse CLI ``--tag`` values into OR-groups: comma-split each value."""
    return [normalize_tags(v.split(",")) for v in values if v and v.strip()]

remove_tags async

remove_tags(kind: TagKind, item_id: str, tags: list[str] | None = None) -> int

Remove tags from an item (all tags if tags is None). Returns removed count.

Source code in src/tdoc_crawler/database/tags.py
48
49
50
51
52
53
54
55
56
57
58
async def remove_tags(kind: TagKind, item_id: str, tags: list[str] | None = None) -> int:
    """Remove tags from an item (all tags if *tags* is None). Returns removed count."""
    rows = await TagEntry.objects.filter(kind=kind.value, item_id=item_id).all()
    targets = normalize_tags(tags) if tags is not None else [row.tag for row in rows]
    target_set = set(targets)
    removed = 0
    for row in rows:
        if row.tag in target_set:
            await row.delete()
            removed += 1
    return removed

Configuration

settings

Configuration settings for 3GPP Crawler using pydantic-settings.

This module provides the ThreeGPPConfig class with nested models for: - PathConfig: File system paths (cache, checkout) - HttpConfig: HTTP caching and request behavior - CredentialsConfig: Portal authentication credentials - CrawlConfig: Crawling behavior and filters

All paths default to sensible defaults but can be overridden via environment variables with the TDC_ prefix.

CheckoutConfig

Bases: BaseSettings

Checkout behaviour: how the main file of a TDoc archive is picked.

A stored manual override always wins; selector only controls the automatic fallback chain (heuristic ranking, or the optional System One decision model before it).

CrawlConfig

Bases: BaseSettings

Crawling behavior and filter configuration.

Controls which documents are crawled based on working groups, date ranges, and other filters.

CredentialsConfig

Bases: BaseSettings

Portal authentication credentials.

Stores credentials for accessing protected resources like the 3GPP portal.

has_credentials property

has_credentials: bool

Check if both username and password are set.

ExtractConfig

Bases: BaseModel

Archive extraction behaviour for downloaded TDoc zips.

Config-file only by design: no CLI option and no environment variable is generated for these settings, because they describe how files land on disk rather than how a single run behaves.

Exclusion patterns use gitignore syntax and layer on top of the <cache-dir>/clutter-ignore file (macOS/Windows/Office clutter).

HttpConfig

Bases: BaseSettings

HTTP client and caching configuration.

Controls HTTP request behavior including timeouts, retries, and caching behavior for web requests.

PathConfig

Bases: BaseSettings

File system path configuration.

Provides paths for cache directory, database, and checkout. All paths support ~ expansion and are resolved to absolute paths.

checkout_dir property

checkout_dir: Path

Path to the checkout directory for documents.

db_file property

db_file: Path

Path to the SQLite database file.

http_cache_file property

http_cache_file: Path

Path to the HTTP cache database file.

ThreeGPPConfig

Bases: BaseSettings

Main configuration class for 3GPP Crawler.

Aggregates all configuration sub-models: - path: File system paths - http: HTTP client and caching behavior - credentials: Portal authentication - crawl: Crawling filters and limits - checkout: Main-file selection strategy and decision-model settings - extract: Archive extraction (flattening and skip patterns), config file only

Configuration can be overridden via environment variables with the appropriate prefixes (TDC_, TDC_EOL_, TDC_CRAWL_, TDC_CHECKOUT_).

ensure_paths

ensure_paths() -> None

Ensure all configured paths exist on the filesystem.

Creates parent directories as needed.

Source code in src/tdoc_crawler/config/settings.py
341
342
343
344
345
346
347
def ensure_paths(self) -> None:
    """Ensure all configured paths exist on the filesystem.

    Creates parent directories as needed.
    """
    self.path.cache_dir.mkdir(parents=True, exist_ok=True)
    self.path.checkout_dir.mkdir(parents=True, exist_ok=True)

from_settings classmethod

from_settings(config_file: Path | str | None = None, cwd: Path | str | None = None) -> ThreeGPPConfig

Create a ThreeGPPConfig instance with config file and environment values.

Config precedence (highest to lowest): 1. CLI explicit config_file parameter 2. Discovered config files (later override earlier) 3. Environment variables (TDC_*) 4. Hard-coded defaults

Parameters:

  • config_file

    (Path | str | None, default: None ) –

    Explicit path to a config file. If provided, only this file is loaded (no discovery).

  • cwd

    (Path | str | None, default: None ) –

    Working directory for project-level config discovery. Defaults to current working directory.

Returns:

Source code in src/tdoc_crawler/config/settings.py
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
@classmethod
def from_settings(
    cls,
    config_file: Path | str | None = None,
    cwd: Path | str | None = None,
) -> ThreeGPPConfig:
    """Create a ThreeGPPConfig instance with config file and environment values.

    Config precedence (highest to lowest):
    1. CLI explicit config_file parameter
    2. Discovered config files (later override earlier)
    3. Environment variables (TDC_*)
    4. Hard-coded defaults

    Args:
        config_file: Explicit path to a config file. If provided, only this
            file is loaded (no discovery).
        cwd: Working directory for project-level config discovery.
             Defaults to current working directory.

    Returns:
        A validated ThreeGPPConfig instance.
    """
    cwd_path = Path(cwd) if cwd is not None else Path.cwd()

    if config_file:
        config_path = Path(config_file)
        merged = load_config_file(config_path)
    else:
        discovered = discover_config_files(cwd=cwd_path)
        configs = [load_config_file(p) for p in discovered]
        merged = merge_configs(configs)

    return cls.model_validate(merged)

HTTP client

http_client

HTTP client factory with hishel caching support.

Re-exports from session module for backward-compatible imports

from tdoc_crawler.http_client import create_cached_session from tdoc_crawler.http_client import download_to_file from tdoc_crawler.http_client import PoolConfig

HttpCacheStats dataclass

HttpCacheStats(entries_before: int, entries_after: int, expired_removed: int, evicted: int, bytes_before: int, bytes_after: int, vacuumed: bool)

Outcome of a cache prune (sizes are physical file bytes).

PoolConfig dataclass

PoolConfig(max_connections: int = 10, max_per_host: int = 5, connection_timeout: float = 30.0, enable_retry: bool = True, retry_attempts: int = 3)

HTTP connection pool configuration.

Attributes:

  • max_connections (int) –

    Maximum number of connection pools to cache

  • max_per_host (int) –

    Maximum number of connections per host

  • connection_timeout (float) –

    Connection timeout in seconds

  • enable_retry (bool) –

    Whether to enable retry logic

  • retry_attempts (int) –

    Number of retry attempts for failed requests

create_cached_session

create_cached_session(http_cache_file: Path | None = None, http_cache_enabled: bool | None = None, pool_config: PoolConfig | None = None, verify: bool | str | None = None, http_config: HttpConfig | None = None) -> requests.Session

Create a requests.Session with hishel caching enabled.

Parameters:

  • http_cache_file

    (Path | None, default: None ) –

    Optional explicit path to the HTTP cache database. Falls back to PathConfig().http_cache_file when caching is enabled.

  • http_cache_enabled

    (bool | None, default: None ) –

    Whether to enable HTTP caching. If None, uses http_config.cache_enabled or True.

  • pool_config

    (PoolConfig | None, default: None ) –

    Optional connection pool configuration. When provided, pool settings are applied to the active adapter (cache adapter when caching is enabled, HTTPAdapter otherwise).

  • verify

    (bool | str | None, default: None ) –

    SSL certificate verification mode. Can be bool or CA bundle path.

  • http_config

    (HttpConfig | None, default: None ) –

    HttpConfig from ThreeGPPConfig. Controls caching TTL, retries, SSL.

Returns:

  • Session –

    Configured requests.Session with caching enabled (unless disabled)

Source code in src/tdoc_crawler/http_client/session.py
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
def create_cached_session(
    http_cache_file: Path | None = None,
    http_cache_enabled: bool | None = None,
    pool_config: PoolConfig | None = None,
    verify: bool | str | None = None,
    http_config: HttpConfig | None = None,
) -> requests.Session:
    """Create a requests.Session with hishel caching enabled.

    Args:
        http_cache_file: Optional explicit path to the HTTP cache database.
                         Falls back to PathConfig().http_cache_file when caching is enabled.
        http_cache_enabled: Whether to enable HTTP caching. If None, uses http_config.cache_enabled or True.
        pool_config: Optional connection pool configuration.
                When provided, pool settings are applied to the active adapter
                (cache adapter when caching is enabled, HTTPAdapter otherwise).
        verify: SSL certificate verification mode. Can be bool or CA bundle path.
        http_config: HttpConfig from ThreeGPPConfig. Controls caching TTL, retries, SSL.

    Returns:
        Configured requests.Session with caching enabled (unless disabled)
    """
    # Create session, use requests with browser-like headers to avoid 403 Forbidden
    # Disable HTTP/3 to avoid MustDowngradeError with servers advertising Alt-Svc: h3
    session = requests.Session(disable_http3=True)
    for header_name, header_value in BROWSER_HEADERS.items():
        session.headers[header_name] = header_value

    # Use http_config for SSL if no explicit verify provided
    verify_mode = http_config.verify_ssl if verify is None and http_config is not None else resolve_ssl_verify(verify, http_config)
    session.verify = verify_mode

    # Resolve cache enabled flag: explicit param → http_config → default True
    if http_cache_enabled is None:
        http_cache_enabled = http_config.cache_enabled if http_config is not None else True

    # If caching is disabled, optionally configure a pooled HTTP adapter and return.
    # niquests handles SSL natively, so no custom SSL context injection is needed.
    if not http_cache_enabled:
        if pool_config is not None:
            retry_attempts = pool_config.retry_attempts if pool_config.enable_retry else 0
            retry_strategy = Retry(
                total=retry_attempts,
                backoff_factor=1,
                status_forcelist=[429, 500, 502, 503, 504],
                allowed_methods=["HEAD", "GET", "OPTIONS"],
            )
            adapter = HTTPAdapter(
                pool_connections=pool_config.max_connections,
                pool_maxsize=pool_config.max_per_host,
                max_retries=retry_strategy,
            )
            session.mount("http://", adapter)
            session.mount("https://", adapter)
            logger.debug(f"Configured connection pool without caching: max_connections={pool_config.max_connections}, max_per_host={pool_config.max_per_host}")

        logger.debug("Creating plain HTTP session (caching disabled)")
        return session

    # Resolve cache settings from http_config or defaults
    ttl = http_config.cache_ttl if http_config is not None else 7200
    max_retries_count = http_config.max_retries if http_config is not None else 3
    # Bound the live cache size. 0 means unlimited (hishel's ``None``), and
    # hishel evicts least-recently-used entries when this cap is exceeded.
    cache_max_bytes = http_config.cache_max_bytes if http_config is not None else 2 * 1024**3
    storage_max_size = cache_max_bytes if cache_max_bytes > 0 else None

    # Determine cache file path — use explicit param or fall back to PathConfig default
    cache_file = str(http_cache_file) if http_cache_file is not None else str(PathConfig().http_cache_file)

    logger.debug("Creating cached HTTP session with cache_db=%s)", cache_file)
    logger.debug("HTTP cache config: ttl=%s, max_retries=%s", ttl, max_retries_count)

    # Create SQLite storage backend. WAL mode gives concurrent reads while
    # writing and avoids "database is locked" under the threaded sub-interpreter
    # Open the connection ourselves so we can set PRAGMA journal_mode=WAL.
    # WAL mode gives concurrent reads while writing and avoids "database is
    # locked" under the threaded sub-interpreter workers that crawl_tdocs
    # uses under the MCP stdio server. Hishel accepts a pre-built connection,
    # but treats database_path as a plain path so URI-based params like
    # ?journal_mode=WAL won't work without this hook.
    cache_path = Path(cache_file)
    cache_path.parent.mkdir(parents=True, exist_ok=True)
    connection = sqlite3.connect(str(cache_path), check_same_thread=False)
    connection.execute("PRAGMA journal_mode=WAL")
    connection.execute("PRAGMA synchronous=NORMAL")
    storage = SqliteStorage(
        connection=connection,
        default_ttl=ttl,
        max_size=storage_max_size,
    )

    # Configure retry strategy for the cache adapter.
    # If pool_config is set, reuse its retry settings so caching + pooling share one adapter.
    retry_attempts = pool_config.retry_attempts if pool_config and pool_config.enable_retry else max_retries_count
    max_retries = Retry(
        total=retry_attempts,
        backoff_factor=1,
        status_forcelist=[429, 500, 502, 503, 504],
        allowed_methods=["HEAD", "GET", "OPTIONS"],
    )

    # Create a single cache adapter with pool settings so both behaviors are active.
    cache_adapter = _NiquetsCacheAdapter(
        pool_connections=pool_config.max_connections if pool_config is not None else 10,
        pool_maxsize=pool_config.max_per_host if pool_config is not None else 10,
        max_retries=max_retries,
        storage=storage,
    )

    # Mount the cache adapter
    session.mount("http://", cache_adapter)
    session.mount("https://", cache_adapter)

    if pool_config is not None:
        logger.debug(f"Configured cache + pool adapter: max_connections={pool_config.max_connections}, max_per_host={pool_config.max_per_host}")

    return session

download_to_file

download_to_file(url: str, target_file: Path, session: Session | None = None, http_cache_file: Path | None = None, http_cache_enabled: bool | None = None, pool_config: PoolConfig | None = None, verify: bool | str | None = None, http_config: HttpConfig | None = None) -> None

Download a file from URL to destination path.

Parameters:

  • url

    (str) –

    Source URL

  • target_file

    (Path) –

    Destination file path

  • session

    (Session | None, default: None ) –

    Optional requests.Session to reuse. If None, a temporary cached session is created and closed.

  • http_cache_file

    (Path | None, default: None ) –

    Optional explicit path to the HTTP cache database. Falls back to PathConfig default.

  • http_cache_enabled

    (bool | None, default: None ) –

    Whether to enable HTTP caching. If None, defaults to http_config.cache_enabled or True.

  • pool_config

    (PoolConfig | None, default: None ) –

    Optional connection pool configuration.

  • verify

    (bool | str | None, default: None ) –

    SSL certificate verification mode. Can be bool or CA bundle path.

  • http_config

    (HttpConfig | None, default: None ) –

    HttpConfig from ThreeGPPConfig. Controls caching, retries, SSL.

Raises:

Source code in src/tdoc_crawler/http_client/session.py
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
def download_to_file(
    url: str,
    target_file: Path,
    session: requests.Session | None = None,
    http_cache_file: Path | None = None,
    http_cache_enabled: bool | None = None,
    pool_config: PoolConfig | None = None,
    verify: bool | str | None = None,
    http_config: HttpConfig | None = None,
) -> None:
    """Download a file from URL to destination path.

    Args:
        url: Source URL
        target_file: Destination file path
        session: Optional requests.Session to reuse. If None, a temporary cached session is created and closed.
        http_cache_file: Optional explicit path to the HTTP cache database. Falls back to PathConfig default.
        http_cache_enabled: Whether to enable HTTP caching. If None, defaults to http_config.cache_enabled or True.
        pool_config: Optional connection pool configuration.
        verify: SSL certificate verification mode. Can be bool or CA bundle path.
        http_config: HttpConfig from ThreeGPPConfig. Controls caching, retries, SSL.

    Raises:
        ValueError: If URL scheme is not supported
        FileNotFoundError: If download fails
    """
    lowered = url.lower()
    allowed_schemes = ("ftp://", "http://", "https://")
    if not lowered.startswith(allowed_schemes):
        msg = f"unsupported-url-scheme: {url!r}"
        raise ValueError(msg)

    target_file.parent.mkdir(parents=True, exist_ok=True)

    # Use provided session or create a new one (might be used for multiple downloads, so we don't want to create a new session for each)
    temp_session: requests.Session | None = None
    if session is None:
        temp_session = create_cached_session(
            http_cache_file=http_cache_file,
            http_cache_enabled=http_cache_enabled,
            pool_config=pool_config,
            verify=verify,
            http_config=http_config,
        )
        active_session = temp_session
    else:
        active_session = session

    try:
        timeout = pool_config.connection_timeout if pool_config is not None else 60
        response = active_session.get(url, timeout=timeout, stream=True)
        response.raise_for_status()
        with target_file.open("wb") as target:
            for chunk in response.iter_content(chunk_size=8192):
                if chunk:
                    target.write(chunk)

    finally:
        if temp_session:
            temp_session.close()

prune_http_cache

prune_http_cache(cache_file: str | Path, *, max_bytes: int | None = None, now: float | None = None) -> HttpCacheStats

Remove expired entries, optionally evict to max_bytes, and vacuum.

Safe to call at any time: a missing or uninitialised database is a no-op, and a concurrent writer that holds a lock only prevents the VACUUM (logged at debug level) — row deletions still succeed.

Parameters:

  • cache_file

    (str | Path) –

    Path to the hishel SQLite cache database.

  • max_bytes

    (int | None, default: None ) –

    When set, evict the least-recently-used entries until the live database size is at or below this many bytes.

  • now

    (float | None, default: None ) –

    Overridable clock for deterministic tests.

Returns:

  • HttpCacheStats –

    class:HttpCacheStats describing what was reclaimed.

Source code in src/tdoc_crawler/http_client/cache_maintenance.py
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
def prune_http_cache(cache_file: str | Path, *, max_bytes: int | None = None, now: float | None = None) -> HttpCacheStats:
    """Remove expired entries, optionally evict to *max_bytes*, and vacuum.

    Safe to call at any time: a missing or uninitialised database is a no-op,
    and a concurrent writer that holds a lock only prevents the ``VACUUM``
    (logged at debug level) — row deletions still succeed.

    Args:
        cache_file: Path to the hishel SQLite cache database.
        max_bytes: When set, evict the least-recently-used entries until the
            live database size is at or below this many bytes.
        now: Overridable clock for deterministic tests.

    Returns:
        :class:`HttpCacheStats` describing what was reclaimed.
    """
    cache_file_path = Path(cache_file)
    empty = HttpCacheStats(0, 0, 0, 0, 0, 0, False)
    if not cache_file_path.exists():
        return empty

    now = time.time() if now is None else now
    connection = sqlite3.connect(str(cache_file_path), timeout=5)
    connection.execute("PRAGMA busy_timeout=5000")
    # Cascading deletes of stream chunks depend on foreign keys being enabled,
    # the same way hishel's own connection does.
    connection.execute("PRAGMA foreign_keys=ON")
    try:
        tables = {row[0] for row in connection.execute("SELECT name FROM sqlite_master WHERE type='table'")}
        if "entries" not in tables:
            return empty

        bytes_before = cache_file_path.stat().st_size
        entries_before = int(connection.execute("SELECT COUNT(*) FROM entries").fetchone()[0])

        expired = connection.execute(
            "DELETE FROM entries WHERE expires_at < ? AND last_accessed_at < ?",
            (now, now - SOFT_DELETE_TTL),
        )
        expired_removed = max(0, expired.rowcount)
        connection.commit()

        evicted = 0
        if max_bytes is not None and _live_size(connection) > max_bytes:
            for _ in range(EVICTION_MAX_BATCHES):
                if _live_size(connection) <= max_bytes:
                    break
                batch = connection.execute(
                    "DELETE FROM entries WHERE id IN ( SELECT id FROM entries ORDER BY last_accessed_at LIMIT ?)",
                    (EVICTION_BATCH_SIZE,),
                )
                deleted = max(0, batch.rowcount)
                connection.commit()
                evicted += deleted
                if deleted == 0:
                    break

        vacuumed = True
        try:
            connection.execute("VACUUM")
        except sqlite3.Error:
            vacuumed = False
            logger.debug("Could not vacuum HTTP cache %s (a writer may hold a lock)", cache_file_path)

        entries_after = int(connection.execute("SELECT COUNT(*) FROM entries").fetchone()[0])
        bytes_after = cache_file_path.stat().st_size
        return HttpCacheStats(
            entries_before=entries_before,
            entries_after=entries_after,
            expired_removed=expired_removed,
            evicted=evicted,
            bytes_before=bytes_before,
            bytes_after=bytes_after,
            vacuumed=vacuumed,
        )
    finally:
        connection.close()

resolve_ssl_verify

resolve_ssl_verify(verify: bool | str | None = None, http_config: HttpConfig | None = None) -> bool | str

Resolve SSL verification behavior from explicit argument or HttpConfig.

Resolution order: 1. Explicit verify parameter 2. http_config.verify_ssl if provided 3. Default (True)

Parameters:

  • verify

    (bool | str | None, default: None ) –

    Optional explicit SSL verification mode.

  • http_config

    (HttpConfig | None, default: None ) –

    Optional HttpConfig. When provided, verify_ssl is used as fallback.

Returns:

  • bool | str –

    Either a boolean verification flag or a certificate bundle path.

Source code in src/tdoc_crawler/http_client/session.py
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
def resolve_ssl_verify(
    verify: bool | str | None = None,
    http_config: HttpConfig | None = None,
) -> bool | str:
    """Resolve SSL verification behavior from explicit argument or HttpConfig.

    Resolution order:
    1. Explicit `verify` parameter
    2. http_config.verify_ssl if provided
    3. Default (True)

    Args:
        verify: Optional explicit SSL verification mode.
        http_config: Optional HttpConfig. When provided, verify_ssl is used as fallback.

    Returns:
        Either a boolean verification flag or a certificate bundle path.
    """
    if verify is not None:
        resolved = verify
    elif http_config is not None:
        resolved = http_config.verify_ssl
    else:
        resolved = True

    if resolved is False:
        logger.warning("SSL certificate verification is disabled. This is insecure.")

    return resolved