Skip to content

Development Guide

This guide describes how to set up your environment for contributing to 3gpp-crawler.

Setup

  1. Clone the repository:
git clone https://github.com/jr2804/3gpp-crawler.git
cd 3gpp-crawler
  1. Install the toolchain and dependencies:
mise install
uv sync --all-extras
  1. Install pre-commit hooks:
uv run pre-commit install

Task definitions live in .config/mise/conf.d/*.toml; mise run lists them.

Workflow

Running Tests

All tests use pytest. The project aims for 70%+ coverage.

# Run all tests
uv run pytest

# Run with coverage report
uv run pytest --cov=tdoc_crawler --cov-report=html

# Run specific test file
uv run pytest tests/test_database.py -v

Code Quality

mise provides pyreorder (ty too), so use the tasks rather than the bare tools. These must be clean before a change is handed back:

mise run format      # pyreorder + ruff format (mutates files)
mise run lint        # ruff check --fix --unsafe-fixes
mise run typecheck   # ty check src/ tests/  (never `uv run ty` — ty lives in mise)

Fix type errors properly; # type: ignore is not accepted. Run uv run pytest for the suite (see above). mise run all chains test + lint + spell + format + format-md + docs; typecheck is not part of it, so run it explicitly.

Documentation Standards

  • Update the relevant docs in docs/ when adding/changing CLI commands (see docs/index.md for the documentation index).
  • Write Google-style docstrings for all functions.
  • Keep .agents/history/ logs updated for significant changes.

Architecture Overview

  • models/: Pydantic models (the "source of truth" for data structures).
  • tdocs/: TDoc crawling, sources (FTP, Portal, WhatTheSpec), checkout, and main-document selection.
  • meetings/: Meeting data handling.
  • specs/: Specification operations.
  • workitems/: Work item catalogue (DynaReport lists, enrichment, audit).
  • database/: SQLite/Oxyde persistence layer.
  • cli/: Typer-based command definitions (tdoc_app.py, spec_app.py, mgmt_app.py entry points).
  • executors/, workers/: Crawl execution and parallel workers.
  • http_client/: Cached HTTP session management.
  • config/: Configuration management (see below).
  • mcp_server.py: stdio MCP server exposing the same operations as tools.

Configuration System

Two complementary systems:

  1. ThreeGPPConfig (pydantic-settings, alias TDocCrawlerConfig) — Type-safe configuration from files/env vars
  2. CacheManager (runtime paths) — File system path resolution

ThreeGPPConfig (Settings)

Use for all configurable behavior (timeouts, credentials, limits, etc.):

from tdoc_crawler.config import ThreeGPPConfig

# Load with automatic discovery (3gpp-crawler.toml, env vars)
config = ThreeGPPConfig.from_settings()

# Or with explicit config file
config = ThreeGPPConfig.from_settings(config_file=Path("./my-config.toml"))

# Access nested config
config.path.cache_dir      # Path to cache directory
config.http.timeout        # HTTP timeout in seconds
config.credentials.username  # Portal username
config.crawl.workers       # Concurrent crawl workers

Config file discovery order (later overrides earlier):

  1. Global: ~/.config/3gpp-crawler/config.toml, then ~/.config/3gpp-crawler/conf.d/*
  2. Project: .config/.3gpp-crawler/conf.d/*, .config/.3gpp-crawler/config.toml, .config/3gpp-crawler.toml, .3gpp-crawler/config.toml, .3gpp-crawler.toml, 3gpp-crawler.toml

Precedence: CLI args > Config file > Environment variables > Defaults

Supported formats: TOML (primary), YAML, JSON — accepted at every location.

Environment variable prefixes:

  • TDC_* — Path settings
  • TDC_EOL_* — Portal credentials
  • TDC_CRAWL_* — Crawl filters
  • HTTP_CACHE_* — HTTP cache settings
  • TDC_CHECKOUT_* — Main-document selection (checkout.selector, TypeSafe System One settings)

CacheManager (Runtime Paths)

Single Source of Truth: All file paths MUST use CacheManager.

from tdoc_crawler.config import resolve_cache_manager

manager = resolve_cache_manager()
manager.root              # cache root directory
manager.db_file           # SQLite database
manager.http_cache_file   # HTTP cache
manager.checkout_dir      # Document checkout directory

The CacheManager is instantiated by the CLI wrapper. Library users must register their own instance at program start.

Why This Matters

  • DRY principle: Path logic defined once, used everywhere
  • Configurability: Users can override via TDC_CACHE_DIR env var
  • Consistency: All components use identical paths
  • Testability: Easy to swap in test directories

Antipatterns

Overly Careful Error Handling

# ❌ WRONG — inconsistent return types, None overuse, error "encoding"
def get_info(number: str|int|None, message: str|None|Any) -> InfoObject|str|None:
    if isinstance(tdoc_id, None):
        raise ValueError("tdoc_id cannot be None")
    try:
        ...
        return info_object
    except SomeSpecificError:
        return None
    except AnotherError:
        return "Error: Invalid input"

# ✅ CORRECT — consistent return types, minimal checking, let it fail
def get_info(number: str|int, message: str) -> InfoObject:
    ...
    return info_object

Defensive CacheManager Access

# ❌ WRONG — boilerplate fallback hides dev errors
try:
    manager = resolve_cache_manager()
except CacheManagerNotRegisteredError:
    try:
        manager = CacheManager(default_cache_dir).register()
    except Exception as e:
        raise RuntimeError("Failed to create CacheManager.") from e

# ✅ CORRECT — let it burn if not registered
manager = resolve_cache_manager()

TYPE_CHECKING for Circular Imports

Avoid typing.TYPE_CHECKING as a workaround for circular imports. It indicates structural problems. Refactor instead — extract shared types to a neutral models/ layer.

Database Migrations

There are no migrations by design (database/base.py::_ensure_schema). An existing database whose tables/columns do not match the Oxyde models fails hard with schema-mismatch and the missing table.column list. Adding a model (e.g. MainFileOverride) therefore requires rebuilding the database with 3gpp-crawler database clear-all (backs up tags, deletes the database file, keeps the downloaded TDocs) and re-crawling. Add --purge-checkouts only when the files themselves are suspect. Never add an auto-create or auto-alter fallback to paper over it.

Design Principles

Directory Layout

~/.3gpp-crawler/
├── checkout/          # Downloaded/extracted documents (TDocs, Specs)
├── 3gpp_crawler.db
└── http-cache-v2.sqlite3

Checkouts contain the raw downloaded/extracted source files.

SourceKind-Aware Processing

Checkout and fetch logic branches on the item kind: TDocs and Specs have fundamentally different fetch and file-resolution logic:

  • TDocs: checked out via FTP/meeting folder structure, resolved through resolve_tdoc_checkout_path() in tdocs/operations/checkout.py; the main document inside the archive is chosen by resolve_main_file() in tdocs/operations/main_file.py
  • Specs: checked out from Specs/archive/{series}/{spec_number}/; release and URL resolution live in SpecDownloader (specs/downloads.py)

Never assume all sources are TDocs.

Spec Release Resolution Must Be Concrete

Spec releases must always be resolved to a full 3-part version (e.g. "18.0.0") before being stored or used for file resolution. Ambiguous selectors like "latest", "18", or "18.0" are resolved via database lookup at the earliest possible point, inside SpecDownloader._resolve_spec_url() (specs/downloads.py).

Why: The spec checkout directory (checkout/Specs/archive/{series}/{spec}/) is shared across all releases of a spec. After downloading REL19, files for REL18 and REL19 coexist as version-specific subdirectories (e.g. 26261-g00/ for REL19, 26261-i00/ for REL18). If a downstream function receives an unresolved "latest" and scans the whole tree, it can return files from the wrong release.

Two-layer contract:

  1. Resolve early — _resolve_spec_url() in specs/downloads.py matches the requested release through normalize_release() + _filter_versions_by_release() and pins the concrete 3-part version before any file operation.
  2. Respect downstream — fetch_spec_files() uses the pinned version to scan only the version-specific extracted directory, never the shared parent.

Rule: Never pass ambiguous release selectors to fetch/convert functions. Never scan a shared checkout directory without filtering by release.

Function Renaming Follows Scope

When a function's scope changes, rename it to match its current behavior, not its historical behavior. Names should describe what the function does today.

Project Structure

Generate on the fly (never hardcode listings):

rg --files | tree-cli --fromfile