Development Guide¶
This guide describes how to set up your environment for contributing to 3gpp-crawler.
Setup¶
Using uv + mise (recommended)¶
- Clone the repository:
git clone https://github.com/jr2804/3gpp-crawler.git
cd 3gpp-crawler
- Install the toolchain and dependencies:
mise install
uv sync --all-extras
- Install pre-commit hooks:
uv run pre-commit install
Task definitions live in .config/mise/conf.d/*.toml; mise run lists them.
Workflow¶
Running Tests¶
All tests use pytest. The project aims for 70%+ coverage.
# Run all tests
uv run pytest
# Run with coverage report
uv run pytest --cov=tdoc_crawler --cov-report=html
# Run specific test file
uv run pytest tests/test_database.py -v
Code Quality¶
mise provides pyreorder (ty too), so use the tasks rather than the bare
tools. These must be clean before a change is handed back:
mise run format # pyreorder + ruff format (mutates files)
mise run lint # ruff check --fix --unsafe-fixes
mise run typecheck # ty check src/ tests/ (never `uv run ty` — ty lives in mise)
Fix type errors properly; # type: ignore is not accepted. Run
uv run pytest for the suite (see above). mise run all chains
test + lint + spell + format + format-md + docs; typecheck is not part of
it, so run it explicitly.
Documentation Standards¶
- Update the relevant docs in
docs/when adding/changing CLI commands (see docs/index.md for the documentation index). - Write Google-style docstrings for all functions.
- Keep
.agents/history/logs updated for significant changes.
Architecture Overview¶
models/: Pydantic models (the "source of truth" for data structures).tdocs/: TDoc crawling, sources (FTP, Portal, WhatTheSpec), checkout, and main-document selection.meetings/: Meeting data handling.specs/: Specification operations.workitems/: Work item catalogue (DynaReport lists, enrichment, audit).database/: SQLite/Oxyde persistence layer.cli/: Typer-based command definitions (tdoc_app.py,spec_app.py,mgmt_app.pyentry points).executors/,workers/: Crawl execution and parallel workers.http_client/: Cached HTTP session management.config/: Configuration management (see below).mcp_server.py: stdio MCP server exposing the same operations as tools.
Configuration System¶
Two complementary systems:
ThreeGPPConfig(pydantic-settings, aliasTDocCrawlerConfig) — Type-safe configuration from files/env varsCacheManager(runtime paths) — File system path resolution
ThreeGPPConfig (Settings)¶
Use for all configurable behavior (timeouts, credentials, limits, etc.):
from tdoc_crawler.config import ThreeGPPConfig
# Load with automatic discovery (3gpp-crawler.toml, env vars)
config = ThreeGPPConfig.from_settings()
# Or with explicit config file
config = ThreeGPPConfig.from_settings(config_file=Path("./my-config.toml"))
# Access nested config
config.path.cache_dir # Path to cache directory
config.http.timeout # HTTP timeout in seconds
config.credentials.username # Portal username
config.crawl.workers # Concurrent crawl workers
Config file discovery order (later overrides earlier):
- Global:
~/.config/3gpp-crawler/config.toml, then~/.config/3gpp-crawler/conf.d/* - Project:
.config/.3gpp-crawler/conf.d/*,.config/.3gpp-crawler/config.toml,.config/3gpp-crawler.toml,.3gpp-crawler/config.toml,.3gpp-crawler.toml,3gpp-crawler.toml
Precedence: CLI args > Config file > Environment variables > Defaults
Supported formats: TOML (primary), YAML, JSON — accepted at every location.
Environment variable prefixes:
TDC_*— Path settingsTDC_EOL_*— Portal credentialsTDC_CRAWL_*— Crawl filtersHTTP_CACHE_*— HTTP cache settingsTDC_CHECKOUT_*— Main-document selection (checkout.selector, TypeSafe System One settings)
CacheManager (Runtime Paths)¶
Single Source of Truth: All file paths MUST use CacheManager.
from tdoc_crawler.config import resolve_cache_manager
manager = resolve_cache_manager()
manager.root # cache root directory
manager.db_file # SQLite database
manager.http_cache_file # HTTP cache
manager.checkout_dir # Document checkout directory
The CacheManager is instantiated by the CLI wrapper. Library users must register their own instance at program start.
Why This Matters¶
- DRY principle: Path logic defined once, used everywhere
- Configurability: Users can override via
TDC_CACHE_DIRenv var - Consistency: All components use identical paths
- Testability: Easy to swap in test directories
Antipatterns¶
Overly Careful Error Handling¶
# ❌ WRONG — inconsistent return types, None overuse, error "encoding"
def get_info(number: str|int|None, message: str|None|Any) -> InfoObject|str|None:
if isinstance(tdoc_id, None):
raise ValueError("tdoc_id cannot be None")
try:
...
return info_object
except SomeSpecificError:
return None
except AnotherError:
return "Error: Invalid input"
# ✅ CORRECT — consistent return types, minimal checking, let it fail
def get_info(number: str|int, message: str) -> InfoObject:
...
return info_object
Defensive CacheManager Access¶
# ❌ WRONG — boilerplate fallback hides dev errors
try:
manager = resolve_cache_manager()
except CacheManagerNotRegisteredError:
try:
manager = CacheManager(default_cache_dir).register()
except Exception as e:
raise RuntimeError("Failed to create CacheManager.") from e
# ✅ CORRECT — let it burn if not registered
manager = resolve_cache_manager()
TYPE_CHECKING for Circular Imports¶
Avoid typing.TYPE_CHECKING as a workaround for circular imports. It indicates structural problems. Refactor instead — extract shared types to a neutral models/ layer.
Database Migrations¶
There are no migrations by design (database/base.py::_ensure_schema). An
existing database whose tables/columns do not match the Oxyde models fails
hard with schema-mismatch and the missing table.column list. Adding a
model (e.g. MainFileOverride) therefore requires rebuilding the database
with 3gpp-crawler database clear-all (backs up tags, deletes the database
file, keeps the downloaded TDocs) and re-crawling. Add --purge-checkouts
only when the files themselves are suspect. Never add an auto-create or
auto-alter fallback to paper over it.
Design Principles¶
Directory Layout¶
~/.3gpp-crawler/
├── checkout/ # Downloaded/extracted documents (TDocs, Specs)
├── 3gpp_crawler.db
└── http-cache-v2.sqlite3
Checkouts contain the raw downloaded/extracted source files.
SourceKind-Aware Processing¶
Checkout and fetch logic branches on the item kind: TDocs and Specs have fundamentally different fetch and file-resolution logic:
- TDocs: checked out via FTP/meeting folder structure, resolved through
resolve_tdoc_checkout_path()intdocs/operations/checkout.py; the main document inside the archive is chosen byresolve_main_file()intdocs/operations/main_file.py - Specs: checked out from
Specs/archive/{series}/{spec_number}/; release and URL resolution live inSpecDownloader(specs/downloads.py)
Never assume all sources are TDocs.
Spec Release Resolution Must Be Concrete¶
Spec releases must always be resolved to a full 3-part version (e.g. "18.0.0") before being stored or used for file resolution. Ambiguous selectors like "latest", "18", or "18.0" are resolved via database lookup at the earliest possible point, inside SpecDownloader._resolve_spec_url() (specs/downloads.py).
Why: The spec checkout directory (checkout/Specs/archive/{series}/{spec}/) is shared across all releases of a spec. After downloading REL19, files for REL18 and REL19 coexist as version-specific subdirectories (e.g. 26261-g00/ for REL19, 26261-i00/ for REL18). If a downstream function receives an unresolved "latest" and scans the whole tree, it can return files from the wrong release.
Two-layer contract:
- Resolve early —
_resolve_spec_url()inspecs/downloads.pymatches the requested release throughnormalize_release()+_filter_versions_by_release()and pins the concrete 3-part version before any file operation. - Respect downstream —
fetch_spec_files()uses the pinned version to scan only the version-specific extracted directory, never the shared parent.
Rule: Never pass ambiguous release selectors to fetch/convert functions. Never scan a shared checkout directory without filtering by release.
Function Renaming Follows Scope¶
When a function's scope changes, rename it to match its current behavior, not its historical behavior. Names should describe what the function does today.
Project Structure¶
Generate on the fly (never hardcode listings):
rg --files | tree-cli --fromfile