Configuration & HTTP Caching¶
HTTP Caching¶
To prevent rate-limiting and improve performance, 3gpp-crawler caches all
HTTP requests by default.
Default Behavior¶
- Cache location:
~/.3gpp-crawler/http-cache-v2.sqlite3 - TTL: 2 hours (7200 seconds), configured with
http.cache_ttl/HTTP_CACHE_TTL.
Configuration¶
Via .env or config file (see Configuration):
HTTP_CACHE_TTL=7200
ETSI Online (EOL) Credentials¶
Credentials provide access to the authenticated 3GPP portal endpoints. They are optional: single/missing TDocs are fetched from the official (unauthenticated) portal by default, with WhatTheSpec used only as a third-party fallback when credentials are unavailable.
Why credentials?¶
- Official data: metadata directly from 3GPP endpoints — no third-party service that can change or disappear.
- Full metadata:
--full-metadatafetching requires authentication. - Reliability: keeps working even if WhatTheSpec is down or its API changes.
Resolution order¶
- CLI parameters (
--eol-username,--eol-password). - Environment variables (
TDC_EOL_USERNAME,TDC_EOL_PASSWORD). - Interactive prompt (if
TDC_EOL_PROMPT=true).
Directory Structure¶
By default, all data is stored in ~/.3gpp-crawler:
| Path | Purpose |
|---|---|
3gpp_crawler.db |
Main metadata database. |
http-cache-v2.sqlite3 |
HTTP request cache (hishel 2 schema; older http-cache.sqlite3 files are not reused). |
checkout/ |
Downloaded and extracted TDocs and spec files, for offline review. |
Override the root directory with the --cache-dir flag on most commands
or via TDC_CACHE_DIR.