Skip to content

Crawling Metadata

Crawling gathers metadata from the official sources (3GPP FTP server, 3GPP Portal; WhatTheSpec only as an optional convenience/fallback) and stores it in your local SQLite database.

Prerequisites

Meetings must be crawled first: the TDoc crawl relies on the meetings database to know which meeting directories to visit on the FTP server.

Commands

meetings crawl

Crawl meeting metadata from the 3GPP portal.

3gpp-crawler meetings crawl                    # all working groups
3gpp-crawler meetings crawl -s S4              # SA4 subgroup only
3gpp-crawler meetings crawl --limit-meetings-per-subwg 10
Option Description
-w, --working-group WG Working groups to crawl (repeatable): RAN, SA, CT.
-s, --sub-group SG Sub-working groups to crawl (repeatable), e.g. S4, R1.
--incremental / --full Incremental skips existing meetings; --full reprocesses.
--limit-meetings-per-subwg N Maximum meetings per sub-working group.
--eol-username USER ETSI Online account (faster authenticated access).

crawl (alias: ct)

Crawl TDoc metadata from the 3GPP FTP server.

3gpp-crawler tdoc crawl                       # all TDocs (after meetings crawl)
3gpp-crawler tdoc crawl -w RAN -s R1 -s R2    # specific subgroups
3gpp-crawler tdoc crawl --start-date 2024     # date-filtered
3gpp-crawler tdoc crawl --workers 8           # parallel workers
3gpp-crawler tdoc crawl -m 'SA4#138'          # one meeting
3gpp-crawler tdoc crawl -m 'SA4#13*'          # wildcard over meeting names
Option Description
-w, --working-group WG Working groups to crawl (repeatable).
-s, --sub-group SG Sub-working groups to crawl (repeatable).
-m, --meeting PATTERN Restrict the crawl to meetings (repeatable): portal short name (SA4#138), normalized name (S4-138), fnmatch wildcard (SA4#13*), or numeric meeting ID. A leading 3GPP on either side is ignored (SA4#134 matches a meeting stored as 3GPPSA4#134). A pattern that matches no locally known meeting fails the crawl with suggestions.
--incremental / --full Incremental skips existing TDocs.
--workers N Parallel workers (default: 4).
--executor TYPE Advanced (hidden from --help): serial, threading, multiprocessing, subinterpreter.
--timeout / --max-retries Per-request timeout and retry count for meeting document lists (default 3 retries with backoff); a meeting that keeps failing is reported per meeting instead of silently dropped.
--checkout Download and extract crawled TDocs to the checkout folder.
--start-date / --end-date ISO 8601 date filters.
--source / --title / --agenda Glob patterns; -ex variants exclude.
--limit-tdocs N Limit TDocs per crawl.
--limit-meetings N Limit meetings per working group/subgroup.
--limit-meetings-per-subwg N Limit meetings per sub-working group.
--limit-subwgs N Limit sub-working groups.
--cache-dir / --http-cache / --verbosity Path, HTTP cache and log-level overrides.

Each crawled meeting document list also carries the Related WIs column (comma-separated work item IDs) plus the specification and CR columns. Those links are stored with the TDoc, so tdoc query --work-item ULBC-MED, tdoc query --spec-number 26.114 and work-items query work without an extra pass — see Querying metadata. Work item titles, releases, parents and impacted specifications come from the per-working-group DynaReport work item lists, which are crawled separately with 3gpp-crawler work-items crawl.

The meeting's agenda document (the document-list row of type agenda) is parsed during the crawl to recover the parent agenda items that the document list does not contain. Agenda filters then match leaf and parent items together — see Querying metadata.

A crawl that ends with per-meeting errors lists every failed meeting and exits with code 1, so a dropped meeting can no longer pass as a clean run.

crawl (work items)

Crawl the 3GPP work item catalogue from the DynaReport work item lists (https://www.3gpp.org/dynareport?code=TSG-WG--S4--wis.htm and siblings). Each lead body (SP, S1…S6, R1…R6, C1…C6, plus historical ones like G1, T2) publishes its own current list, including in-flight work items.

3gpp-crawler work-items crawl                  # every lead body (discovered)
3gpp-crawler work-items crawl -s S4            # one lead body
3gpp-crawler work-items crawl -w SA            # whole SA area (SP, S1..S6)
3gpp-crawler work-items crawl -s R1 -s R2      # several lead bodies
Option Description
-w, --working-group WG Crawl the area's lead bodies (RAN, SA, CT).
-s, --sub-group SG Crawl specific lead bodies (repeatable; historical codes like GP or T2 work too).

Without -w/-s the lead bodies are discovered from the global WI-List page. work-items enrich then fills status, parent/child and spec/CR links from the portal detail pages, and work-items audit compares the catalogue against the global list. Transient blocks (HTTP 403/429) are retried with backoff; a genuinely missing page fails fast as not found. The command lists every failure and exits with code 1 while any work item remains unfilled.

crawl (specs)

Crawl technical specification (TS/TR) metadata.

3gpp-crawler spec crawl                       # all specs from all sources
3gpp-crawler spec crawl 26260                 # specific spec
3gpp-crawler spec crawl 26260 --checkout      # also download the documents
Option Description
SPEC_NUMBERS Positional spec numbers (dotted or undotted).
--release REL Release filter (default latest).
--checkout Also download the spec documents.

Use spec query --working-group SA4 (or -w S4) to filter an already-crawled catalog by working group; spec crawl takes no working-group filter.

Sources (3gpp, whatthespec) are tried in that order: the official 3GPP source first, WhatTheSpec only as fallback.

Shortcuts

The tdoc-crawler and spec-crawler entry points run the same commands without the group prefix:

tdoc-crawler crawl --start-date 2024
spec-crawler crawl 26260