Spider-Tree: A High-Performance Hierarchical Web Crawler

Context and Problem

The idea originated during the preliminary redesign phase of a medium-sized website. To estimate migration timelines and reorganize the information architecture, a basic yet critical baseline metric was needed: understanding exactly which and how many pages composed the domain.

Existing tools presented clear practical limitations:

Traditional SEO tools: Software like Screaming Frog or similar scripts export CSV sheets with thousands of unstructured records. Analyzing a flat list provides no visual perception of the hierarchy, depth levels, or logical branches.
Browser-based crawlers: Launching headless Chromium instances solely to extract <a> tags easily consumes between 800 MB and 1.5 GB of RAM before even beginning the actual work.
System utilities: The terminal tree command is ideal for readability, but it is bound to the local file system and has no native web counterpart.

The objective was therefore to build a streamlined tool: a targeted crawler designed to discard static assets (images, CSS, scripts, fonts) and focus exclusively on navigable HTML pages, mapping the site structure using the exact logic of a directory tree.

Critical HTTP Architecture Bottlenecks

Treating the web like a file system immediately encounters two technical issues inherent to the network's graph-like nature.

1. URI Loops and Ambiguity

A website easily generates multiple paths pointing to the exact same resource:

Page fragments (/services#pricing versus /services)
Relative paths (../../about-us)
Non-uniform trailing slash handling (/blog versus /blog/)
Discrepancies between subdomains (www.domain.com vs domain.com) and query string parameters (tracking, filters).

Without upstream normalization, the crawler risks infinite visiting cycles or an uncontrolled proliferation of duplicates within the same branch.

2. Logical, Non-Physical Directories

On disk, if /projects/web/portfolio.html exists, the intermediate directory /projects/web/ physically exists. On the web, this is not the case: a server may serve a 200 OK on the full endpoint /projects/web/portfolio.html, but return a 404 or a redirect if /projects/web/ is queried directly.

Building a continuous tree requires a strategy to handle these missing intermediate nodes without breaking the visual hierarchy.

Implementation Choices: Tries and Inferred Nodes

The solution relies on adopting a Trie paired with a deterministic normalization pipeline.

                    [ Root: https://example.com ]
                                │
                  ┌─────────────┴─────────────┐
                  ▼                           ▼
            /products/ (inferred)      /contacts/ (200 OK)
                  │
        ┌─────────┴─────────┐
        ▼                   ▼
  software/ (inferred)   hardware/ (200 OK)
        │
        ▼
  antivirus.html (200 OK)

Normalization: Every extracted URL is stripped of anchors, validated against the domain perimeter, and processed into an absolute format.
Inferred Nodes: When inserting a path like /products/software/antivirus.html into the Trie, the string is tokenized by / separators. If intermediate segments do not correspond to successfully scanned pages, they are instantiated as dummy nodes. This ensures the tree maintains structural consistency.
Network Load Management: The crawler defines strict connection timeouts, applies boundary filters on the domain to prevent unwanted external links, and leverages standard resources provided by webmasters (robots.txt and sitemap.xml).

Architecture and Stack Selection

The primary architectural design choice was to minimize external dependencies.

┌─────────────────────────────────────────────────────────────┐
│                 Desktop GUI (CustomTkinter)                 │
│               4-step Wizard - Dark/Light Mode               │
└──────────────────────────────┬──────────────────────────────┘
                               │ Thread Decoupling
┌──────────────────────────────▼──────────────────────────────┐
│                    Spider-Tree Core Engine                  │
│       Python Standard Library (zero external packages)      │
│       urllib - html.parser - concurrent.futures - xml       │
└──────────────────────────────┬──────────────────────────────┘
                               │
        ┌──────────────────────┼──────────────────────┐
        ▼                      ▼                      ▼
  [ ASCII Tree ]       [ Markdown / JSON ]   [ Standalone HTML ]

1. Scanning Engine: Standard Library

The core is developed entirely using native Python modules (urllib, html.parser, concurrent.futures, xml.etree). This eliminates the footprint and runtime complexity of tools like Scrapy or Playwright, ensuring:

Instant startup.
Portable execution without heavy virtual environments or version conflicts.
Minimal memory consumption.

2. Graphical User Interface (CustomTkinter)

To avoid the overhead typical of an Electron application, the GUI is built using CustomTkinter and Pillow: offering a native desktop interface, reduced loading times, and direct execution via startup scripts.

3. Operational Workflow

[ 1. Target URL ] ──► [ 2. Format ] ──► [ 3. Live Scan ] ──► [ 4. Export ]
                           (Select       (Separate worker,         (ASCII, JSON,
                           output)       timer & counters)         Markdown, HTML)

The workflow follows linear steps:

Target: Input of the root URL.
Format: Selection of the desired output file formats.
Execution: Real-time tracking with a timer and metrics on processed and queued pages.
Export: Saving the report to disk or opening it immediately.

Engine Functional Characteristics

Combined Discovery

Prior to parsing links within the HTML code, the script queries robots.txt and extracts the URLs present in the indicated sitemaps (including recursive sitemapindex files). This step establishes a baseline to reach even pages devoid of internal links in the navigation menu or body.

[ Target Domain ]
        │
        ├─► [ Phase 1: Passive Discovery ] ──► robots.txt ──► sitemap.xml ──┐ (Seed URLs)
        │                                                                  ▼
        └─► [ Phase 2: Active Crawling ] ◄──────────────────────────────────┘
                  │
                  ├─► ThreadPoolExecutor (concurrent, max 3 MB per resource)
                  ├─► Normalization & Deduplication with Locks
                  └─► TrieTree Population (real + inferred nodes)

Concurrency Management

Page retrieval and tree population are handled via a thread pool managed with ThreadPoolExecutor. Access to the visitation queues and shared structures is synchronized via locking primitives (threading.Lock) to prevent duplicate requests. To avoid saturating memory with anomalous assets, downloading is truncated if it exceeds a predefined threshold per resource (3 MB). Processing runs on a dedicated thread, keeping the GUI consistently responsive.

Standalone HTML Output

The HTML export produces a single static file embedding minimal styling and scripts without references to CDNs or external resources:

Instant text-field filtering to isolate individual paths.
Expandable and collapsable node structure to consult only necessary branches.
Explicit indication of HTTP status codes (200, 301, 404).

Performance Benchmark

Parameter	Headless Environments (Chromium)	Spider-Tree
Average RAM Usage	800 MB — 1.5 GB	20 — 50 MB
Initialization Time	3 — 8 seconds	Less than 0.5 seconds
Engine Dependencies	Browser binaries and Node/Python libraries	None (Standard Library)
HTML Report Size	Directory with linked dependencies	Under 100 KB (single file)

Technical Considerations and Future Developments

The use of the Trie demonstrated how proper in-memory data structuring renders complex frameworks unnecessary for tasks of this nature. The resource savings enable complete scans to run even on machines with stringent hardware limitations.

Upcoming implementations include:

Optional Headless Module: On-demand integration of a JS renderer to map Single Page Applications built on client-side frameworks.
Structural Diff Engine: A CLI/GUI utility to compare two versions of the same tree (e.g., pre- and post-migration) and track path modifications, redirects, and error codes.
Directed Graph Representation: An export option to visually inspect internal link density in addition to the simple directory hierarchy.

Spider Tree