scraping-debugging-tools

Generic, site-agnostic tools and docs for figuring out why a browser-driven (webdriver/CDP) scraper is broken — blank pages, hangs, wrong content — instead of guessing.

Start here: docs/debugging-webdriver-scrapers.md

Scripts

Requires uv and Python 3.12+ (uv will fetch a matching interpreter automatically if you don't have one).

uv sync

# What actually loaded? Any failed requests? Does the page reload mid-load?
uv run scripts/network_snapshot.py "<url>" \
    --wait 20 \
    --wait-for-selector '<css selector for the content that should render>' \
    --proxy "user:pass@host:port"   # optional

# Live per-request detail (e.g. the remote IP a request exited on)
uv run scripts/live_network_capture.py "<neutral_url>" "<target_url>"

Both scripts drive a real Chrome via SeleniumBase's CDP mode and dump their findings (network requests, console errors, page source) as JSON/HTML for inspection. By default they write to tmp/<script name>/ in the repo root (gitignored) so the output is right there in your IDE's file tree — pass --out-dir to put it somewhere else. See the docs for how to read the output and when to reach for which script.

Layout

docs/     methodology + worked case studies
scripts/  the actual tools
tmp/      gitignored scratch space - default output dir for scripts/