scraper-tester

A local server of small test "websites" for developing and testing web scrapers — with a live admin dashboard showing exactly what your scraper sent, at what rate, and what it got back.

Quick start

uv sync
just run          # migrates the sqlite db, starts http://localhost:8900
                  # (:8900 because ScrapeCraft's API owns :8000; override
                  # with SCRAPER_TESTER_PORT)

Test apps

URL	Tests
/profile/	Parsing page sections (lists, tables, spans) and downloading images
/search/	Form submission — direct GET or headless-browser click
/headers/	Header/proxy verification — echoes your request (HTML or ?format=json)
/shop/	Extraction mini-site: paginated listings (rel=next), A/B card layouts, detail pages, carousels for grouped extraction, JSON API, sitemap index + URL feed
/paginate/	Every next-link style (rel / text / class / id) plus a 60-page chain for depth caps
/trouble/	Failures & validation: statuses, flaky-then-recover, slow, oversize, empty, mislabeled content-type, redirect chains, marker phrases, a 429 rate limiter
/crawl/	Interlinked mini-wiki: keep/ tree with chains and cycles, skip/ for deny rules, robots-disallowed private/, dead links, external links, a 100-link page
/hooks/	Webhook sink — records every POST to /hooks/sink (headers, signature, body) for webhook-delivery verification
/js/	JS-only content: client-side/delayed rendering, load-more, infinite scroll, and an action gauntlet (type → Enter → select → hover → click_all → scroll → click)
/botwall/	Challenge page whose pass cookie only JS can mint, with a short server TTL — bot_trigger validation and the browser-warms-cookie-for-httpx session-refresh flow
/login/	Form + JS login (scraper / tester-pass-1337) guarding account pages — LoginConfig, session pools, generator warming with credentials

/robots.txt disallows /crawl/private/ and /trouble/ for robots-compliance scrapers (it's advisory — the server never blocks anything itself).

ScrapeCraft test scrapers

scrapecraft/ holds committed scraper/extractor/parser definitions that exercise these apps — one per ScrapeCraft feature. just sync-scrapers upserts them into a local ScrapeCraft instance, along with the input sets that hold each scraper's seed URLs; see scrapecraft/README.md.

All fake data is seeded (SCRAPER_TESTER_SEED, default 1337) — identical on every run, so scraper tests can assert exact values.

Admin — http://localhost:8000/admin/

• Per-app request rates: live charts (req/s by status class, last 5 min), totals, and average response times — measured server-side, so you can verify your scraper's configured rate without trusting the scraper.
• Live request log with full stored headers per request.
• Clear data per app, or all of it, for fresh stats.

Commands

just run            # migrate + start dev server (reload on)
just test           # pytest
just lint           # ruff check + format check
just makemigration "msg"
just clean-db       # delete the sqlite db

Adding a test app

Copy src/scraper_tester/apps/_example/ and edit — no wiring needed. Full guide: docs/adding-an-app.md.