ScrapeCraft

A self-hosted platform for managing and running web scrapers. It handles the whole download side of scraping — scheduling, queueing, rate limiting, proxy pools, sessions, retries, artifact storage, and metrics — and can turn downloaded pages into structured data with its extractor subsystem. Everything is driven through a REST API and a web UI.

Highlights

• Scrapers & runs — define a scraper (seed URLs, fetch method, validation, throttle, schedule), trigger runs via API/UI/cron, and watch per-URL task state live over SSE.
• Pluggable fetchers — httpx for fast HTTP(S), selenium-grid for real Chrome via a grid, seleniumbase for stealth UC Chrome; each runs on its own Celery worker pool, and you can write your own fetcher.
• Extractors — versioned, test-covered field pipelines that turn pages into structured data, with live per-page extraction, feedback loops (inject detail pages, feed items into downstream scrapers), drift analytics, and retention/GC. See docs/extractors.md.
• Operations built in — GCRA-based rate limiting, proxy pools, session/cookie generators, robots.txt gating, completion webhooks, dead-letter handling, Prometheus metrics at /metrics.
• Bring your own AI agent — an MCP server lets your own local agent read a stored page's structure, propose selectors, preview a draft across several pages, and add regression tests. No model runs inside ScrapeCraft; publishing stays with you.

Stack: Python 3.14 / FastAPI / Celery / SQLAlchemy + Alembic on Postgres and Valkey (Redis-compatible), with a React + Vite frontend. No auth yet — the MVP is open on localhost by design.

Getting started

Prerequisites: uv, Docker + Compose, just, Node 20+ (for the UI), and optionally overmind to run the whole stack from one terminal.

# 1. Install dependencies (backend + frontend; also creates .env from the example)
just setup

# 2. Environment — defaults work out of the box; edit .env if you want overrides
#    (the justfile auto-loads .env)

# 3. Infrastructure + database (postgres :5433, valkey :6380)
docker compose -f docker-compose.dev.yml up -d postgres valkey
just migrate

# 4. Run the whole core stack (API, workers, beat, frontend) via overmind
just dev

Then open the web UI at http://localhost:5173 (API docs at http://localhost:8000/api/docs).

No overmind? Run the pieces in separate terminals instead:

just api        # REST API on :8000
just worker     # httpx pool worker
just web        # frontend dev server on :5173
just beat       # only needed for cron schedules + webhook retries (run exactly one)

Smoke test — downloads three example.com pages end-to-end and prints the results:

uv run python examples/run_quickstart.py

For the full walkthrough (verification steps, browser pools, Apple Silicon notes, troubleshooting), see docs/DEVELOPMENT.md.

Configuration

Settings come from SCRAPECRAFT_* env vars or a .env file in the repo root; .env.example documents the common ones and defaults live in backend/scrapecraft/settings.py. The defaults match the dev compose services, so a fresh clone needs no .env at all. For any shared or production environment, generate a real encryption key:

python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
# → SCRAPECRAFT_ENCRYPTION_KEY=...

Reaching the frontend dev server through a reverse proxy? Vite rejects unknown hostnames ("Blocked request. This host is not allowed"). Allow yours with DEV_ALLOWED_HOSTS (comma-separated) in the root .env, then restart the frontend (overmind restart webui):

DEV_ALLOWED_HOSTS=dev.example.com,5173.example.net

Common commands

Command	What it does
just dev	run the whole core stack via overmind (Procfile.dev)
just api / just worker / just web	run the API / httpx worker / frontend individually
just beat	Celery beat scheduler (cron schedules, reconciler, webhook retries)
just extract-worker	worker for batch/live extraction runs
just selenium-worker / just seleniumbase-worker	opt-in browser pool workers
just migrate / just db-reset	apply migrations / wipe and rebuild the dev DB
just test / just lint / just format	pytest / every pre-commit hook (ruff, ty, anti-slop, deptry, oxlint, prettier) / ruff format
just hooks	install the pre-commit hooks (just setup does this too)
just web-test / just web-types	frontend tests / regenerate the typed API client (offline)
just web-types-check	fail if frontend/src/api/schema.d.ts is stale against the backend
just web-lint / just web-format	frontend oxlint + prettier check / prettier write + oxlint autofix
just test-integration	gated real-broker integration tests

Run just --list for everything.

Repository layout

backend/scrapecraft/   FastAPI app, Celery execution, services, models, migrations
frontend/              React + Vite UI
fetchers/              fetcher plugin packages (httpx, selenium-grid, seleniumbase, dummy)
packages/              scrapecraft-sdk (shared fetcher contract)
examples/              runnable scraper examples (see examples/README.md)
docs/                  guides + architecture/feature specs under docs/superpowers/

Documentation

• Local development & testing guide — fresh clone → running stack → real scrape
• Extractors — authoring guide for structured-data extraction
• Writing a fetcher — build and install a custom fetcher package
• MCP server — point your own local AI agent at a running instance
• Examples — copy-paste scraper configs
• Architecture spec: docs/superpowers/specs/2026-06-19-scrapecraft-architecture-design.md