tapas-archive

Archive Tapas web comics (and novels) before the site shuts down: one self-contained, shareable bundle per series plus a metadata-only searchable catalog.

Quickstart

just install                      # uv sync + pre-commit hooks
uv run tapas-archive --help

Pipeline

Every stage records its progress in data/state.sqlite, so any command can be stopped (Ctrl-C) and re-run; it picks up where it left off and never refetches finished work.

Step	Command	What it does
1a	tapas-archive discover sweep	Probes /series/{id} for every integer id (ids are dense, ~340k). Records hits with their slug. Complete but takes a few hours.
1b	tapas-archive discover landing	Walks the listing API behind the menu tabs (comics, community, mature, novels) per genre and sort. Fast, gives rich metadata, but each query caps at 10k results.
2	tapas-archive fetch series	Per series: saves the info page + full episode list to data/raw/series/{id}/ and parses them (title, creators, genres, tags, description, stats, schedule).
3	tapas-archive fetch episodes	Per episode: saves the viewer JSON and downloads every image straight away (image URLs carry a token that expires within the hour). Novels get content.html.
3b	tapas-archive fetch comments	Per episode: every top-level comment page (10 per page) and the replies under each thread, saved raw and as comments.json next to the episode. Run after the bulk content.
4	tapas-archive bundle	Writes series.json, cover/thumbnails, and an offline viewer (index.html + data.js) into each series folder and zips it to data/bundles/.
5	tapas-archive catalog build	Builds data/catalog/catalog.sqlite (FTS5 search over titles, descriptions, tags, creators) + copies covers. No episode content.
—	tapas-archive catalog serve	Local browse/search UI at http://127.0.0.1:8765/catalog/ that also opens archived series and bundle zips.
—	tapas-archive reparse	Re-runs the parsers over saved raw files after a parser fix, no network.
—	tapas-archive status	Counts per stage.
—	tapas-archive age-gated	Lists series whose info page sits behind age verification and writes data/age_gated.json (to revisit with a logged-in cookie).

Typical run:

uv run tapas-archive discover landing              # quick: ~15k series with metadata
uv run tapas-archive discover sweep                 # thorough: every id
uv run tapas-archive fetch series --concurrency 8
uv run tapas-archive fetch episodes --concurrency 8 --series-parallel 2
uv run tapas-archive fetch comments               # after the bulk; --max-pages caps huge threads
uv run tapas-archive bundle
uv run tapas-archive catalog build && uv run tapas-archive catalog serve

Useful flags: --id 12345 (repeatable) restricts any stage to specific series; --retry re-queues errors; --limit N bounds a run; --delay 0.5 adds per-request politeness; TAPAS_ARCHIVE_DATA=/Volumes/big/tapas relocates data/; TAPAS_ARCHIVE_DB=/var/lib/tapas/state.sqlite moves the state database on its own (keep it on local disk when the data root is NFS — SQLite's WAL locking doesn't work reliably over network filesystems). Both can go in .env.local.

Proxies: put TAPAS_PROXY=http://user:pass@host:8000 and optionally TAPAS_PROXY_PORTS=8001-8005 in .env.local (gitignored). Each port becomes its own route with its own rate limit and WAF cooldown, so five sticky ports give five times the throughput. A route that gets WAF-blocked or stops answering is paused (with growing cooldowns) and the others carry on; if every route is paused, requests wait rather than fail, so an unattended run stalls instead of marking work as errors, and resumes by itself once a proxy is unblocked. Rotating proxies means Ctrl-C, edit .env.local, re-run: every stage continues where it stopped.

TAPAS_CDN_DIRECT=1 sends image downloads (the bulk of the bytes) straight to the image CDN instead of through the proxies; site and API requests stay proxied. Image URLs are signed per path, not per IP, so this works across routes. If the CDN starts refusing the direct IP while the proxies still get through, images fall back to the proxies and direct is retried after a growing cooldown.

Staging on local disk (slow or flaky NFS store)

Thousands of small writes a minute can stall an NFS server, and on a hard mount a stalled write hangs the fetch with it. Instead, fetch into local disk and move settled files across in the background, so a store outage only pauses the mover:

# .env.local
TAPAS_ARCHIVE_DATA=/home/me/tapas-staging          # local disk: fetches write here
TAPAS_ARCHIVE_STORE=/mnt/nfs/tapas                  # bulk store: the mover fills this
TAPAS_ARCHIVE_DB=/home/me/tapas/state.sqlite

uv run tapas-archive move                           # second tmux window; a pass every 5 min

For bundle, the data root must be the store (bundles read every series folder), so stage only the zips: set TAPAS_ARCHIVE_BUNDLES=/home/me/tapas-staging/bundles and run tapas-archive move --from /home/me/tapas-staging to copy them into the store's bundles/.

The mover handles raw/, archive/ and bundles/, taking only files untouched for 2 minutes (--min-age), copying each via .part + rename, and deleting the local copy only after that succeeds. Its log line shows the local backlog; if that keeps growing, raise --workers or check the store. Before bundle/catalog, drain it with move --once --min-age 0 and point TAPAS_ARCHIVE_DATA back at the store.

Layout on disk

data/
  state.sqlite                    progress + parsed metadata (the source of truth)
  raw/series/{id}/                info.html, info.json, episodes.json, ep/{episode_id}.json
  archive/{id}-{slug}/            the shareable bundle folder
    index.html  data.js           offline viewer (double-click index.html)
    series.json  README.txt  cover.*
    episodes/{scene}-{episode_id}/  001.jpg …  episode.json  thumb.*  [content.html]  [comments.json]
  bundles/{id}-{slug}.zip         the same folder zipped
  catalog/catalog.sqlite          searchable metadata; catalog/index.html + covers/

Notes

• Anonymous access only: episodes the listing flags as paid / wait-until-free are recorded as locked and never requested (no content, no comments). status reports how many, and a later pass with an account can clear the flag and fetch them. Mature series' info pages are age-gated, but their episode lists and content still work, so they are archived with metadata from the listing API.
• Comments (with replies) are captured by fetch comments and shown under each episode in the offline viewer. Avatars are linked, not downloaded.
• All archive data is plain files under data/; point TAPAS_ARCHIVE_DATA at the big storage mount on the dedicated VM and TAPAS_ARCHIVE_DB at local disk.