A crawler’s hard problem is not fetching a page — it’s staying cheap and unblocked at volume. Showtimes Crawler was built as the senior-crawler assessment for International Showtimes with a target of 25,000+ cinema sites, so the whole design question was: which parts must be centrally governed so that scaling doesn’t mean scaling the number of places a mistake can hide?
The shape is producer–consumer. POST /crawl validates and enqueues, returning a job id immediately; a BullMQ worker (Redis-backed) picks it up, fetches, extracts title, meta description, favicon, scripts, stylesheets, and images, and writes the result onto the job payload with 24-hour retention; clients poll GET /status/:id. The fetch step sits behind a pluggable PageFetcher with two interchangeable engines — Axios for plain HTTP, Puppeteer for pages that need JS — so engine choice is a per-job parameter rather than a fork in the codebase. Cheap first: the HTTP engine handles the common case, and the browser is the escalation, not the default. Anti-blocking lives in one place too: round-robin user-agent rotation, proxy rotation supporting authenticated http://user:pass@host:port, and worker-level rate limiting, all applied to every job by construction instead of by remembering to.
Operationally it’s built to be inspected. Swagger UI at /docs for interactive endpoint testing, a Bull Board dashboard at /admin/queues to watch live jobs and their return values, GET /health that reports 200 only when Redis and the worker are actually connected (503 otherwise, so a platform probe can’t be fooled by a live HTTP server with a dead worker), and a /roadmap walkthrough documenting the architecture and its shortcomings. Ships as a multi-stage Docker image running non-root with dumb-init for child-process reaping and a configurable system Chromium path, plus a Render blueprint that provisions the service and managed Redis with the connection string injected.
