Design a Large-Scale Web Crawler

Published 3 October 2026

System Design — Large-Scale Web Crawler

The system design round asked me to design a web crawler at very large scale. After clarification, the assumed requirements were roughly tens of billions of pages, periodic recrawling, output feeding a search index, and no JavaScript rendering requirement.

Crawling Policy and Deduplication

The interviewer went deeply into balancing crawl priority with per-host politeness, handling and caching robots.txt, URL-level and content-level deduplication, and what happens when a probabilistic deduplication mechanism produces a false positive.

Scale and Reliability

Other follow-ups covered crawler traps such as infinite calendars and dynamically generated URLs, distributing work across crawler nodes, recovering when workers fail, persisting frontier state, and identifying likely bottlenecks if the entire corpus had to be refreshed within 24 hours.