Search Engine System Design (Indexing, Ranking & Scale)

Design a web search engine like Google that can index billions of web pages and return relevant results for user queries.

Constraints

Functional

Crawl and index web pages, search, rank by relevance, autocomplete, spell check

Non-functional

Billions of pages, < 100ms search, 99.99% uptime, index freshness

Scale

50B pages, ~50 KB/page; 5B queries/day, peak ~60K/s; index ~2.5 PB, compressed ~500 TB

Stages ahead

1Requirement Analysis
2API Design
3High-Level Design
4HLD Extensions
5Trade-offs