
Web Crawler System Design: Why It's Harder Than a While Loop
Master the web crawler system design interview: two-tier URL frontier, async DNS, Bloom filter dedup, and consistent hashing in 45 minutes.
Page 43 of 61 · Ideas for curious minds in tech.
Back to the archive
Master the web crawler system design interview: two-tier URL frontier, async DNS, Bloom filter dedup, and consistent hashing in 45 minutes.

The complete notification system design interview walkthrough: architecture, Kafka queues, fanout, idempotency, and how to spend your 45 minutes.

Master the Twitter news feed system design interview: fan-out on write, the celebrity problem, the hybrid model, and how to pace 45 minutes.

Master the Instagram system design interview: fan-out feed generation, presigned media uploads, SQL vs NoSQL tradeoffs, and a 45-minute time plan.

The complete WhatsApp system design interview walkthrough: WebSockets, Cassandra schema, presence, group fan-out, and how to fill 45 minutes.

Learn how to design Pastebin in a system design interview: URL generation, KGS, caching layers, storage split, expiration, and the 45-minute pacing guide.

URL shortener system design interview decoded: base62 encoding, distributed counters, 301 vs 302, and the caching hierarchy that handles 115K reads per second.

The Aho-Corasick algorithm matches every pattern in one O(n+M+k) pass. Failure links, output merging, and 14 implementations explained.

How to design a rate limiter in a system design interview: 5 algorithms, Redis implementation, and the clarification trap most candidates fall into.

The exponential search algorithm runs in O(log i) not O(log n), making it faster than binary search for front-heavy or unbounded sorted arrays.

Interpolation search algorithm achieves O(log log n) on uniform data by estimating probe position. Formula, proof, and 14 language implementations.

The ternary search algorithm finds the peak of any unimodal function in O(log n). Proof, 14 implementations, vs binary search, and golden section search.

Master the convex hull algorithm: Graham Scan, Andrew's Monotone Chain, cross product math, and 14 language implementations with worked LeetCode examples.

Centroid decomposition turns O(n²) tree path queries into O(n log n). Learn the algorithm, the LCA invariant, and 14 language implementations.

Sqrt decomposition splits an array into blocks for O(sqrt(n)) range queries and updates. Simpler than a segment tree and handles Mo's algorithm in 30 lines.

Mo's algorithm solves offline range queries in O(n√n). Includes zigzag proof, block size derivation, and implementations in 14 languages.

Master the eulerian path algorithm: degree conditions, Hierholzer's O(E) DFS, and 14 language implementations. Includes LeetCode 332 and 2097 walkthroughs.

Learn bridges and articulation points: the O(V+E) DFS algorithm that finds every cut edge and cut vertex in a graph, with 14 language implementations.

The maximum flow algorithm explained: Edmonds-Karp, residual graphs, the O(VE²) termination proof, max-flow min-cut theorem, and 14 language implementations.

Hopcroft-Karp algorithm solves maximum bipartite matching in O(E√V). Proof, König's theorem, 14 implementations, and LeetCode pattern recognition.

Tarjan's algorithm finds all strongly connected components in one DFS pass. O(V+E) time, O(V) space, with 14 language implementations.

Kosaraju's algorithm finds every strongly connected component in O(V+E) with two DFS passes. Full proof, 14 language implementations, and 2-SAT worked example.

The suffix automaton builds in O(n), fits 2n-1 states, and answers substring frequency, distinct count, and LCS queries. Implementations in 14 languages.

Fibonacci heap: O(1) amortized decrease-key with full proofs, cascading cuts demystified, and complete implementations in all 14 major languages.