Consistent Hashing is a cornerstone technology for distributed systems. Many of software developers don’t realize it, but Consistent Hashing is needed in many places: load balancers, caches, CDNs, id generators, databases, chats / social networks, and many other systems.
This topic consists of:
Problem with rehashing and why we need hashing to be CONSISTENT Hash space and hash ring BASIC approach (introduced by Karger et al. at MIT) Advanced approach with VIRTUAL NODES These items are disclosed in a very interesting Chapter 5 of the book:
Four standard steps for system design interview. However, I would think about them wider: as about four initial steps to design the software.
Step 1. Understand the problem and establish design scope Step 2. Propose high-level design and get buy-in Step 3. Design deep dive Step 4. Wrap up The chapter 3 of the book discovers details about each step, good questions to ask (to think about), DO’s and DONT’s.
A great generic plan for scaling any app from zero to millions of users.
Single server setup Selection and usage of database Vertical scaling vs horizontal scaling approaches. And why you should prefer horizontal Adding load balancer for horizontal scaling Adding database replication for horizontal scaling Adding cache Adding CDN Stateless vs Stateful architecture and using external state storage Adding extra Data Centers Adding Message queue Adding Logging, Metrics, and Automation Scaling database (sharding) and futher steps… All of these is carefully but briefly disclosed in the Chapter 1 of the book:
How to integrate event-driven microservices with request-response APIs?
There are two types of external events:
Autonomously Generated Events (analytical events) Reactively Generated Events (events from request-reply) There are two approaches of processing and serving requests using stateful services:
using internal state stores (with silly or with smart routing) using external state stores (with regular or with composite microservice) Ways of handling requests within an event-driven workflow:
Earlier the book club of our company has studied excellent book:
Martin Kleppmann - Designing Data-Intensive Applications
This is the best book I have read about building complex scalable software systems. 💪
As usually I prepared an overview and mind-map.
Chapter 6 contains everything the DEV team should consider when designing storage for big data:
Partition aka Shard aka Region aka Tablet aka vNode aka vBucket. It is another approach for storing the data in addition to Replication (reviewed in the previous chapter) How to partition key-value data (primary index).
