Data Storage

Key-Value stores are the most basic but widely used data storages.

Design of key-value store consists of understanding the following topics:

What do we want from key-value store?
Single server key-value store
DISTRIBUTED key-value store:
- CAP theorem
- Real-world trade-offs for distributed systems
System components:
- Data partition
- Data replication
- Consistency
- Inconsistency resolution: Versioning
- Handling all types of failures: Failure detection, Handling TEMPORARY failures, Handling PERMANENT failures, Handling data center outage
System architecture diagram
Write path
Read path

These items are disclosed in a very interesting Chapter 6 of the book:

Earlier the book club of our company has studied excellent book:

Martin Kleppmann - Designing Data-Intensive Applications

This is the best book I have read about building complex scalable software systems. 💪

As usually I prepared an overview and mind-map.

Chapter 12 is a summary of the book and a visionary view of the future.

Data Integration.
- Overview of the ways we have to integrate data.
- Causality and why we need Total Order and Idempotency.
- Transactions and Linearizability
- Limitations of Total Order.
- Lambda architecture and unifying batch and stream processing as the most perspective approach.
Unbundling Databases.
- Overview of composing data storages together.
- Designing apps around Dataflow.
- Usage of derived states.
Aiming for Correctness: what problems to consider and how to deal with them.
- End-to-end fencing token.
- How to process multi-partition requests.
- Timeliness and Integrity issues. Apology workflow in business.
- Meta approach: Trust, but Verify.
Doing the Right Thing.
- Predictive Analytics is discriminating people! We have responsibility and accountability here.
- Privacy is conflicting with Tracking. Total surveillance should be legislated and self-regulated.