Designing Data-Intensive Applications

Earlier the book club of our company has studied excellent book:

Martin Kleppmann - Designing Data-Intensive Applications

This is the best book I have read about building complex scalable software systems. 💪

As usually I prepared an overview and mind-map.

Chapter 12 is a summary of the book and a visionary view of the future.

• Data Integration.
  • Overview of the ways we have to integrate data.
  • Causality and why we need Total Order and Idempotency.
  • Transactions and Linearizability
  • Limitations of Total Order.
  • Lambda architecture and unifying batch and stream processing as the most perspective approach.
• Unbundling Databases.
  • Overview of composing data storages together.
  • Designing apps around Dataflow.
  • Usage of derived states.
• Aiming for Correctness: what problems to consider and how to deal with them.
  • End-to-end fencing token.
  • How to process multi-partition requests.
  • Timeliness and Integrity issues. Apology workflow in business.
  • Meta approach: Trust, but Verify.
• Doing the Right Thing.
  • Predictive Analytics is discriminating people! We have responsibility and accountability here.
  • Privacy is conflicting with Tracking. Total surveillance should be legislated and self-regulated.

Download full mind map (PDF)

Earlier the book club of our company has studied excellent book:

Martin Kleppmann - Designing Data-Intensive Applications

This is the best book I have read about building complex scalable software systems. 💪

As usually I prepared an overview and mind-map.

Chapter 11 discovers all aspects about Stream Processing. If your system needs to process some data on-the-fly then your DEV team should learn this info.

• Approaches for transmitting events: Direct messaging, Messaging Systems and Partitioned Logs. Their implementations, pros and cons.
• How to use Streams for databases. Sync databases, Change Data Capture (CDC), Event Sourcing. State, Streams, and Immutability.
• Nuances of Processing Streams. Useful use cases, reasoning about Time, 3 types of stream Joins, Fault Tolerance.

Download full mind map (PDF)

Earlier the book club of our company has studied excellent book:

Martin Kleppmann - Designing Data-Intensive Applications

This is the best book I have read about building complex scalable software systems. 💪

As usually I prepared an overview and mind-map.

Chapter 10 discovers all aspects about big data Batch Processing. If your system needs to process some data then your DEV team should learn this info.

• Unix tools for batch processing and brilliant concept of pipes.
• MapReduce and Distribute File Systems. How this approach solves problems of Unix pipes. Fault Tolerance and Partitioning. Usage and implementations of Joins, Grouping, Mapping. Available tools and problems of this approach.
• What is beyond MapReduce. Dataflow engines, Graph processing, High-level APIs and MPP databases. Dealing with Fault Tolerance and Partitioning. Implementations, problems, what to use and when.

Download full mind map (PDF)

Earlier the book club of our company has studied excellent book:

Martin Kleppmann - Designing Data-Intensive Applications

This is the best book I have read about building complex scalable software systems. 💪

As usually I prepared an overview and mind-map.

Chapter 9 tells about Consistency and Consensus in distributed systems. It covers the following topics:

• What is consistency and eventual consistency
• Linearizability. Why it is needed. Difference from Serializability. How to implement Linearizability. The cost of Linearizability. CAP theorem.
• Ordering Guarantees. What is ordering and causality. It’s relation to Linearizability. Sequence Number Ordering and how to implement it. Total Order Broadcast and how to implement it.
• Distributed Transactions and Consensus. Why we need Consensus and Distributed Transactions. How to implement them, related problems and software that helps.

Summary:

Earlier the book club of our company has studied excellent book:

Martin Kleppmann - Designing Data-Intensive Applications

This is the best book I have read about building complex scalable software systems. 💪

As usually I prepared an overview and mind-map.

Chapter 8 discovers non-database related problems of distributed systems. DEV teams should consider them when designing distributed software.

• Faults and Partial Failures. The need to build a reliable system from unreliable components.
• Problem 1: Unreliable Networks. Why they happen. How to detect them. How to deal with them. Synchronous and Asynchronous networks. Hybrid networks and emulating hybrid networks.
• Problem 2: Unreliable Clocks. Monotonic Clocks vs Time-of-Day Clocks. What to use and what not to use in different cases. Good practices.
• Problem 3: Knowledge, Truth, and Lies. What is Truth in distributed systems. The Byzantine Generals Problem. System Model and Reality: what to use for what cases.

Download full mind map (PDF)

Earlier the book club of our company has studied excellent book:

Martin Kleppmann - Designing Data-Intensive Applications

This is the best book I have read about building complex scalable software systems. 💪

As usually I prepared an overview and mind-map.

Chapter 7 is all your DEV team should know about Transactions:

• The purpose of transactions
• The concept of transaction: ACID, BASE, single-object and multi-object transactions
• Weak Isolation Levels: Read Committed, Snapshot Isolation and Repeatable Read. Problems and preventing them: Lost Updates, Write Skew and Phantoms
• Serializability and three approaches: Actual Serial Execution, Two-Phase Locking (2PL), Serializable Snapshot Isolation (SSI). Potential problems and performance.

Download full mind map (PDF)

Earlier the book club of our company has studied excellent book:

Martin Kleppmann - Designing Data-Intensive Applications

This is the best book I have read about building complex scalable software systems. 💪

As usually I prepared an overview and mind-map.

Chapter 6 contains everything the DEV team should consider when designing storage for big data:

• Partition aka Shard aka Region aka Tablet aka vNode aka vBucket. It is another approach for storing the data in addition to Replication (reviewed in the previous chapter)
• How to partition key-value data (primary index). Problems with partitioning - skew and hotspot. Approaches: key range and hash of key.
• Partitioning for secondary indexes: Local index and Global index
• Rebalancing partitions as you grow. Bad and good aproaches, problems and how to deal with them. Manual vs automated rebalancing.
• Request routing. Different aproaches, issues and solutions.

Download full mind map (PDF)

Earlier the book club of our company has studied excellent book:

Martin Kleppmann - Designing Data-Intensive Applications

This is the best book I have read about building complex scalable software systems. 💪

As usually I prepared an overview and mind-map.

Chapter 5:

• Intro. How to scale apps. Replicating and partitioning.
• Three algos of replicating
• Single-leader Replication
  • Leaders and Followers
  • Sync and async replication
  • Adding new Followers
  • Handling node outages
  • Technical implementations and all potential problems
• Multi-Leader Replication
  • Use-cases when it is good
  • Handling write conflicts
  • Three topologies and potential problems
• Leaderless Replication
  • Writing to the database when a node is down
  • Quorums and problems with them
  • Detecting concurrent writes and how to resolve them

Download full mind map (PDF)

Earlier this year the book club of our company has studied excellent book:

Martin Kleppmann - Designing Data-Intensive Applications

This is the best book I have read about building complex scalable software systems. 💪

As usually I prepared an overview and mind-map.

Chapter 4:

• What is evolvability. Backward and Forward compatibility
• Approaches to encode data:
  • JSON, XML, and their binary variants
  • Thrift and Protobuf
  • Apache Avro
• Models of data flow
  • Through databases
  • Through services: REST, SOAP, RPC and the future
  • Through message brokers - when they are better and when they are not

Much more details in the mind-map:

Earlier this year the book club of our company has studied excellent book:

Martin Kleppmann - Designing Data-Intensive Applications

This is the best book I have read about building complex scalable software systems. 💪

As usually I prepared an overview and mind-map.

Chapter 3:

• Data structures:
  • Log-structured. SSTables / LSM-trees (when we don’t update anything but write to the end). A very cool idea of how to store data.
    • Sorted files.
    • Indexes for each one.
    • Moreover, the indexes can be created not for all the records, because they are sorted, and if the sizes of the records are the same, then the search between two known indexed records is a binary O(log n) search.
    • We always write to the last file.
    • The process of merging files is a school algo: how to write O(n) merge of two sorted arrays into one sorted array.
  • Update-in-place. B trees (when we directly update records). Very smart too.
    • Trees - a simpler idea, but here it is very interesting sub-idea with segments that are tuned for the work of disk drives - and hence the possible problems with SSDs due to many segment rewrites
• Indexing: primary, secondary, multi-column, full-text
  • About indexes - it is obvious that miracles do not happen and additional structures are needed.
  • Interestingly, sometimes values can be stored inside the indexes.
  • Unfortunately, too few info about full-text search - it is interesting to learn more about it.
• OLTP vs OLAP. The clear separation of OLAP / OLTP is very interesting.
• Column-based storage.
  • There is a very interesting aspect about data compression: how many zeros, then how many ones - a kind of compressor. This is possible only in memory - just for one next SSTable

Much more details in the mind-map: