Partitioning

Overviewing basics of event processing in Event-Driven Architectures:

• Typical structure of microservice
• Typical types of event transformations, 2 branching scenarios, merging streams
• Repartitioning events and when it can be useful
• Copartitioning events and when it is needed
• Assigning Partitions to a Consumer Instance. Three strategies to do this.
• Recovering from stateless processing instance failures.

These topics are disclosed in the Chapter 5 of the book we are currently studying:

“Building Event-Driven Microservices: Leveraging Organizational Data at Scale” by Adam Bellemare

Introduction into Event-driven microservice (EDM) architecture consists of the following topics:

• two topologies
• content of events
• three types of events
• table-stream duality
• schemas for defining event data
• single writer principle
• features of event broker
• event broker vs message broker
• single source of truth principle
• scaling using containers and VMs
• microservice taxes that we should pay

These topics are disclosed in the Chapter 2 of the book we are currently studying:

Earlier the book club of our company has studied excellent book:

Martin Kleppmann - Designing Data-Intensive Applications

This is the best book I have read about building complex scalable software systems. 💪

As usually I prepared an overview and mind-map.

Chapter 10 discovers all aspects about big data Batch Processing. If your system needs to process some data then your DEV team should learn this info.

• Unix tools for batch processing and brilliant concept of pipes.
• MapReduce and Distribute File Systems. How this approach solves problems of Unix pipes. Fault Tolerance and Partitioning. Usage and implementations of Joins, Grouping, Mapping. Available tools and problems of this approach.
• What is beyond MapReduce. Dataflow engines, Graph processing, High-level APIs and MPP databases. Dealing with Fault Tolerance and Partitioning. Implementations, problems, what to use and when.

Download full mind map (PDF)

Earlier the book club of our company has studied excellent book:

Martin Kleppmann - Designing Data-Intensive Applications

This is the best book I have read about building complex scalable software systems. 💪

As usually I prepared an overview and mind-map.

Chapter 7 is all your DEV team should know about Transactions:

• The purpose of transactions
• The concept of transaction: ACID, BASE, single-object and multi-object transactions
• Weak Isolation Levels: Read Committed, Snapshot Isolation and Repeatable Read. Problems and preventing them: Lost Updates, Write Skew and Phantoms
• Serializability and three approaches: Actual Serial Execution, Two-Phase Locking (2PL), Serializable Snapshot Isolation (SSI). Potential problems and performance.

Download full mind map (PDF)

Earlier the book club of our company has studied excellent book:

Martin Kleppmann - Designing Data-Intensive Applications

This is the best book I have read about building complex scalable software systems. 💪

As usually I prepared an overview and mind-map.

Chapter 6 contains everything the DEV team should consider when designing storage for big data:

• Partition aka Shard aka Region aka Tablet aka vNode aka vBucket. It is another approach for storing the data in addition to Replication (reviewed in the previous chapter)
• How to partition key-value data (primary index). Problems with partitioning - skew and hotspot. Approaches: key range and hash of key.
• Partitioning for secondary indexes: Local index and Global index
• Rebalancing partitions as you grow. Bad and good aproaches, problems and how to deal with them. Manual vs automated rebalancing.
• Request routing. Different aproaches, issues and solutions.

Download full mind map (PDF)

Earlier the book club of our company has studied excellent book:

Martin Kleppmann - Designing Data-Intensive Applications

This is the best book I have read about building complex scalable software systems. 💪

As usually I prepared an overview and mind-map.

Chapter 5:

• Intro. How to scale apps. Replicating and partitioning.
• Three algos of replicating
• Single-leader Replication
  • Leaders and Followers
  • Sync and async replication
  • Adding new Followers
  • Handling node outages
  • Technical implementations and all potential problems
• Multi-Leader Replication
  • Use-cases when it is good
  • Handling write conflicts
  • Three topologies and potential problems
• Leaderless Replication
  • Writing to the database when a node is down
  • Quorums and problems with them
  • Detecting concurrent writes and how to resolve them

Download full mind map (PDF)