Deterministic Stream Processing is a foundation for building scalable event-driven systems. The lack of determinism can be very painful for the business (imagine lost financial transactions, missed alerts, wrong data aggregation).
There are certain tips and tricks on how to implement determinism. Keywords are:
Timestamps Event Scheduling Watermarks Stream Time Also you should understand the nature of Out-of-Order and Late-Arriving Events. And strategies for handling them.
Also need to support reprocessing.
Overviewing basics of event processing in Event-Driven Architectures:
Typical structure of microservice Typical types of event transformations, 2 branching scenarios, merging streams Repartitioning events and when it can be useful Copartitioning events and when it is needed Assigning Partitions to a Consumer Instance. Three strategies to do this. Recovering from stateless processing instance failures. These topics are disclosed in the Chapter 5 of the book we are currently studying:
Data Liberation is the process of moving from monolith towards microservices by decoupling systems in terms of data dependencies.
There are three patterns for Data Liberation:
Query-based Log-based Table-base Each pattern has its own pros and cons, as well as other important considerations.
Data definition changes (data structure migrations) must also be supported by the chosen Data Liberation approach.
There are different Liberation frameworks/tools that simplify the process of Data Liberation.
The Event-Driven model heavily relies on the QUALITY of events.
Good quality events are:
explicitly defined via contracts have comments support evolution with backward and forward compatibility support code generation breaking changes are well thought Good events are implemented using the right tools:
use Avro/Thrift/Protobuf formats and never use JSON! use the right event broker (such as Pulsar) Good events are designed to:
contain all the info needed by consumers use separate streams for each event type use the right data types for their fields (don’t use string for numbers, use enums, etc.
Introduction into Event-driven microservice (EDM) architecture consists of the following topics:
two topologies content of events three types of events table-stream duality schemas for defining event data single writer principle features of event broker event broker vs message broker single source of truth principle scaling using containers and VMs microservice taxes that we should pay These topics are disclosed in the Chapter 2 of the book we are currently studying:
Our company’s books club has selected a book about Microservices. It is “Building Event-Driven Microservices: Leveraging Organizational Data at Scale” by Adam Bellemare
Chapter 1 contains intro info:
Types of architectures and differences between them:
Traditional monolith-style architectures Service-oriented architectures (SOAs) Event-driven microservice (EDM) architectures Levels of Communication Structures and related Conway’s Law:
business implementation data Problems with traditional architectures (monolith & SOA) when you need to:
