Deterministic Stream Processing is a foundation for building scalable event-driven systems. The lack of determinism can be very painful for the business (imagine lost financial transactions, missed alerts, wrong data aggregation).
There are certain tips and tricks on how to implement determinism. Keywords are:
Timestamps Event Scheduling Watermarks Stream Time Also you should understand the nature of Out-of-Order and Late-Arriving Events. And strategies for handling them.
Also need to support reprocessing.
Overviewing basics of event processing in Event-Driven Architectures:
Typical structure of microservice Typical types of event transformations, 2 branching scenarios, merging streams Repartitioning events and when it can be useful Copartitioning events and when it is needed Assigning Partitions to a Consumer Instance. Three strategies to do this. Recovering from stateless processing instance failures. These topics are disclosed in the Chapter 5 of the book we are currently studying:
Data Liberation is the process of moving from monolith towards microservices by decoupling systems in terms of data dependencies.
There are three patterns for Data Liberation:
Query-based Log-based Table-base Each pattern has its own pros and cons, as well as other important considerations.
Data definition changes (data structure migrations) must also be supported by the chosen Data Liberation approach.
There are different Liberation frameworks/tools that simplify the process of Data Liberation.
The Event-Driven model heavily relies on the QUALITY of events.
Good quality events are:
explicitly defined via contracts have comments support evolution with backward and forward compatibility support code generation breaking changes are well thought Good events are implemented using the right tools:
use Avro/Thrift/Protobuf formats and never use JSON! use the right event broker (such as Pulsar) Good events are designed to:
contain all the info needed by consumers use separate streams for each event type use the right data types for their fields (don’t use string for numbers, use enums, etc.
Introduction into Event-driven microservice (EDM) architecture consists of the following topics:
two topologies content of events three types of events table-stream duality schemas for defining event data single writer principle features of event broker event broker vs message broker single source of truth principle scaling using containers and VMs microservice taxes that we should pay These topics are disclosed in the Chapter 2 of the book we are currently studying:
Our company’s books club has selected a book about Microservices. It is “Building Event-Driven Microservices: Leveraging Organizational Data at Scale” by Adam Bellemare
Chapter 1 contains intro info:
Types of architectures and differences between them:
Traditional monolith-style architectures Service-oriented architectures (SOAs) Event-driven microservice (EDM) architectures Levels of Communication Structures and related Conway’s Law:
business implementation data Problems with traditional architectures (monolith & SOA) when you need to:
Did you know that losing $1000 and then winning $900 (i.e. getting a result of -$100 in total) is more pleasant than winning $1000 and then losing $900 (i.e. getting a result of +$100 in total)!??
What has happened in the RECENT PAST drives our emotional response much more than how we are doing overall! It is a problem if you are making important prediction.
I’ve already told about the truthseeking group (The Buddy System) as a great way to improve decision-making process.
How to share information inside such a group and how to disagree efficiently (to find a good solution)? And how to communicate outside of such efficient group?
There is a set of rules called “CUDOS” that can be used for this…
People are not ready to consider themselves the source of the problem they are facing (their skill), they blame other people/circumstances (luck). How to overcome this issue?
One of the solution is to create “truth-seeking group” with your colleagues/friends/partners/“buddies”. And such a group should follow specific rules that would increase rationality…
How people should learn by watching outcomes and why they do not learn?
There are many unknown circumstances - so we do not know the real cause of the outcome. Luck also plays a big part here, so we don’t know if it’s luck or skill.
