Heavyweight Stream Processing Frameworks are another foundation/pattern to build your microservices.
These frameworks are highly scalable and allow you to efficiently solve many analytical tasks. But they are not always good for stateful event-driven microservice application patterns.
Heavyweight frameworks operate using centralized resource clusters, which may require additional operational overhead, monitoring, and coordination to integrate successfully into a microservice framework. However, recent innovations move these frameworks toward container management solutions (CMS) such as Kubernetes that should reduce your efforts.
Basic Producer and Consumer (BPC) is a simple pattern for creating microservices. It forms the foundation of stateless and stateful services.
You can use it for:
interfacing layer between event streams and legacy systems leverage external stream processing systems to augment capabilities. But it requires your extra effort for:
simple state materialization event scheduling timestamp-based decision making All of these is disclosed in the Chapter 10 of the book we are currently studying:
Function-as-a-Service (FaaS) is an area of cloud computing that is growing rapidly.
There are following issues to conside here:
Principles of FaaS Open source and 3rd party FaaS providers 4 components to consider when building microservices as functions Cold Start vs Warm Start Different Triggers that can start FaaS: Triggering Based on New Events, Triggering Based on Consumer Group Lag, Triggering on a Schedule, Triggering Using Webhooks, Triggering on Resource Events Maintaining State Two patterns of functions calling other functions: Event-driven and Direct-call Termination and Shutdown Tuning and Scaling All of these is disclosed in the Chapter 9 of the book we are currently studying:
There are two patterns of building event-driven microservices:
Choreography (without centralized coordination) Orchestration (with centralized coordination) Both have pros and cons (however what I’ve learnt is that Orchestration is a better pattern)
How to implement Distributed Transactions with Choreography and with Orchestration.
How to AVOID implementing Distributed Transactions - a Compensation Workflows approach.
All of these is disclosed in the Chapter 8 of the book we are currently studying:
Event-Driven Microservices need to materialize states. And it comes with the important things to think about. Two approaches to choose from:
Internal state store or External state store Both have pros and cons. Both have important scalability and recovery considerations.
Two approaches for changing data structures:
Bebuilding and Migration Transactions and how to emulate them in order to implement Effectively Once processing.
All of these is disclosed in the Chapter 7 of the book we are currently studying:
Deterministic Stream Processing is a foundation for building scalable event-driven systems. The lack of determinism can be very painful for the business (imagine lost financial transactions, missed alerts, wrong data aggregation).
There are certain tips and tricks on how to implement determinism. Keywords are:
Timestamps Event Scheduling Watermarks Stream Time Also you should understand the nature of Out-of-Order and Late-Arriving Events. And strategies for handling them.
Also need to support reprocessing.
How to talk to your users when you are building a Startup? What is the best book to read on this topic? Three common mistakes everybody makes. Five great questions you can ask in every user interview. How to talk to users during three stages: idea stage, prototype stage and launched stage.
All of this is disclosed in Y Combinator’s Startup School - Lesson “How to Talk to Users”.
As usually, here is my summary mind map:
Overviewing basics of event processing in Event-Driven Architectures:
Typical structure of microservice Typical types of event transformations, 2 branching scenarios, merging streams Repartitioning events and when it can be useful Copartitioning events and when it is needed Assigning Partitions to a Consumer Instance. Three strategies to do this. Recovering from stateless processing instance failures. These topics are disclosed in the Chapter 5 of the book we are currently studying:
Data Liberation is the process of moving from monolith towards microservices by decoupling systems in terms of data dependencies.
There are three patterns for Data Liberation:
Query-based Log-based Table-base Each pattern has its own pros and cons, as well as other important considerations.
Data definition changes (data structure migrations) must also be supported by the chosen Data Liberation approach.
There are different Liberation frameworks/tools that simplify the process of Data Liberation.
The Event-Driven model heavily relies on the QUALITY of events.
Good quality events are:
explicitly defined via contracts have comments support evolution with backward and forward compatibility support code generation breaking changes are well thought Good events are implemented using the right tools:
use Avro/Thrift/Protobuf formats and never use JSON! use the right event broker (such as Pulsar) Good events are designed to:
contain all the info needed by consumers use separate streams for each event type use the right data types for their fields (don’t use string for numbers, use enums, etc.
