Stream

The 4th pattern to build microservices is to use Lightweight Frameworks.

Lightweight Frameworks provide similar functionality to Heavyweight Frameworks, but they heavily rely on:

• the event broker
• the container management system (CMS)

In many cases they exceed Heavyweight Frameworks.

Different apps can use any/different resources from the cluster which are better fit their needs. While still provide Scaling and Recovering from Failures (again by heavily relying on event broker and CMS).

The Event-Driven model heavily relies on the QUALITY of events.

Good quality events are:

• explicitly defined via contracts
• have comments
• support evolution with backward and forward compatibility
• support code generation
• breaking changes are well thought

Good events are implemented using the right tools:

• use Avro/Thrift/Protobuf formats and never use JSON!
• use the right event broker (such as Pulsar)

Good events are designed to:

• contain all the info needed by consumers
• use separate streams for each event type
• use the right data types for their fields (don’t use string for numbers, use enums, etc.)
• don’t use type field and fields based on it
• be as small as possible
• consider requirements from consumers
• not just signals - they contain all info needed by consumers

These topics are disclosed in the Chapter 3 of the book we are currently studying:

Earlier the book club of our company has studied excellent book:

Martin Kleppmann - Designing Data-Intensive Applications

This is the best book I have read about building complex scalable software systems. 💪

As usually I prepared an overview and mind-map.

Chapter 11 discovers all aspects about Stream Processing. If your system needs to process some data on-the-fly then your DEV team should learn this info.

• Approaches for transmitting events: Direct messaging, Messaging Systems and Partitioned Logs. Their implementations, pros and cons.
• How to use Streams for databases. Sync databases, Change Data Capture (CDC), Event Sourcing. State, Streams, and Immutability.
• Nuances of Processing Streams. Useful use cases, reasoning about Time, 3 types of stream Joins, Fault Tolerance.

Download full mind map (PDF)