Archive for 2022

Building Event-Driven Microservices - Chapter 6 - Deterministic Stream Processing

Building Event-Driven Microservices - Chapter 6 - Deterministic Stream Processing

Translations: RU
Deterministic Stream Processing is a foundation for building scalable event-driven systems. The lack of determinism can be very painful for the business (imagine lost financial transactions, missed alerts, wrong data aggregation). There are certain tips and tricks on how to implement determinism. Keywords are: Timestamps Event Scheduling Watermarks Stream Time Also you should understand the nature of Out-of-Order and Late-Arriving Events. And strategies for handling them. Also need to support reprocessing.
Startup School - How to Talk to Users

Startup School - How to Talk to Users

Translations: RU
How to talk to your users when you are building a Startup? What is the best book to read on this topic? Three common mistakes everybody makes. Five great questions you can ask in every user interview. How to talk to users during three stages: idea stage, prototype stage and launched stage. All of this is disclosed in Y Combinator’s Startup School - Lesson “How to Talk to Users”. As usually, here is my summary mind map:
Building Event-Driven Microservices - Chapter 5 - Event-Driven Processing Basics

Building Event-Driven Microservices - Chapter 5 - Event-Driven Processing Basics

Translations: RU
Overviewing basics of event processing in Event-Driven Architectures: Typical structure of microservice Typical types of event transformations, 2 branching scenarios, merging streams Repartitioning events and when it can be useful Copartitioning events and when it is needed Assigning Partitions to a Consumer Instance. Three strategies to do this. Recovering from stateless processing instance failures. These topics are disclosed in the Chapter 5 of the book we are currently studying:
Building Event-Driven Microservices - Chapter 4 - Integrating Event Driven Architectures with Existing Systems

Building Event-Driven Microservices - Chapter 4 - Integrating Event Driven Architectures with Existing Systems

Translations: RU
Data Liberation is the process of moving from monolith towards microservices by decoupling systems in terms of data dependencies. There are three patterns for Data Liberation: Query-based Log-based Table-base Each pattern has its own pros and cons, as well as other important considerations. Data definition changes (data structure migrations) must also be supported by the chosen Data Liberation approach. There are different Liberation frameworks/tools that simplify the process of Data Liberation.
Building Event-Driven Microservices - Chapter 3 - Communication and Data Contracts

Building Event-Driven Microservices - Chapter 3 - Communication and Data Contracts

Translations: RU
The Event-Driven model heavily relies on the QUALITY of events. Good quality events are: explicitly defined via contracts have comments support evolution with backward and forward compatibility support code generation breaking changes are well thought Good events are implemented using the right tools: use Avro/Thrift/Protobuf formats and never use JSON! use the right event broker (such as Pulsar) Good events are designed to: contain all the info needed by consumers use separate streams for each event type use the right data types for their fields (don’t use string for numbers, use enums, etc.
Ingenius EMPERA Blockchain and PoM

Ingenius EMPERA Blockchain and PoM

Translations: RU
I accidentally met with the founders of EMPERA blockchain. As you may know, despite the fact that I believe in the blockchain future, I am a bit skeptical about new projects and ideas in this field as there are many pure scams and unprofessional enthusiasts. However, after reading all the papers and after a deep talk with the founders I am now very excited about this particular project. And I want to share with you my insights (as well as a few skeptical unclarities to be 100% honest).
Building Event-Driven Microservices - Chapter 2 - Event Driven Microservice Fundamentals

Building Event-Driven Microservices - Chapter 2 - Event Driven Microservice Fundamentals

Translations: RU
Introduction into Event-driven microservice (EDM) architecture consists of the following topics: two topologies content of events three types of events table-stream duality schemas for defining event data single writer principle features of event broker event broker vs message broker single source of truth principle scaling using containers and VMs microservice taxes that we should pay These topics are disclosed in the Chapter 2 of the book we are currently studying:
Building Event-Driven Microservices - Chapter 1 - Why Event-Driven Microservices

Building Event-Driven Microservices - Chapter 1 - Why Event-Driven Microservices

Translations: RU
Our company’s books club has selected a book about Microservices. It is “Building Event-Driven Microservices: Leveraging Organizational Data at Scale” by Adam Bellemare Chapter 1 contains intro info: Types of architectures and differences between them: Traditional monolith-style architectures Service-oriented architectures (SOAs) Event-driven microservice (EDM) architectures Levels of Communication Structures and related Conway’s Law: business implementation data Problems with traditional architectures (monolith & SOA) when you need to:
Camunda - overview

Camunda - overview

Translations: RU

There are a number of businesses and business tasks when it is necessary to visually design, change and control the execution of business processes.

Large companies like SAP, IBM, Oracle play in this market and offer their heavy and expensive solutions.

But there is also a lighter and cheaper solution in this field.

Thinking in Bets - Chapter 6 - Mental Time Travel

Thinking in Bets - Chapter 6 - Mental Time Travel

Translations: RU

Did you know that losing $1000 and then winning $900 (i.e. getting a result of -$100 in total) is more pleasant than winning $1000 and then losing $900 (i.e. getting a result of +$100 in total)!??

What has happened in the RECENT PAST drives our emotional response much more than how we are doing overall! It is a problem if you are making important prediction.

There are to ways to overcome this problem:

Thinking in Bets - Chapter 4 - The Buddy System

Thinking in Bets - Chapter 4 - The Buddy System

Translations: RU

People are not ready to consider themselves the source of the problem they are facing (their skill), they blame other people/circumstances (luck). How to overcome this issue?

One of the solution is to create “truth-seeking group” with your colleagues/friends/partners/“buddies”. And such a group should follow specific rules that would increase rationality…

Thinking in Bets - Chapter 3 - Bet to learn

Thinking in Bets - Chapter 3 - Bet to learn

Translations: RU

How people should learn by watching outcomes and why they do not learn?

There are many unknown circumstances - so we do not know the real cause of the outcome. Luck also plays a big part here, so we don’t know if it’s luck or skill.

Also we are distorted with our biases:

  • self-serving bias
  • and inverted self-serving bias

How to overcome our biases and learn better?

Thinking in Bets - Chapter 2 - Wanna bet?

Thinking in Bets - Chapter 2 - Wanna bet?

Translations: RU

Poker players make decisions by making bets. Why do we all need to learn to make decisions using bets too?

There is a trap of “Motivated reasoning” bias. But, fortunately, there are facilitating questions that direct our minds in the right direction.

We must also move away from using the all-or-nothing confidence scale. Instead, we must learn to measure our confidence in percentages. This will not only help us win more, but it will also make us more credible communicators and make teamwork around us more efficient.

Communication Genius - Lesson 7 - I-message

Communication Genius - Lesson 7 - I-message

Translations: RU

It is impossible to live without conflicts. But how to make the conflict constructive?

Thomas Gordon, author of “Parent Effectiveness Training”, invented the “I-message” that is now taught in many business education programs. Lesson 8 of Communication Genius describes I-message in detail:

  • What NOT to say during conflicts
  • What is the I-message
  • What is the structure of the I-message
Thinking in Bets - Chapter 1 - Life is Poker, not Chess

Thinking in Bets - Chapter 1 - Life is Poker, not Chess

Translations: RU

Our Rational Answer books club has selected a book for February. It is “Thinking in Bets” by Annie Duke: https://www.amazon.com/Thinking-Bets-Making-Smarter-Decisions-ebook/dp/B074DG9LQF

This book is written by a multi-million dollar poker tournaments winner (and her brother is also a multi-million dollar poker tournaments winner - so it’s a family business ;))

She teaches how to apply poker thinking to make decisions in financial markets, strategic planning, human resources, law, entrepreneurship…

Ethereum and Solidity - Lesson 01 - What is Ethereum

Ethereum and Solidity - Lesson 01 - What is Ethereum

Translations: RU

There is quite a high demand for blockchain development these days. We do some experience with it as a company (one complex multi-year project), but personally I only had high-level knowledge and I wanted to improve it…

So me and few of my colleagues have started the course “Ethereum and Solidity: The Complete Developer’s Guide” (https://www.udemy.com/course/ethereum-and-solidity-the-complete-developers-guide/)

Noise - Part III - Noise in Predictive Judgments

Noise - Part III - Noise in Predictive Judgments

Translations: RU

How to measure the noise in our daily decisions?

Which is better - expert judgment or mechanical prediction using very simple models/rules? Surprisingly, the model of you beats you! Even a simple regression model based on your past decisions!

How to make rules/models even less noisy? How can rules/models become more biased sometimes?

Why don’t people always use models (if they are so good) and still rely on their bad judgment?

When do we not need models?

Communication Genius - Lesson 3 - Toasting

Communication Genius - Lesson 3 - Toasting

Translations: RU

I continue sharing my impression and knowledge from the Communication Genius course which I started in January 2022. Lesson 3 is full of useful information concerning SAYING TOASTS. There are tips what to say to be remembered and some phrases which sound good during eating and drinking in informal atmosphere.

Communication Genius - Lesson 2 - Elevator pitch

Communication Genius - Lesson 2 - Elevator pitch

Translations: RU

In January 2022, I started the Communication Genius course from one of the best public speaking coaches. Lesson 2 is full of simple but efficient instructions how to communicate quickly in a stressful situation (ELEVATOR PITCH). There are tips what to say and there are also some things which are better not to use in a quick conversation.

Clean Architecture - PART VI - Details

Clean Architecture - PART VI - Details

Translations: RU
The book club of our company has chosen a new wonderful book for reading: Robert Martin - Clean Architecture - a Craftsman’s Guide to Software Structure and Design 👍 The part VI undermines some foundations 😀: Do you know that Database is a “detail”? An unimportant minor low-level non-essential feature that can be neglected in architecture design! Do you know the same about the Web? It is just an unimportant IO device that should also be neglected in architecture design!
Communication Genius - Lesson 1 - Communication preparation algorithm

Communication Genius - Lesson 1 - Communication preparation algorithm

Translations: RU

In January 2022, I started the Communication Genius course from one of the best public speaking coaches - to be a bit less toxic and more efficient 🙂

Lesson 1 discovers:

  • What is the purpose of communication?
  • What are the two parts of success?
  • Why should you clearly define and write down your real goal?
  • How to consider your Audience?
  • What is the Format and its elements: Time, Place, Dress Code and Genre and how to use them?
  • How to choose the right Moment? Examples of good and bad moments?
Clean Architecture - PART V - Architecture

Clean Architecture - PART V - Architecture

Translations: RU
The book club of our company has chosen a new wonderful book for reading: Robert Martin - Clean Architecture - a Craftsman’s Guide to Software Structure and Design 👍 Fifth part of the book contains A LOT of useful information: What is software architecture? What type of interdependencies can exist? How to draw borderlines between components? What type of borders exist? How to distribute policies by the levels? What are business rules, entities and use cases?
Noise - Part II - Your Mind is a Measuring Instrument

Noise - Part II - Your Mind is a Measuring Instrument

Translations: RU

Interesting to learn how your mind makes judgments. Two types of judgments: predictive and evaluative. How to measure errors from bias and noise. How to measure the cost of noise. And how to deal with catastrophic consequences.

Drilling down the noise to three components: system noise, level noise, and pattern noise. How to measure (and compensate) the noises. Occasional noise as a part of pattern noise. How to compensate occasional noise: wisdom-of-crowds effect, crowd-within-the-one effect, second answer effect, and dialectical bootstrapping tool.

How the group work influences the noise. Informational cascade and social pressure cascade. How to deal with them. The effect of group polarization.

Noise - Part I - Finding Noise

Noise - Part I - Finding Noise

Translations: RU

Do you realize how seriously your organization is affected by bad decisions? There are two types of errors in human judgments: Bias and Noise. Bias is a very widely recognized problem, and you definitely heard about it. Noise is not so widely recognized. Yet, it affects MANY decisions in ALL the industries.

There is a huge Noise in government decisions. There is a huge Noise in commercial companies.

Noise can be made visible and can be reduced. It is harder to measure the noise in unique singular decisions, but it is still there and it still can be reduced.

Designing Data-Intensive Applications - Chapter 12 - The Future of Data Systems

Designing Data-Intensive Applications - Chapter 12 - The Future of Data Systems

Translations: RU
Earlier the book club of our company has studied excellent book: Martin Kleppmann - Designing Data-Intensive Applications This is the best book I have read about building complex scalable software systems. 💪 As usually I prepared an overview and mind-map. Chapter 12 is a summary of the book and a visionary view of the future. Data Integration. Overview of the ways we have to integrate data. Causality and why we need Total Order and Idempotency.
Designing Data-Intensive Applications - Chapter 11 - Stream Processing

Designing Data-Intensive Applications - Chapter 11 - Stream Processing

Translations: RU
Earlier the book club of our company has studied excellent book: Martin Kleppmann - Designing Data-Intensive Applications This is the best book I have read about building complex scalable software systems. 💪 As usually I prepared an overview and mind-map. Chapter 11 discovers all aspects about Stream Processing. If your system needs to process some data on-the-fly then your DEV team should learn this info. Approaches for transmitting events: Direct messaging, Messaging Systems and Partitioned Logs.
Designing Data-Intensive Applications - Chapter 10 - Batch Processing

Designing Data-Intensive Applications - Chapter 10 - Batch Processing

Translations: RU
Earlier the book club of our company has studied excellent book: Martin Kleppmann - Designing Data-Intensive Applications This is the best book I have read about building complex scalable software systems. 💪 As usually I prepared an overview and mind-map. Chapter 10 discovers all aspects about big data Batch Processing. If your system needs to process some data then your DEV team should learn this info. Unix tools for batch processing and brilliant concept of pipes.
Designing Data-Intensive Applications - Chapter 9 - Consistency and Consensus

Designing Data-Intensive Applications - Chapter 9 - Consistency and Consensus

Translations: RU
Earlier the book club of our company has studied excellent book: Martin Kleppmann - Designing Data-Intensive Applications This is the best book I have read about building complex scalable software systems. 💪 As usually I prepared an overview and mind-map. Chapter 9 tells about Consistency and Consensus in distributed systems. It covers the following topics: What is consistency and eventual consistency Linearizability. Why it is needed. Difference from Serializability.
Designing Data-Intensive Applications - Chapter 8 - The Trouble with Distributed Systems

Designing Data-Intensive Applications - Chapter 8 - The Trouble with Distributed Systems

Translations: RU
Earlier the book club of our company has studied excellent book: Martin Kleppmann - Designing Data-Intensive Applications This is the best book I have read about building complex scalable software systems. 💪 As usually I prepared an overview and mind-map. Chapter 8 discovers non-database related problems of distributed systems. DEV teams should consider them when designing distributed software. Faults and Partial Failures. The need to build a reliable system from unreliable components.
Designing Data-Intensive Applications - Chapter 7 - Transactions

Designing Data-Intensive Applications - Chapter 7 - Transactions

Translations: RU
Earlier the book club of our company has studied excellent book: Martin Kleppmann - Designing Data-Intensive Applications This is the best book I have read about building complex scalable software systems. 💪 As usually I prepared an overview and mind-map. Chapter 7 is all your DEV team should know about Transactions: The purpose of transactions The concept of transaction: ACID, BASE, single-object and multi-object transactions Weak Isolation Levels: Read Committed, Snapshot Isolation and Repeatable Read.
Designing Data-Intensive Applications - Chapter 6 - Partitioning

Designing Data-Intensive Applications - Chapter 6 - Partitioning

Translations: RU
Earlier the book club of our company has studied excellent book: Martin Kleppmann - Designing Data-Intensive Applications This is the best book I have read about building complex scalable software systems. 💪 As usually I prepared an overview and mind-map. Chapter 6 contains everything the DEV team should consider when designing storage for big data: Partition aka Shard aka Region aka Tablet aka vNode aka vBucket. It is another approach for storing the data in addition to Replication (reviewed in the previous chapter) How to partition key-value data (primary index).
Designing Data-Intensive Applications - Chapter 5 - Replication

Designing Data-Intensive Applications - Chapter 5 - Replication

Translations: RU
Earlier the book club of our company has studied excellent book: Martin Kleppmann - Designing Data-Intensive Applications This is the best book I have read about building complex scalable software systems. 💪 As usually I prepared an overview and mind-map. Chapter 5: Intro. How to scale apps. Replicating and partitioning. Three algos of replicating Single-leader Replication Leaders and Followers Sync and async replication Adding new Followers Handling node outages Technical implementations and all potential problems Multi-Leader Replication Use-cases when it is good Handling write conflicts Three topologies and potential problems Leaderless Replication Writing to the database when a node is down Quorums and problems with them Detecting concurrent writes and how to resolve them Download full mind map (PDF)