Archive for 2022

System Design Interview - Chapter 4 - Design a Rate Limiter

System Design Interview - Chapter 4 - Design a Rate Limiter

Translations: RU
Every popular software should have a Rate Limiter. It prevents DDOS attack, reduces cost and prevents servers from being overloaded. There are some tricky questions to be considered during implementation of Rate Limiter: Where to put Rate Limiter: client-side, server-side, gateway? Algorithms for rate limiting. There are many algorithms with pros and cons: Token bucket, Leaking bucket, Fixed window counter, Sliding window log, Sliding window counter. Your business needs will define the right algorithm.
System Design Interview - Chapter 3 - A Framework for System Design Interviews

System Design Interview - Chapter 3 - A Framework for System Design Interviews

Translations: RU
Four standard steps for system design interview. However, I would think about them wider: as about four initial steps to design the software. Step 1. Understand the problem and establish design scope Step 2. Propose high-level design and get buy-in Step 3. Design deep dive Step 4. Wrap up The chapter 3 of the book discovers details about each step, good questions to ask (to think about), DO’s and DONT’s.
System Design Interview - Chapter 2 - Back-to-the-envelope estimation

System Design Interview - Chapter 2 - Back-to-the-envelope estimation

Translations: RU
A very short Chapter 2 is about how to make rough estimates to start from the most important parts when designing the software. Some concepts that EVERY software developer should know: “Power of two” Standard latency numbers! How fast is memory, how slow is disk, etc… Availability numbers What are key metrics you should think about These concepts and numbers are disclosed in chapter two of the book:
System Design Interview - Chapter 1 - Scale from zero to millions of users

System Design Interview - Chapter 1 - Scale from zero to millions of users

Translations: RU
A great generic plan for scaling any app from zero to millions of users. Single server setup Selection and usage of database Vertical scaling vs horizontal scaling approaches. And why you should prefer horizontal Adding load balancer for horizontal scaling Adding database replication for horizontal scaling Adding cache Adding CDN Stateless vs Stateful architecture and using external state storage Adding extra Data Centers Adding Message queue Adding Logging, Metrics, and Automation Scaling database (sharding) and futher steps… All of these is carefully but briefly disclosed in the Chapter 1 of the book:
Building Event-Driven Microservices - Chapter 16 Deploying Event-Driven Microservices

Building Event-Driven Microservices - Chapter 16 Deploying Event-Driven Microservices

Translations: RU
Principles of Microservice Deployment Differentes between Continuous integration (CI), Continuous delivery and Continuous deployment. Deployment patterns: The Basic Full-Stop Deployment Pattern The Rolling Update Pattern The Breaking Schema Change Pattern: two options here - Eventual Migration and Synchronized Migration The Blue-Green Deployment Pattern All of these is disclosed in the Chapter 16 of the book: “Building Event-Driven Microservices: Leveraging Organizational Data at Scale” by Adam Bellemare Sharing my mind map with all the details as usual:
Building Event-Driven Microservices - Chapter 15 - Testing Event Driven Microservices.

Building Event-Driven Microservices - Chapter 15 - Testing Event Driven Microservices.

Translations: RU
Testing event-driven solutions is a challenging task. It includes: Common principles Unit testing: stateless and stateful Testing the Topology (I bet you didn’t do it! ;) Testing Schema Evolution and Compatibility Integration testing: Local, Remote and Hybrid - a set of common sense considerations not to forgot about Choosing the testing strategy All of these is disclosed in the Chapter 15 of the book: “Building Event-Driven Microservices: Leveraging Organizational Data at Scale” by Adam Bellemare
Building Event-Driven Microservices - Chapter 14 - Supportive Tooling

Building Event-Driven Microservices - Chapter 14 - Supportive Tooling

Translations: RU
Quite often the tooling is a forgotten area during the beginning of development. This chapter overlooks the tooling that can help build and maintain event-driven microservices. Useful checklist! This topic includes: Org rules Documentation, tagging Quotas Schema registry Offset management ACL State management and app reset Consumer offset lag monitoring Container Management Controls Cluster Creation and Management Dependency Tracking All of these is disclosed in the Chapter 14 of the book:
Building Event-Driven Microservices - Chapter 13 - Integrating Event-Driven and Request-Response Microservices

Building Event-Driven Microservices - Chapter 13 - Integrating Event-Driven and Request-Response Microservices

Translations: RU
How to integrate event-driven microservices with request-response APIs? There are two types of external events: Autonomously Generated Events (analytical events) Reactively Generated Events (events from request-reply) There are two approaches of processing and serving requests using stateful services: using internal state stores (with silly or with smart routing) using external state stores (with regular or with composite microservice) Ways of handling requests within an event-driven workflow:
Building Event-Driven Microservices - Chapter 12 - Lightweight Framework Microservices

Building Event-Driven Microservices - Chapter 12 - Lightweight Framework Microservices

Translations: RU
The 4th pattern to build microservices is to use Lightweight Frameworks. Lightweight Frameworks provide similar functionality to Heavyweight Frameworks, but they heavily rely on: the event broker the container management system (CMS) In many cases they exceed Heavyweight Frameworks. Different apps can use any/different resources from the cluster which are better fit their needs. While still provide Scaling and Recovering from Failures (again by heavily relying on event broker and CMS).
Building Event-Driven Microservices - Chapter 11 - Heavyweight Framework Microservices

Building Event-Driven Microservices - Chapter 11 - Heavyweight Framework Microservices

Translations: RU
Heavyweight Stream Processing Frameworks are another foundation/pattern to build your microservices. These frameworks are highly scalable and allow you to efficiently solve many analytical tasks. But they are not always good for stateful event-driven microservice application patterns. Heavyweight frameworks operate using centralized resource clusters, which may require additional operational overhead, monitoring, and coordination to integrate successfully into a microservice framework. However, recent innovations move these frameworks toward container management solutions (CMS) such as Kubernetes that should reduce your efforts.
Building Event-Driven Microservices - Chapter 10 - Basic Producer and Consumer Microservices

Building Event-Driven Microservices - Chapter 10 - Basic Producer and Consumer Microservices

Translations: RU
Basic Producer and Consumer (BPC) is a simple pattern for creating microservices. It forms the foundation of stateless and stateful services. You can use it for: interfacing layer between event streams and legacy systems leverage external stream processing systems to augment capabilities. But it requires your extra effort for: simple state materialization event scheduling timestamp-based decision making All of these is disclosed in the Chapter 10 of the book we are currently studying:
Building Event-Driven Microservices - Chapter 9 - Microservices Using Function as a Service

Building Event-Driven Microservices - Chapter 9 - Microservices Using Function as a Service

Translations: RU
Function-as-a-Service (FaaS) is an area of cloud computing that is growing rapidly. There are following issues to conside here: Principles of FaaS Open source and 3rd party FaaS providers 4 components to consider when building microservices as functions Cold Start vs Warm Start Different Triggers that can start FaaS: Triggering Based on New Events, Triggering Based on Consumer Group Lag, Triggering on a Schedule, Triggering Using Webhooks, Triggering on Resource Events Maintaining State Two patterns of functions calling other functions: Event-driven and Direct-call Termination and Shutdown Tuning and Scaling All of these is disclosed in the Chapter 9 of the book we are currently studying:
Building Event-Driven Microservices - Chapter 8 - Building Workflows with Microservices

Building Event-Driven Microservices - Chapter 8 - Building Workflows with Microservices

Translations: RU
There are two patterns of building event-driven microservices: Choreography (without centralized coordination) Orchestration (with centralized coordination) Both have pros and cons (however what I’ve learnt is that Orchestration is a better pattern) How to implement Distributed Transactions with Choreography and with Orchestration. How to AVOID implementing Distributed Transactions - a Compensation Workflows approach. All of these is disclosed in the Chapter 8 of the book we are currently studying:
Building Event-Driven Microservices - Chapter 7 - Stateful-Streaming

Building Event-Driven Microservices - Chapter 7 - Stateful-Streaming

Translations: RU
Event-Driven Microservices need to materialize states. And it comes with the important things to think about. Two approaches to choose from: Internal state store or External state store Both have pros and cons. Both have important scalability and recovery considerations. Two approaches for changing data structures: Bebuilding and Migration Transactions and how to emulate them in order to implement Effectively Once processing. All of these is disclosed in the Chapter 7 of the book we are currently studying:
Building Event-Driven Microservices - Chapter 6 - Deterministic Stream Processing

Building Event-Driven Microservices - Chapter 6 - Deterministic Stream Processing

Translations: RU
Deterministic Stream Processing is a foundation for building scalable event-driven systems. The lack of determinism can be very painful for the business (imagine lost financial transactions, missed alerts, wrong data aggregation). There are certain tips and tricks on how to implement determinism. Keywords are: Timestamps Event Scheduling Watermarks Stream Time Also you should understand the nature of Out-of-Order and Late-Arriving Events. And strategies for handling them. Also need to support reprocessing.
Startup School - How to Talk to Users

Startup School - How to Talk to Users

Translations: RU
How to talk to your users when you are building a Startup? What is the best book to read on this topic? Three common mistakes everybody makes. Five great questions you can ask in every user interview. How to talk to users during three stages: idea stage, prototype stage and launched stage. All of this is disclosed in Y Combinator’s Startup School - Lesson “How to Talk to Users”. As usually, here is my summary mind map:
Building Event-Driven Microservices - Chapter 5 - Event-Driven Processing Basics

Building Event-Driven Microservices - Chapter 5 - Event-Driven Processing Basics

Translations: RU
Overviewing basics of event processing in Event-Driven Architectures: Typical structure of microservice Typical types of event transformations, 2 branching scenarios, merging streams Repartitioning events and when it can be useful Copartitioning events and when it is needed Assigning Partitions to a Consumer Instance. Three strategies to do this. Recovering from stateless processing instance failures. These topics are disclosed in the Chapter 5 of the book we are currently studying:
Building Event-Driven Microservices - Chapter 4 - Integrating Event Driven Architectures with Existing Systems

Building Event-Driven Microservices - Chapter 4 - Integrating Event Driven Architectures with Existing Systems

Translations: RU
Data Liberation is the process of moving from monolith towards microservices by decoupling systems in terms of data dependencies. There are three patterns for Data Liberation: Query-based Log-based Table-base Each pattern has its own pros and cons, as well as other important considerations. Data definition changes (data structure migrations) must also be supported by the chosen Data Liberation approach. There are different Liberation frameworks/tools that simplify the process of Data Liberation.
Building Event-Driven Microservices - Chapter 3 - Communication and Data Contracts

Building Event-Driven Microservices - Chapter 3 - Communication and Data Contracts

Translations: RU
The Event-Driven model heavily relies on the QUALITY of events. Good quality events are: explicitly defined via contracts have comments support evolution with backward and forward compatibility support code generation breaking changes are well thought Good events are implemented using the right tools: use Avro/Thrift/Protobuf formats and never use JSON! use the right event broker (such as Pulsar) Good events are designed to: contain all the info needed by consumers use separate streams for each event type use the right data types for their fields (don’t use string for numbers, use enums, etc.
Ingenius EMPERA Blockchain and PoM

Ingenius EMPERA Blockchain and PoM

Translations: RU
I accidentally met with the founders of EMPERA blockchain. As you may know, despite the fact that I believe in the blockchain future, I am a bit skeptical about new projects and ideas in this field as there are many pure scams and unprofessional enthusiasts. However, after reading all the papers and after a deep talk with the founders I am now very excited about this particular project. And I want to share with you my insights (as well as a few skeptical unclarities to be 100% honest).
Building Event-Driven Microservices - Chapter 2 - Event Driven Microservice Fundamentals

Building Event-Driven Microservices - Chapter 2 - Event Driven Microservice Fundamentals

Translations: RU
Introduction into Event-driven microservice (EDM) architecture consists of the following topics: two topologies content of events three types of events table-stream duality schemas for defining event data single writer principle features of event broker event broker vs message broker single source of truth principle scaling using containers and VMs microservice taxes that we should pay These topics are disclosed in the Chapter 2 of the book we are currently studying:
Building Event-Driven Microservices - Chapter 1 - Why Event-Driven Microservices

Building Event-Driven Microservices - Chapter 1 - Why Event-Driven Microservices

Translations: RU
Our company’s books club has selected a book about Microservices. It is “Building Event-Driven Microservices: Leveraging Organizational Data at Scale” by Adam Bellemare Chapter 1 contains intro info: Types of architectures and differences between them: Traditional monolith-style architectures Service-oriented architectures (SOAs) Event-driven microservice (EDM) architectures Levels of Communication Structures and related Conway’s Law: business implementation data Problems with traditional architectures (monolith & SOA) when you need to:
Camunda - overview

Camunda - overview

Translations: RU

There are a number of businesses and business tasks when it is necessary to visually design, change and control the execution of business processes.

Large companies like SAP, IBM, Oracle play in this market and offer their heavy and expensive solutions.

But there is also a lighter and cheaper solution in this field.

Thinking in Bets - Chapter 6 - Mental Time Travel

Thinking in Bets - Chapter 6 - Mental Time Travel

Translations: RU

Did you know that losing $1000 and then winning $900 (i.e. getting a result of -$100 in total) is more pleasant than winning $1000 and then losing $900 (i.e. getting a result of +$100 in total)!??

What has happened in the RECENT PAST drives our emotional response much more than how we are doing overall! It is a problem if you are making important prediction.

There are to ways to overcome this problem:

Thinking in Bets - Chapter 4 - The Buddy System

Thinking in Bets - Chapter 4 - The Buddy System

Translations: RU

People are not ready to consider themselves the source of the problem they are facing (their skill), they blame other people/circumstances (luck). How to overcome this issue?

One of the solution is to create “truth-seeking group” with your colleagues/friends/partners/“buddies”. And such a group should follow specific rules that would increase rationality…

Thinking in Bets - Chapter 3 - Bet to learn

Thinking in Bets - Chapter 3 - Bet to learn

Translations: RU

How people should learn by watching outcomes and why they do not learn?

There are many unknown circumstances - so we do not know the real cause of the outcome. Luck also plays a big part here, so we don’t know if it’s luck or skill.

Also we are distorted with our biases:

  • self-serving bias
  • and inverted self-serving bias

How to overcome our biases and learn better?

Thinking in Bets - Chapter 2 - Wanna bet?

Thinking in Bets - Chapter 2 - Wanna bet?

Translations: RU

Poker players make decisions by making bets. Why do we all need to learn to make decisions using bets too?

There is a trap of “Motivated reasoning” bias. But, fortunately, there are facilitating questions that direct our minds in the right direction.

We must also move away from using the all-or-nothing confidence scale. Instead, we must learn to measure our confidence in percentages. This will not only help us win more, but it will also make us more credible communicators and make teamwork around us more efficient.

Communication Genius - Lesson 7 - I-message

Communication Genius - Lesson 7 - I-message

Translations: RU

It is impossible to live without conflicts. But how to make the conflict constructive?

Thomas Gordon, author of “Parent Effectiveness Training”, invented the “I-message” that is now taught in many business education programs. Lesson 8 of Communication Genius describes I-message in detail:

  • What NOT to say during conflicts
  • What is the I-message
  • What is the structure of the I-message
Thinking in Bets - Chapter 1 - Life is Poker, not Chess

Thinking in Bets - Chapter 1 - Life is Poker, not Chess

Translations: RU

Our Rational Answer books club has selected a book for February. It is “Thinking in Bets” by Annie Duke: https://www.amazon.com/Thinking-Bets-Making-Smarter-Decisions-ebook/dp/B074DG9LQF

This book is written by a multi-million dollar poker tournaments winner (and her brother is also a multi-million dollar poker tournaments winner - so it’s a family business ;))

She teaches how to apply poker thinking to make decisions in financial markets, strategic planning, human resources, law, entrepreneurship…

Ethereum and Solidity - Lesson 01 - What is Ethereum

Ethereum and Solidity - Lesson 01 - What is Ethereum

Translations: RU

There is quite a high demand for blockchain development these days. We do some experience with it as a company (one complex multi-year project), but personally I only had high-level knowledge and I wanted to improve it…

So me and few of my colleagues have started the course “Ethereum and Solidity: The Complete Developer’s Guide” (https://www.udemy.com/course/ethereum-and-solidity-the-complete-developers-guide/)

Noise - Part III - Noise in Predictive Judgments

Noise - Part III - Noise in Predictive Judgments

Translations: RU

How to measure the noise in our daily decisions?

Which is better - expert judgment or mechanical prediction using very simple models/rules? Surprisingly, the model of you beats you! Even a simple regression model based on your past decisions!

How to make rules/models even less noisy? How can rules/models become more biased sometimes?

Why don’t people always use models (if they are so good) and still rely on their bad judgment?

When do we not need models?

Communication Genius - Lesson 3 - Toasting

Communication Genius - Lesson 3 - Toasting

Translations: RU

I continue sharing my impression and knowledge from the Communication Genius course which I started in January 2022. Lesson 3 is full of useful information concerning SAYING TOASTS. There are tips what to say to be remembered and some phrases which sound good during eating and drinking in informal atmosphere.

Communication Genius - Lesson 2 - Elevator pitch

Communication Genius - Lesson 2 - Elevator pitch

Translations: RU

In January 2022, I started the Communication Genius course from one of the best public speaking coaches. Lesson 2 is full of simple but efficient instructions how to communicate quickly in a stressful situation (ELEVATOR PITCH). There are tips what to say and there are also some things which are better not to use in a quick conversation.

Clean Architecture - PART VI - Details

Clean Architecture - PART VI - Details

Translations: RU
The book club of our company has chosen a new wonderful book for reading: Robert Martin - Clean Architecture - a Craftsman’s Guide to Software Structure and Design 👍 The part VI undermines some foundations 😀: Do you know that Database is a “detail”? An unimportant minor low-level non-essential feature that can be neglected in architecture design! Do you know the same about the Web? It is just an unimportant IO device that should also be neglected in architecture design!
Communication Genius - Lesson 1 - Communication preparation algorithm

Communication Genius - Lesson 1 - Communication preparation algorithm

Translations: RU

In January 2022, I started the Communication Genius course from one of the best public speaking coaches - to be a bit less toxic and more efficient 🙂

Lesson 1 discovers:

  • What is the purpose of communication?
  • What are the two parts of success?
  • Why should you clearly define and write down your real goal?
  • How to consider your Audience?
  • What is the Format and its elements: Time, Place, Dress Code and Genre and how to use them?
  • How to choose the right Moment? Examples of good and bad moments?
Clean Architecture - PART V - Architecture

Clean Architecture - PART V - Architecture

Translations: RU
The book club of our company has chosen a new wonderful book for reading: Robert Martin - Clean Architecture - a Craftsman’s Guide to Software Structure and Design 👍 Fifth part of the book contains A LOT of useful information: What is software architecture? What type of interdependencies can exist? How to draw borderlines between components? What type of borders exist? How to distribute policies by the levels? What are business rules, entities and use cases?
Noise - Part II - Your Mind is a Measuring Instrument

Noise - Part II - Your Mind is a Measuring Instrument

Translations: RU

Interesting to learn how your mind makes judgments. Two types of judgments: predictive and evaluative. How to measure errors from bias and noise. How to measure the cost of noise. And how to deal with catastrophic consequences.

Drilling down the noise to three components: system noise, level noise, and pattern noise. How to measure (and compensate) the noises. Occasional noise as a part of pattern noise. How to compensate occasional noise: wisdom-of-crowds effect, crowd-within-the-one effect, second answer effect, and dialectical bootstrapping tool.

How the group work influences the noise. Informational cascade and social pressure cascade. How to deal with them. The effect of group polarization.

Noise - Part I - Finding Noise

Noise - Part I - Finding Noise

Translations: RU

Do you realize how seriously your organization is affected by bad decisions? There are two types of errors in human judgments: Bias and Noise. Bias is a very widely recognized problem, and you definitely heard about it. Noise is not so widely recognized. Yet, it affects MANY decisions in ALL the industries.

There is a huge Noise in government decisions. There is a huge Noise in commercial companies.

Noise can be made visible and can be reduced. It is harder to measure the noise in unique singular decisions, but it is still there and it still can be reduced.

Designing Data-Intensive Applications - Chapter 12 - The Future of Data Systems

Designing Data-Intensive Applications - Chapter 12 - The Future of Data Systems

Translations: RU
Earlier the book club of our company has studied excellent book: Martin Kleppmann - Designing Data-Intensive Applications This is the best book I have read about building complex scalable software systems. 💪 As usually I prepared an overview and mind-map. Chapter 12 is a summary of the book and a visionary view of the future. Data Integration. Overview of the ways we have to integrate data. Causality and why we need Total Order and Idempotency.
Designing Data-Intensive Applications - Chapter 11 - Stream Processing

Designing Data-Intensive Applications - Chapter 11 - Stream Processing

Translations: RU
Earlier the book club of our company has studied excellent book: Martin Kleppmann - Designing Data-Intensive Applications This is the best book I have read about building complex scalable software systems. 💪 As usually I prepared an overview and mind-map. Chapter 11 discovers all aspects about Stream Processing. If your system needs to process some data on-the-fly then your DEV team should learn this info. Approaches for transmitting events: Direct messaging, Messaging Systems and Partitioned Logs.
Designing Data-Intensive Applications - Chapter 10 - Batch Processing

Designing Data-Intensive Applications - Chapter 10 - Batch Processing

Translations: RU
Earlier the book club of our company has studied excellent book: Martin Kleppmann - Designing Data-Intensive Applications This is the best book I have read about building complex scalable software systems. 💪 As usually I prepared an overview and mind-map. Chapter 10 discovers all aspects about big data Batch Processing. If your system needs to process some data then your DEV team should learn this info. Unix tools for batch processing and brilliant concept of pipes.
Designing Data-Intensive Applications - Chapter 9 - Consistency and Consensus

Designing Data-Intensive Applications - Chapter 9 - Consistency and Consensus

Translations: RU
Earlier the book club of our company has studied excellent book: Martin Kleppmann - Designing Data-Intensive Applications This is the best book I have read about building complex scalable software systems. 💪 As usually I prepared an overview and mind-map. Chapter 9 tells about Consistency and Consensus in distributed systems. It covers the following topics: What is consistency and eventual consistency Linearizability. Why it is needed. Difference from Serializability.
Designing Data-Intensive Applications - Chapter 8 - The Trouble with Distributed Systems

Designing Data-Intensive Applications - Chapter 8 - The Trouble with Distributed Systems

Translations: RU
Earlier the book club of our company has studied excellent book: Martin Kleppmann - Designing Data-Intensive Applications This is the best book I have read about building complex scalable software systems. 💪 As usually I prepared an overview and mind-map. Chapter 8 discovers non-database related problems of distributed systems. DEV teams should consider them when designing distributed software. Faults and Partial Failures. The need to build a reliable system from unreliable components.
Designing Data-Intensive Applications - Chapter 7 - Transactions

Designing Data-Intensive Applications - Chapter 7 - Transactions

Translations: RU
Earlier the book club of our company has studied excellent book: Martin Kleppmann - Designing Data-Intensive Applications This is the best book I have read about building complex scalable software systems. 💪 As usually I prepared an overview and mind-map. Chapter 7 is all your DEV team should know about Transactions: The purpose of transactions The concept of transaction: ACID, BASE, single-object and multi-object transactions Weak Isolation Levels: Read Committed, Snapshot Isolation and Repeatable Read.
Designing Data-Intensive Applications - Chapter 6 - Partitioning

Designing Data-Intensive Applications - Chapter 6 - Partitioning

Translations: RU
Earlier the book club of our company has studied excellent book: Martin Kleppmann - Designing Data-Intensive Applications This is the best book I have read about building complex scalable software systems. 💪 As usually I prepared an overview and mind-map. Chapter 6 contains everything the DEV team should consider when designing storage for big data: Partition aka Shard aka Region aka Tablet aka vNode aka vBucket. It is another approach for storing the data in addition to Replication (reviewed in the previous chapter) How to partition key-value data (primary index).
Designing Data-Intensive Applications - Chapter 5 - Replication

Designing Data-Intensive Applications - Chapter 5 - Replication

Translations: RU
Earlier the book club of our company has studied excellent book: Martin Kleppmann - Designing Data-Intensive Applications This is the best book I have read about building complex scalable software systems. 💪 As usually I prepared an overview and mind-map. Chapter 5: Intro. How to scale apps. Replicating and partitioning. Three algos of replicating Single-leader Replication Leaders and Followers Sync and async replication Adding new Followers Handling node outages Technical implementations and all potential problems Multi-Leader Replication Use-cases when it is good Handling write conflicts Three topologies and potential problems Leaderless Replication Writing to the database when a node is down Quorums and problems with them Detecting concurrent writes and how to resolve them Download full mind map (PDF)