Every popular software should have a Rate Limiter. It prevents DDOS attack, reduces cost and prevents servers from being overloaded.
There are some tricky questions to be considered during implementation of Rate Limiter:
Where to put Rate Limiter: client-side, server-side, gateway? Algorithms for rate limiting. There are many algorithms with pros and cons: Token bucket, Leaking bucket, Fixed window counter, Sliding window log, Sliding window counter. Your business needs will define the right algorithm.
Four standard steps for system design interview. However, I would think about them wider: as about four initial steps to design the software.
Step 1. Understand the problem and establish design scope Step 2. Propose high-level design and get buy-in Step 3. Design deep dive Step 4. Wrap up The chapter 3 of the book discovers details about each step, good questions to ask (to think about), DO’s and DONT’s.
A very short Chapter 2 is about how to make rough estimates to start from the most important parts when designing the software.
Some concepts that EVERY software developer should know:
“Power of two” Standard latency numbers! How fast is memory, how slow is disk, etc… Availability numbers What are key metrics you should think about These concepts and numbers are disclosed in chapter two of the book:
A great generic plan for scaling any app from zero to millions of users.
Single server setup Selection and usage of database Vertical scaling vs horizontal scaling approaches. And why you should prefer horizontal Adding load balancer for horizontal scaling Adding database replication for horizontal scaling Adding cache Adding CDN Stateless vs Stateful architecture and using external state storage Adding extra Data Centers Adding Message queue Adding Logging, Metrics, and Automation Scaling database (sharding) and futher steps… All of these is carefully but briefly disclosed in the Chapter 1 of the book:
Principles of Microservice Deployment Differentes between Continuous integration (CI), Continuous delivery and Continuous deployment. Deployment patterns:
The Basic Full-Stop Deployment Pattern The Rolling Update Pattern The Breaking Schema Change Pattern: two options here - Eventual Migration and Synchronized Migration The Blue-Green Deployment Pattern All of these is disclosed in the Chapter 16 of the book:
“Building Event-Driven Microservices: Leveraging Organizational Data at Scale” by Adam Bellemare
Sharing my mind map with all the details as usual:
Testing event-driven solutions is a challenging task. It includes:
Common principles Unit testing: stateless and stateful Testing the Topology (I bet you didn’t do it! ;) Testing Schema Evolution and Compatibility Integration testing: Local, Remote and Hybrid - a set of common sense considerations not to forgot about Choosing the testing strategy All of these is disclosed in the Chapter 15 of the book:
“Building Event-Driven Microservices: Leveraging Organizational Data at Scale” by Adam Bellemare
Quite often the tooling is a forgotten area during the beginning of development. This chapter overlooks the tooling that can help build and maintain event-driven microservices. Useful checklist! This topic includes:
Org rules Documentation, tagging Quotas Schema registry Offset management ACL State management and app reset Consumer offset lag monitoring Container Management Controls Cluster Creation and Management Dependency Tracking All of these is disclosed in the Chapter 14 of the book:
How to integrate event-driven microservices with request-response APIs?
There are two types of external events:
Autonomously Generated Events (analytical events) Reactively Generated Events (events from request-reply) There are two approaches of processing and serving requests using stateful services:
using internal state stores (with silly or with smart routing) using external state stores (with regular or with composite microservice) Ways of handling requests within an event-driven workflow:
The 4th pattern to build microservices is to use Lightweight Frameworks.
Lightweight Frameworks provide similar functionality to Heavyweight Frameworks, but they heavily rely on:
the event broker the container management system (CMS) In many cases they exceed Heavyweight Frameworks.
Different apps can use any/different resources from the cluster which are better fit their needs. While still provide Scaling and Recovering from Failures (again by heavily relying on event broker and CMS).
Heavyweight Stream Processing Frameworks are another foundation/pattern to build your microservices.
These frameworks are highly scalable and allow you to efficiently solve many analytical tasks. But they are not always good for stateful event-driven microservice application patterns.
Heavyweight frameworks operate using centralized resource clusters, which may require additional operational overhead, monitoring, and coordination to integrate successfully into a microservice framework. However, recent innovations move these frameworks toward container management solutions (CMS) such as Kubernetes that should reduce your efforts.
Basic Producer and Consumer (BPC) is a simple pattern for creating microservices. It forms the foundation of stateless and stateful services.
You can use it for:
interfacing layer between event streams and legacy systems leverage external stream processing systems to augment capabilities. But it requires your extra effort for:
simple state materialization event scheduling timestamp-based decision making All of these is disclosed in the Chapter 10 of the book we are currently studying:
Function-as-a-Service (FaaS) is an area of cloud computing that is growing rapidly.
There are following issues to conside here:
Principles of FaaS Open source and 3rd party FaaS providers 4 components to consider when building microservices as functions Cold Start vs Warm Start Different Triggers that can start FaaS: Triggering Based on New Events, Triggering Based on Consumer Group Lag, Triggering on a Schedule, Triggering Using Webhooks, Triggering on Resource Events Maintaining State Two patterns of functions calling other functions: Event-driven and Direct-call Termination and Shutdown Tuning and Scaling All of these is disclosed in the Chapter 9 of the book we are currently studying:
There are two patterns of building event-driven microservices:
Choreography (without centralized coordination) Orchestration (with centralized coordination) Both have pros and cons (however what I’ve learnt is that Orchestration is a better pattern)
How to implement Distributed Transactions with Choreography and with Orchestration.
How to AVOID implementing Distributed Transactions - a Compensation Workflows approach.
All of these is disclosed in the Chapter 8 of the book we are currently studying:
Event-Driven Microservices need to materialize states. And it comes with the important things to think about. Two approaches to choose from:
Internal state store or External state store Both have pros and cons. Both have important scalability and recovery considerations.
Two approaches for changing data structures:
Bebuilding and Migration Transactions and how to emulate them in order to implement Effectively Once processing.
All of these is disclosed in the Chapter 7 of the book we are currently studying:
Deterministic Stream Processing is a foundation for building scalable event-driven systems. The lack of determinism can be very painful for the business (imagine lost financial transactions, missed alerts, wrong data aggregation).
There are certain tips and tricks on how to implement determinism. Keywords are:
Timestamps Event Scheduling Watermarks Stream Time Also you should understand the nature of Out-of-Order and Late-Arriving Events. And strategies for handling them.
Also need to support reprocessing.
How to talk to your users when you are building a Startup? What is the best book to read on this topic? Three common mistakes everybody makes. Five great questions you can ask in every user interview. How to talk to users during three stages: idea stage, prototype stage and launched stage.
All of this is disclosed in Y Combinator’s Startup School - Lesson “How to Talk to Users”.
As usually, here is my summary mind map:
Overviewing basics of event processing in Event-Driven Architectures:
Typical structure of microservice Typical types of event transformations, 2 branching scenarios, merging streams Repartitioning events and when it can be useful Copartitioning events and when it is needed Assigning Partitions to a Consumer Instance. Three strategies to do this. Recovering from stateless processing instance failures. These topics are disclosed in the Chapter 5 of the book we are currently studying:
Data Liberation is the process of moving from monolith towards microservices by decoupling systems in terms of data dependencies.
There are three patterns for Data Liberation:
Query-based Log-based Table-base Each pattern has its own pros and cons, as well as other important considerations.
Data definition changes (data structure migrations) must also be supported by the chosen Data Liberation approach.
There are different Liberation frameworks/tools that simplify the process of Data Liberation.
The Event-Driven model heavily relies on the QUALITY of events.
Good quality events are:
explicitly defined via contracts have comments support evolution with backward and forward compatibility support code generation breaking changes are well thought Good events are implemented using the right tools:
use Avro/Thrift/Protobuf formats and never use JSON! use the right event broker (such as Pulsar) Good events are designed to:
contain all the info needed by consumers use separate streams for each event type use the right data types for their fields (don’t use string for numbers, use enums, etc.
I accidentally met with the founders of EMPERA blockchain. As you may know, despite the fact that I believe in the blockchain future, I am a bit skeptical about new projects and ideas in this field as there are many pure scams and unprofessional enthusiasts.
However, after reading all the papers and after a deep talk with the founders I am now very excited about this particular project. And I want to share with you my insights (as well as a few skeptical unclarities to be 100% honest).
Introduction into Event-driven microservice (EDM) architecture consists of the following topics:
two topologies content of events three types of events table-stream duality schemas for defining event data single writer principle features of event broker event broker vs message broker single source of truth principle scaling using containers and VMs microservice taxes that we should pay These topics are disclosed in the Chapter 2 of the book we are currently studying:
Our company’s books club has selected a book about Microservices. It is “Building Event-Driven Microservices: Leveraging Organizational Data at Scale” by Adam Bellemare
Chapter 1 contains intro info:
Types of architectures and differences between them:
Traditional monolith-style architectures Service-oriented architectures (SOAs) Event-driven microservice (EDM) architectures Levels of Communication Structures and related Conway’s Law:
business implementation data Problems with traditional architectures (monolith & SOA) when you need to:
Did you know that losing $1000 and then winning $900 (i.e. getting a result of -$100 in total) is more pleasant than winning $1000 and then losing $900 (i.e. getting a result of +$100 in total)!??
What has happened in the RECENT PAST drives our emotional response much more than how we are doing overall! It is a problem if you are making important prediction.
This is a continuation of my previous post Real Projects with Ethereum) - about how to setup a development infrastructure around real-life smart contract.
I’ve already told about the truthseeking group (The Buddy System) as a great way to improve decision-making process.
How to share information inside such a group and how to disagree efficiently (to find a good solution)? And how to communicate outside of such efficient group?
There is a set of rules called “CUDOS” that can be used for this…
People are not ready to consider themselves the source of the problem they are facing (their skill), they blame other people/circumstances (luck). How to overcome this issue?
One of the solution is to create “truth-seeking group” with your colleagues/friends/partners/“buddies”. And such a group should follow specific rules that would increase rationality…
Y Combinator’s motto is “Make Something People Want”. But how to find what people want?
There are seven steps / considerations to gradually get closer to the answer…
How to build a complex blockchain smart contract? What standard issues will you face with arrays, gas, different types of variables, deployment of contracts, etc? And how to solve them?
What is the right architecture for the app that should work with blockchain (and how it differs from traditional web architecture)? What libraries to use? How to configure environment?
Useful advanced info and techniques of writing and testing Smart Contracts: basic data types, reference data types, global variables, gotchas with dynamic arrays, validations and modifiers, debugging, generating random numbers, sending ether.
What qualities are important to become a great communicator? Is it Charisma or Sincerity? Or something else? How to identify your strengths and weaknesses? And how to use them?
What is the right toolset for Blockchain development? Here are some keywords: nodejs, npm, web3, solc, ganache, mocha, metamask, rinkeby, infura, remix…
How people should learn by watching outcomes and why they do not learn?
There are many unknown circumstances - so we do not know the real cause of the outcome. Luck also plays a big part here, so we don’t know if it’s luck or skill.
Poker players make decisions by making bets. Why do we all need to learn to make decisions using bets too?
There is a trap of “Motivated reasoning” bias. But, fortunately, there are facilitating questions that direct our minds in the right direction.
We must also move away from using the all-or-nothing confidence scale. Instead, we must learn to measure our confidence in percentages. This will not only help us win more, but it will also make us more credible communicators and make teamwork around us more efficient.
Why ask questions during a conversation? What mistakes do we make in communication?
Two types of questions.
Three characteristics of a good listener.
The structure of a good question. The structure of the ideal question.
It is impossible to live without conflicts. But how to make the conflict constructive?
Thomas Gordon, author of “Parent Effectiveness Training”, invented the “I-message” that is now taught in many business education programs.
Lesson 8 of Communication Genius describes I-message in detail:
This book is written by a multi-million dollar poker tournaments winner (and her brother is also a multi-million dollar poker tournaments winner - so it’s a family business ;))
She teaches how to apply poker thinking to make decisions in financial markets, strategic planning, human resources, law, entrepreneurship…
There is quite a high demand for blockchain development these days. We do some experience with it as a company (one complex multi-year project), but personally I only had high-level knowledge and I wanted to improve it…
Which is better - expert judgment or mechanical prediction using very simple models/rules? Surprisingly, the model of you beats you! Even a simple regression model based on your past decisions!
How to make rules/models even less noisy? How can rules/models become more biased sometimes?
Why don’t people always use models (if they are so good) and still rely on their bad judgment?
I continue sharing my impression and knowledge from the Communication Genius course which I started in January 2022. Lesson 3 is full of useful information concerning SAYING TOASTS. There are tips what to say to be remembered and some phrases which sound good during eating and drinking in informal atmosphere.
In January 2022, I started the Communication Genius course from one of the best public speaking coaches. Lesson 2 is full of simple but efficient instructions how to communicate quickly in a stressful situation (ELEVATOR PITCH). There are tips what to say and there are also some things which are better not to use in a quick conversation.
The book club of our company has chosen a new wonderful book for reading:
Robert Martin - Clean Architecture - a Craftsman’s Guide to Software Structure and Design
👍
The part VI undermines some foundations 😀:
Do you know that Database is a “detail”? An unimportant minor low-level non-essential feature that can be neglected in architecture design! Do you know the same about the Web? It is just an unimportant IO device that should also be neglected in architecture design!
The book club of our company has chosen a new wonderful book for reading:
Robert Martin - Clean Architecture - a Craftsman’s Guide to Software Structure and Design
👍
Fifth part of the book contains A LOT of useful information:
What is software architecture? What type of interdependencies can exist? How to draw borderlines between components? What type of borders exist? How to distribute policies by the levels? What are business rules, entities and use cases?
Interesting to learn how your mind makes judgments. Two types of judgments: predictive and evaluative. How to measure errors from bias and noise. How to measure the cost of noise. And how to deal with catastrophic consequences.
Drilling down the noise to three components: system noise, level noise, and pattern noise. How to measure (and compensate) the noises. Occasional noise as a part of pattern noise. How to compensate occasional noise: wisdom-of-crowds effect, crowd-within-the-one effect, second answer effect, and dialectical bootstrapping tool.
How the group work influences the noise. Informational cascade and social pressure cascade. How to deal with them. The effect of group polarization.
Do you realize how seriously your organization is affected by bad decisions? There are two types of errors in human judgments: Bias and Noise. Bias is a very widely recognized problem, and you definitely heard about it. Noise is not so widely recognized. Yet, it affects MANY decisions in ALL the industries.
There is a huge Noise in government decisions. There is a huge Noise in commercial companies.
Noise can be made visible and can be reduced. It is harder to measure the noise in unique singular decisions, but it is still there and it still can be reduced.
Earlier the book club of our company has studied excellent book:
Martin Kleppmann - Designing Data-Intensive Applications
This is the best book I have read about building complex scalable software systems. 💪
As usually I prepared an overview and mind-map.
Chapter 12 is a summary of the book and a visionary view of the future.
Data Integration. Overview of the ways we have to integrate data. Causality and why we need Total Order and Idempotency.
Earlier the book club of our company has studied excellent book:
Martin Kleppmann - Designing Data-Intensive Applications
This is the best book I have read about building complex scalable software systems. 💪
As usually I prepared an overview and mind-map.
Chapter 11 discovers all aspects about Stream Processing. If your system needs to process some data on-the-fly then your DEV team should learn this info.
Approaches for transmitting events: Direct messaging, Messaging Systems and Partitioned Logs.
Earlier the book club of our company has studied excellent book:
Martin Kleppmann - Designing Data-Intensive Applications
This is the best book I have read about building complex scalable software systems. 💪
As usually I prepared an overview and mind-map.
Chapter 10 discovers all aspects about big data Batch Processing. If your system needs to process some data then your DEV team should learn this info.
Unix tools for batch processing and brilliant concept of pipes.
Earlier the book club of our company has studied excellent book:
Martin Kleppmann - Designing Data-Intensive Applications
This is the best book I have read about building complex scalable software systems. 💪
As usually I prepared an overview and mind-map.
Chapter 9 tells about Consistency and Consensus in distributed systems. It covers the following topics:
What is consistency and eventual consistency Linearizability. Why it is needed. Difference from Serializability.
Earlier the book club of our company has studied excellent book:
Martin Kleppmann - Designing Data-Intensive Applications
This is the best book I have read about building complex scalable software systems. 💪
As usually I prepared an overview and mind-map.
Chapter 8 discovers non-database related problems of distributed systems. DEV teams should consider them when designing distributed software.
Faults and Partial Failures. The need to build a reliable system from unreliable components.
Earlier the book club of our company has studied excellent book:
Martin Kleppmann - Designing Data-Intensive Applications
This is the best book I have read about building complex scalable software systems. 💪
As usually I prepared an overview and mind-map.
Chapter 7 is all your DEV team should know about Transactions:
The purpose of transactions The concept of transaction: ACID, BASE, single-object and multi-object transactions Weak Isolation Levels: Read Committed, Snapshot Isolation and Repeatable Read.
Earlier the book club of our company has studied excellent book:
Martin Kleppmann - Designing Data-Intensive Applications
This is the best book I have read about building complex scalable software systems. 💪
As usually I prepared an overview and mind-map.
Chapter 6 contains everything the DEV team should consider when designing storage for big data:
Partition aka Shard aka Region aka Tablet aka vNode aka vBucket. It is another approach for storing the data in addition to Replication (reviewed in the previous chapter) How to partition key-value data (primary index).
Earlier the book club of our company has studied excellent book:
Martin Kleppmann - Designing Data-Intensive Applications
This is the best book I have read about building complex scalable software systems. 💪
As usually I prepared an overview and mind-map.
Chapter 5:
Intro. How to scale apps. Replicating and partitioning. Three algos of replicating Single-leader Replication Leaders and Followers Sync and async replication Adding new Followers Handling node outages Technical implementations and all potential problems Multi-Leader Replication Use-cases when it is good Handling write conflicts Three topologies and potential problems Leaderless Replication Writing to the database when a node is down Quorums and problems with them Detecting concurrent writes and how to resolve them Download full mind map (PDF)
