WarpStream and AutoMQ as Kafka Replacements

Imagine you're ordering food at three different types of restaurant:

• REST is like a fancy restaurant with a set menu. The waiter hands you a menu (the URL), and you just pick from it. You don’t need to know how the kitchen works—just use the links provided. Everything is connected, like a web.

• gRPC is like calling the restaurant and ordering with a special code. You don’t care how they cook it—you just want your food. It’s fast and efficient, like a direct tunnel to the kitchen, handling all the tricky details for you.

• OpenAPI is like a build-your-own-burger place. They give you a template, and you choose what goes where. It’s flexible because you can customise the recipe (HTTP mapping) to suit your needs.

In short:

• REST – Simple, follows web standards.

• gRPC – Fast, efficient, hides complexity.

• OpenAPI – Flexible, customisable, but more involved.

The article by Google Cloud explores when to use each one for your API design. It's worth reading to see which "restaurant" best suits your needs!

I’m sure many people come across payment systems at some point in their career. And when things go wrong, it’s 😡 😡.

A good payment system must be reliable, which means it should handle failures properly. Here’s how:

• Payment Status Tracking – Every transaction has a status like "Initiated", "Processing", "Succeeded", "Failed", or "Pending Retry". These statuses are stored in a special database that only adds new records instead of changing old ones. This keeps a full history, making it easier to track issues, maintain accuracy, and handle a large number of payments.

• Retry Queue – Some payments fail due to temporary problems, such as network errors. Instead of failing completely, the system adds them to a queue and retries after a short delay.

• Dead Letter Queue (DLQ) – If a payment keeps failing even after multiple retries, it goes here. This helps identify serious issues and decide whether a manual fix, a customer notification, or a system update is needed.

If you want to build a reliable and fault-tolerant payment system, this article will show you how to implement effective payment retries.

Optimize global IT operations with our World at Work Guide

Explore this ready-to-go guide to support your IT operations in 130+ countries. Discover how:

• Standardizing global IT operations enhances efficiency and reduces overhead

• Ensuring compliance with local IT legislation to safeguard your operations

• Integrating Deel IT with EOR, global payroll, and contractor management optimizes your tech stack

Leverage Deel IT to manage your global operations with ease.

Download free guide

If you have worked with large Kafka clusters, you know they can become unbalanced because compute and storage are closely linked. This means rebalancing requires physically moving data between brokers.

Cruise Control can help by acting as an assistant for Kafka, finding the best way to move data and improve efficiency. It provides features like resource tracking, anomaly detection, and automatic cluster rebalancing. However, it still needs to move data to balance the system.

AutoMQ takes a cloud-native approach by separating compute and storage. With storage in the cloud (object storage), rebalancing only requires updating metadata, with no data movement needed. AutoMQ uses AutoBalancer for self-balancing, streamlining the process and improving efficiency. AutoBalancer includes components like the Metrics Collector, State Manager, and Decision Scheduler to keep everything running smoothly.

If you’re into the world of events, AutoMQ might be something you want to have a look into.

WarpStream presentation at the 2024 Apache Kafka summit

Now that we’re talking about Kafka replacements, I think it’s worth having a look at WarpStream, which takes a similar approach by separating compute and storage (S3).