Hey everyone,

If a queue is falling behind, adding more workers seems like the obvious fix. At incident.io, it made things worse. The first article explains why.

There are also reads on testing a database upgrade with real queries, how one outage spread to other customers, and changing parts of a system without rebuilding everything around them.

Rory Malcolm · incident.io · 6 October 2026

At incident.io, workers spent more and more time checking which jobs other workers had locked. Adding workers made that competition worse. The team changed the process: claim jobs in a short transaction, then do the work separately. Each application instance also got one dispatcher to fetch jobs and hand them to workers, instead of every worker querying Postgres. These changes, alongside other improvements, increased peak capacity 3.5 times in load tests. The new approach has a cost: if a process dies after claiming a job, another worker may have to wait up to ten seconds to take it.

Zuofei Wang and Erluo Li · Airbnb Engineering · 6 October 2026

Will your application behave the same after a database upgrade? Airbnb checks by recording real queries and running them against both the old and new database versions, starting from the same copy of the data. The replay keeps each transaction’s queries in order, so a query that creates a row still comes before one that reads it. This caught a subtle problem: queries using LIMIT without a dependable ORDER BY could return different rows after the upgrade. The article explains how Airbnb captures the traffic and makes those comparisons without changing each application.

Inngest engineering · 15 October 2026

Inngest ran across three data centers, but a job could need information from all three to finish. When one site lost its network connection, those jobs stopped making progress. The impact spread further because failed jobs built up in a shared processing pipeline, delaying customers who didn’t use the failed site at all. The report covers this outage and a separate database-connection failure the following day. Inngest is now moving the services into one data center to remove those cross-site dependencies; that work is still underway.

David Laban and Lambros Petrou · Cloudflare · 13 October 2026

Before Cloudflare’s D1 service can run a query, it needs to find the right customer database. Looking that up in a central Postgres database every time would create a bottleneck, so Cloudflare also keeps the lookup information in its distributed key-value store, Workers KV. The harder part is keeping those copies correct. For example, an old lookup can still point to a deleted database, so the database itself remembers that it’s deleted and rejects the request. The post walks through this and other ways the system recovers when an update reaches one place but fails to reach another.

Netflix database engineering · 14 October 2026

Netflix needs to keep old event data available, even though most queries ask for recent events. Keeping all of it on fast disks is expensive. Its newer approach uses the Cassandra backups it already stores in S3: keep small indexes locally, then fetch the requested data from the backup. This only works after that part of the data has stopped changing and Netflix has checked that the backup is complete. The team modified Cassandra to support it, and explains why this replaced a separate Kafka and Flink pipeline that was costly to run and maintain.

Meta video engineering · 12 October 2026

Meta is changing how video travels from an Instagram Live broadcaster’s phone to its servers. It uses Media over QUIC, a protocol for sending audio and video, but the rest of Instagram’s video system still expects the old format. The team put a translation step where the video first arrives, converting it before passing it on. That lets them roll out the new protocol without replacing the whole system at once. Streaming quality has matched the previous setup so far; sending video all the way to viewers using the new protocol is still future work.

See you next week,
Moein