Hey friends,

It’s been a while. Thanks for sticking around 🙂

I’ve got six reads for you this week. Two are from August and marked as catch-up below.

Uber Engineering · 17 September 2026

Retrying once sounds harmless, until several services in the same chain all do it. Uber’s approach is to mark whether a service caused an error or is just passing it back from somewhere else. The caller nearest the failure can then retry, while callers further up the chain hold off. Uber estimates this prevented 9.5 million extra requests during an outage in 2025.

Cloudflare · 18 September 2026

Cloudflare keeps a table of hash values to choose which cache server gets a request. The team first shrank each entry from eight bytes to six. Then came the bigger saving: removing 90% of the hashes. The extra values barely improved how evenly traffic was spread, and different entries getting the same hash could actually make it worse. Cloudflare reports saving 100 TB of RAM across its network. The rollout needed care too: changing the table sends requests to different servers, where the files may not be cached yet.

Spotify Engineering · 16 September 2026

Spotify says it merged roughly 17,000 code changes in August, up from 8,100 a year earlier, as it expanded its use of AI. Some of its checks weren’t keeping up. One automated dependency update passed them and still failed in production. The team is now improving rollbacks and running automated updates while the people responsible are at work. It also found that checking each release alone could miss small problems piling up over time.

AWS Architecture Blog · 9 September 2026

Adding a queue doesn’t help if your app loses messages before it can send them. This AWS walkthrough cuts a test application off from SQS for increasing periods, from two to 15 minutes, with recovery time between them. You check whether unsent messages survive and whether the workers catch up afterwards. A dead-letter queue holds messages that repeatedly fail during processing. It can’t save a message that never reached SQS, so the app needs to preserve that work itself. This experiment uses permission errors; timeouts need a different test.

Catch-up: Datadog Engineering · 19 August 2026

Adding more Git replicas wasn’t fixing Datadog’s slow builds. Each extra copy also meant more work keeping everything in sync. With gitretriever, a small group of servers fetches from GitHub and a larger group serves the build jobs. The servers handling builds reuse prepared Git data, so adding them doesn’t add more requests to GitHub. GitHub still holds the original code, so the copies can be rebuilt. Along the way, Datadog added an API for internal tools that only needed a file or commit ID, letting them skip a full repository clone.

Catch-up: Netflix Technology Blog · 21 August 2026

Netflix built its own Flink autoscaler when there wasn’t a mature alternative. Its jobs have since become more complex, and the open-source scaler can adjust workers for each stage of a pipeline. The old tool could only resize the whole job. There’s a cost to making those adjustments, though: a large job can take minutes to save its progress, stop and restart. Netflix keeps more spare capacity than the tool’s default to avoid repeatedly shrinking and growing jobs. Both scalers are still running while the team migrates.

Which one did you find most useful? Hit reply and let me know.

See you next week,
Moein