← Community

Retry without making it worse

Add timeouts, backoff with jitter, and a retry budget so failures recover instead of amplifying into an outage. Use when calling an external service, adding retries, or diagnosing a cascading failure.

---
name: retry-without-making-it-worse
category: coding
description: >-
  Add timeouts, backoff with jitter, and a retry budget so failures recover
  instead of amplifying into an outage. Use when calling an external service,
  adding retries, or diagnosing a cascading failure.
---

# Retry Without Making It Worse

Naive retries turn a small failure into a large one. A service slows down, every
caller retries three times, the service now receives four times the traffic, and
it stops responding entirely. The retry logic caused the outage.

Retries are only safe with four things attached: a timeout, backoff, jitter, and
a budget.

## 1. A timeout on everything

A call with no timeout is not a call, it is a resource leak. It holds a
connection, a thread, and a request slot indefinitely, and under load you run out
of all three while the CPU sits idle.

Set two:

- **Connect timeout** — short. A second or two. Failing to establish a connection
  fails fast.
- **Read timeout** — derived from the dependency's p99, not from a round number.
  If it answers in 200ms at p99, a 30-second timeout means 30 seconds of holding
  resources for a call that was never coming back.

Every client library has a default. It is almost always far too long, and often
"none".

Shared publicly by the author and open for any use. See the Terms.