
When a retry helps
Some failures go away on their own: a dropped connection, a timeout, a service that was restarting. Trying again a moment later often works. Other failures never go away, and retrying them only wastes time.
| Failure | Retry? |
|---|---|
| A timeout or a dropped connection | Yes |
| The service says it is overloaded or temporarily unavailable | Yes, after a wait |
| An invalid request or a missing permission | No. It will fail the same way |
| Not found | Usually not |
Why you wait longer each time
If a thousand clients all retry immediately, a service that was already struggling gets a thousand extra requests at its worst moment. Waiting one second, then two, then four, spreads the load out. Doubling the wait like this is called exponential backoff.
Why you add randomness
If every client waits exactly one second, they all come back together one second later. Adding a small random amount to each wait, called jitter, spreads them apart.
attempt 1 fails
wait about 1 second
attempt 2 fails
wait about 2 seconds
attempt 3 fails
wait about 4 seconds
attempt 4 succeeds, or give up and report the failureThe four settings every retry needs
- How many attempts. There must be a limit. Retrying forever turns one failure into a permanent one.
- How long to wait, and how the wait grows.
- Which failures to retry. Only the temporary ones.
- What happens when it gives up. Someone or something has to find out: a list of failed jobs, an alert, a message to the user.
The trap: repeating something that is not safe to repeat
A retry repeats the operation. If the operation charges a card or sends an email, a retry after a lost reply does it twice. That is why retries and idempotency are designed together, and why the idempotency comes first.
Questions you will hear in a meeting
- Where does the retry live: in the caller, in the queue worker, or in both?
- If two layers each retry three times, do we really make nine attempts?
- What does the user see while we are retrying?
- Where do the jobs go that failed every attempt?