
Five-minute meeting prep note
Before any technical meeting: the decision, the code it touches, one fact and one question. How to use it: How to prepare for a technical meeting in five minutes.
Decision: where does the retry logic live?
Code: src/queue/consumer.ts (one attempt, no retry)
src/payments/charge.ts
Fact: a failed job stays failed today.
Question: can a retry charge the customer twice?Project kickoff brief
One page, shared before the kickoff: goal, out of scope, who decides and the riskiest assumption. How to use it: Technical project kickoff: what to settle in the first meeting.
# Payment retries
Goal
Failed card payments are retried safely, so fewer orders fail.
Success: first-try failures that end in a paid order go from 0 to 80%.
Out of scope
Bank transfers. Retries of refunds. A new checkout screen.
Who decides
Scope: product manager. Design: tech lead. Launch date: both.
Riskiest assumption
That a retry can never charge twice. Test this first.
Dates
Design review 25 November. First release behind a flag 6 December.
Open questions
Does the payment provider limit retries per card?Meeting notes
Decisions, open questions, action items with owners, and the files the work lands in. How to use it: Meeting notes developers actually use.
Payment retries, 20 November
Decisions
- Idempotency is handled before retries are added.
- We meet again on Friday.
Open questions
- Does the refunds endpoint have the same double-processing risk? (Sam)
Action items
- Save a unique key per order before calling the payment provider,
and reuse it on every retry. Dana, by Friday.
Files: src/payments/charge.ts, src/queue/consumer.tsFollow-up email
Sent within the hour, to everyone who was invited. How to use it: The follow-up email after a technical meeting: a template.
Subject: Payment retries: decisions and next steps
Hi all, a summary of today's call. Reply if I got anything wrong.
Decided
- Retries live in the queue worker.
- Idempotency comes first: no retries until every charge has a key.
Actions
- Dana: save one key per order and send it with every charge. Friday.
- Sam: check the refunds endpoint for the same risk. Wednesday.
Open
- Does the provider limit retries per card? Sam will ask them.
Next: same time Friday, 30 minutes.Decision record
For decisions that are expensive to reverse, kept in the repository next to the code. How to use it: Write the decision down: a one-page decision record.
# 12. Retries live in the queue worker
Date: 20 November
Status: accepted
## Context
A failed payment job stays failed today. Support sees about two
percent of charges fail on the first try. We want retries, and a
retry must never charge a customer twice.
## Decision
The queue worker retries a failed charge. Before any retry is added,
every charge carries a unique key per order, so a repeated request
counts as one charge.
## Alternatives
- Retry inside the charge function: hides the retry from the queue
and makes timeouts harder to reason about.
- Let the customer retry by hand: no code, but support load stays.
## Consequences
- The worker needs a retry limit and backoff.
- The orders table gets a key column and a migration.Written status update
Instead of a status meeting: status first, decisions needed clearly marked. How to use it: Should this be a meeting? When a written update works better.
Payment retries, week of 17 November
Status: on track for the 6 December release behind a flag.
Done
- Idempotency key saved per order and sent with every charge.
Next
- Retries with backoff in the queue worker.
Needs a decision
- Should failed retries email the customer? (Product, by Friday)
Risks
- The provider has not confirmed its retry limit per card.1:1 running agenda
One shared document with your manager, newest meeting at the top. How to use it: 1:1s with your manager: an agenda for software engineers.
## 20 November
From me
- The retry work is blocked on access to the payment provider's test account.
- I would like feedback on how I ran Tuesday's design review.
- Thinking about on-call: can we talk about the rotation size?
From you
-
Agreed
- (filled in at the end)Blameless postmortem
After an incident: timeline, causes, what went well, and changes with owners and dates. How to use it: Blameless postmortem meetings: an agenda and a template.
# Postmortem: double charges on retried payments
Date of incident: 18 November
Impact: 41 customers charged twice over 3 hours. All refunded.
Severity: high
## Timeline
14:05 Retry change deployed to the payment worker
14:40 First support ticket about a double charge
16:10 Alert on charges per order fires
16:25 Retry turned off with a feature flag
17:05 Refunds started
## Causes
- A retry ran after a timeout even when the first charge had succeeded.
- Charges carried no idempotency key, so the provider saw two requests.
- No alert on more than one charge per order until 16:10.
## What went well
- The feature flag stopped the damage in 15 minutes.
## Changes
- Send one idempotency key per order with every charge. Dana, 25 November.
- Alert when an order has two successful charges. Sam, 22 November.
- Add a retry test that times out after the charge succeeds. Dana, 25 November.How to make templates stick
- Keep them where the work is: in the repository, the ticket tracker or the shared drive the team already opens.
- Fill in the parts that matter and delete the rest. A template is a checklist, not a form.
- Change them when they stop fitting. A template nobody uses is worse than none.