Grab vs Go: Two Go Services, One Retry Budget
Two Go services, one shared payment ledger API, and two retry policies that never met. Grab retried aggressively on 503s. Go tripped its breaker late. On a peak-traffic Tuesday the two behaviours compounded into a retry storm that roughly quadrupled load on the ledger and pushed p95 latency past two seconds. This is what the postmortem found and what the team changed.
The Retry Budget Collision
Grab and Go were internal codenames for two Go services that both called the same payment ledger API. Each service had its own retry configuration, tuned by the team that owned it. Neither team knew the other's numbers. The retry budget existed in two places and belonged to no one.
Both teams had shipped exponential backoff, which is the standard way to reduce load when a dependency is struggling. Exponential backoff multiplicatively decreases the rate of a process to find an acceptable level, as the pattern is usually described. The problem was that the two implementations disagreed on the basics.
When the ledger started returning 503s during a traffic peak, Grab retried immediately and often. Go's circuit breaker, which is designed to prevent cascading failures by detecting unresponsive services and stopping repeated attempts, tripped later than Grab's retries ramped up. For a window of several minutes, both services pushed hard against a dependency that was already struggling.
What made this collision hard to see in advance was that each retry policy was reasonable on its own. A five-retry budget with a short base delay is common for read-heavy paths. A breaker that opens after a sustained failure rate is standard for write paths. The failure mode only appeared when the two ran side by side against the same dependency at the same time.
How Grab and Go Diverged
Grab treated 503s as transient and retried up to five times with a short base delay. Go treated 503s as a signal to open its circuit after a threshold of failures, but the threshold was set roughly three times higher than Grab's effective retry count. The two policies were individually reasonable and jointly wrong.
Neither service had jitter in its backoff. Without jitter, retries from many callers line up on the same schedule and hit the dependency in synchronized waves. That synchrony is what turns a slow dependency into an unavailable one. The pattern is well known, and the fix is cheap, which made its absence harder to explain later.
There was no cross-team owner for retry policy. Each service team owned its own configuration, and the platform team owned the ledger API but not the callers. A related piece on this site about registry quotas and maintainer shipping makes a similar point about shared resources with no shared governance.
One more divergence was the failure window. Grab counted retries over the lifetime of a request, while Go counted failures over a rolling window measured in seconds. Those two clocks rarely agree. During the incident, Grab's retry count was already high while Go's rolling window still showed a healthy failure rate, which delayed the breaker trip by a meaningful margin.
The Incident Window
The incident began during a Tuesday morning peak, when normal traffic already sat near the ledger's comfortable ceiling. Grab's retries added load first. Go's breaker had not yet tripped because its failure threshold counted a different window than Grab's. By the time Go opened its circuit, the damage was done.
Retry amplification is multiplicative, not additive. If each caller retries five times and a fraction of those retries also fail and retry, the effective request count against the dependency can rise several-fold. In this case the postmortem estimated roughly a fourfold amplification over baseline, which matched the latency curve.
Ledger latency crossed two seconds, then stayed there. The on-call engineer was paged at 09:14. The first action was to disable retries on Grab, which dropped load immediately. Go's breaker had already opened by then, so the recovery was faster than the detection.
During the window, the ledger's own retry queue filled. That queue had a fixed depth, and once it was full the ledger began rejecting new writes outright. This is the point where a slow dependency becomes a broken one, and it is usually invisible to the callers because they only see the rejections, not the queue behind them.
What the Postmortem Found
The central finding was that retry budgets were per-service, not per-call-chain. A call chain that passed through three services could multiply retries at every hop, and no single service saw the total. The budget that mattered was the one the dependency experienced, and nobody owned that number.
Circuit breaker thresholds differed by roughly three times between the two services. Grab's retry count and Go's breaker threshold were both defensible in isolation. Together they meant one service kept retrying while the other kept waiting, and the dependency absorbed both behaviours at once.
Neither backoff implementation had jitter. The runbook had no retry-storm playbook, so the on-call engineer had to reason from first principles during the incident. That is a fixable gap, and it is usually cheaper to fix than the code.
There was also no shared metric for retry counts. Grab and Go each exposed a retry counter, but neither was on the same dashboard as the ledger's latency and error rate. The on-call engineer had to correlate them by hand, which cost several minutes at the start of the incident.
Fixing Retry Semantics in Go
The first change was a central retry budget carried through the context. Go's context package already propagates deadlines and cancellation across call boundaries, so the team attached a retry budget to the context and decremented it at each hop. When the budget hit zero, no further retries were allowed, regardless of which service was calling.
The second change was a shared circuit breaker library with jitter built in. Instead of each team picking its own threshold, the library exposed a small set of named profiles and required callers to choose one. The profiles were reviewed by the platform team, which also owned the ledger API.
The third change was per-call-chain retry accounting. A request that entered the system got a single budget, and every retry at every hop drew from it. This is the change that would have prevented the incident, and it is also the one that required the most coordination between teams.
The fourth change was load tests that simulated paired services. The old tests exercised each service against a healthy ledger. The new tests ran both services against a ledger with injected latency and failure, which reproduced the amplification within a staging environment.
One trade-off worth naming: a shared retry budget can make a single service less resilient in isolation. If one caller exhausts the budget, another caller may be denied a retry it would otherwise have made. The team accepted this because the dependency's health mattered more than any single caller's retry count, but it is a real cost and should be documented in the library's README.
Practical Takeaways for 2026
Audit retry budgets across service pairs, not individual services. Pick any two services that share a dependency and compare their retry counts, backoff bases, and breaker thresholds side by side. The mismatch is usually obvious once you look.
Add jitter to every backoff implementation. A random spread of even a few hundred milliseconds breaks the synchronized waves that turn a slow dependency into an unavailable one. This is the cheapest change on the list and the one most often skipped.
Instrument retry counts as first-class metrics. If retries are not on a dashboard next to latency and error rate, they are invisible until they cause an incident. Label them by caller, callee, and call chain.
Own retry policy at the platform level. Service teams can choose profiles, but the profiles themselves should be defined once and reviewed by whoever owns the shared dependency. A related piece on this site about write amplification costs shows how shared-dependency costs accumulate when nobody owns the aggregate.
Write a retry-storm playbook before you need it. The on-call engineer in this incident had to invent the response in real time. A one-page runbook that says which retries to disable first would have cut the incident window substantially.
Finally, treat retry policy as a cross-team contract. The teams that share a dependency should agree on a single budget and a single breaker profile, even if they never share code. The agreement can live in a short document or a shared config repo, but it has to exist somewhere that both teams can see it.
