How we keep events in order across 72 hours of retries

When a customer's CRM goes down for an afternoon, Rivora keeps every event destined for it and retries for up to 72 hours. That part is easy. The hard part is doing it without reordering a single user's events, while every other user and every other destination keeps flowing at full speed.

The problem with one big queue

Our first design had one retry queue per destination. When a request failed, the event went to the back of the queue. That kept the destination's throughput high, but it meant a user's subscription.cancelled could be delivered before their subscription.created. For a CRM, that is the difference between a customer and a ghost.

Ordering only matters per user, so we stopped trying to order anything else.

Lanes, not queues

Today each destination has millions of tiny lanes, one per user. A lane is an append-only list of event IDs. Only the head of a lane can be in flight. If it fails, that lane waits and every other lane keeps going.

// One lane per destination and user.
// Lanes never reorder.
async function deliver(lane: Lane) {
  const head = await lane.peek();
  try {
    await destination.send(head);
    await lane.ack(head.id);
  } catch (err) {
    // this lane waits,
    // other users keep flowing
    lane.retryAt(
      backoff(head.attempts)
    );
  }
}

Lanes are cheap because they are only pointers into the event log. An idle lane costs a few bytes, and a lane disappears as soon as it is empty.

Back-off that adapts to the destination

A naive exponential back-off per lane would hammer a recovering destination with millions of simultaneous retries. Instead, lanes share a destination health signal. When the error rate rises, the retry budget for the whole destination shrinks, and it grows again as successful deliveries come back.

• Healthy destinations retry after 1, 2, 4 and 8 seconds.
• Degraded destinations get one probe per second until they recover.
• After 72 hours, the event moves to the dead-letter view where it can be rewound.

What changed

Since moving to lanes, out-of-order deliveries dropped to zero, the p99 time to drain a backlog fell by 61%, and destinations recover faster because they aren't flooded the moment they come back.