System design

Real-time systems need a recovery story

Connections drop. Events repeat. Design the path back to a consistent state.

A connection is not the source of truth

A WebSocket can deliver changes quickly, but the connection itself should not be the only record of what happened. A client may disconnect while an important update is sent. Keep durable business state in the backend and provide a way to read it again. Live messages then improve responsiveness without becoming the sole means of reconstructing the user’s view.

Expect repetition and reordering

Retries can cause the same event to be processed more than once. Multiple producers can also introduce ordering surprises. Give events stable identifiers and make consumers tolerate duplicates. When order matters for one entity, use an explicit version or sequence and define what should happen when a consumer sees an older update after a newer one.

Reconnect deliberately

After a disconnect, retry with backoff and jitter so clients do not all reconnect at once. Once connected, reconcile the visible state using a snapshot or a supported replay mechanism. Be explicit about what the user sees while the state is uncertain. A reconnecting indicator is more honest than presenting an old queue or presence status as current.

Test recovery as a feature

Disconnect a client, restart a consumer, delay an event, and deliver a duplicate. Check whether the system returns to a consistent state without creating duplicate business actions. Measure recovery time as well as steady-state latency. A real-time experience is only dependable when the recovery path receives the same attention as the live path.

Back to all articles