Prevent killing the backend on restart

Hi,
I’m once again looking for best practices :pray: before enabling pushpin in production, I’d like to understand how to manage service restart.

We’re using pushpin for SSE with a django backend. In case of a pushpin upgrade (or any other reason for a rollout), all pushpin instances would be replaced and thus all clients would need to reconnect, putting a lot of pressure on the backend in a short period of time.

Do you have any advise?

The simplest way to regulate reconnects is to run multiple pushpin instances and stagger their restarts. Even if a single large instance would suffice, provision several small ones so that you have this ability to regulate. Also, it is best if pushpin and your app are separately deployable units. Typically, the need to upgrade or reconfigure pushpin should be infrequent, and so you can avoid mass reconnects by rarely restarting pushpin. For example, maybe you deploy your app daily, but you redeploy pushpin once every few months.

You can also try to regulate reconnects from the client side. If you control your client code, implement backoff delays with minor randomness in the reconnection logic. If you are really concerned, you could even implement a special control message, where just prior to restarting, the server could tell clients to delay their next reconnect after a longer period than normal.

3 Likes

it makes a lot of sense, thank you