We are seeing a weird behaviour while load testing pushpin.
Observation :
Lets say 1000 connections are upgraded by the pushpin instance.
It leads to increase in memory and CPU usage (which is expected).
But after a while if the connections are abruptly closed 3 things happen :
Is there anything we can do to avoid it ?
Hi, thanks for moving the post here.
It’s not unusual for memory usage to go up and stay up. However, there are no known memory leaks in Pushpin and in our experience the kept memory will get reused. For example, if you make lots of connections, close them, and then make lots of connections again, the memory usage should not change much between rounds.
The connection_connected metric might take around 10 seconds to drop. If it still says 1000 connections for much longer than that, then something else is going on. Maybe the OS still thinks the connections are alive (check with netstat or something).
Okay, We will check the netstat as well.
Another observation we have is, across all runs , the prometheus metrics of connection_connected does not update further than ~4500 figure (Even though much more connections are upgraded and receive data).
Meaning it did not reflect any higher value than ~4500 conn.
Also It got reset only after 1 hour.
Update: We got the linear increase in metrics back after removing the service-mesh. 
@jkarneges We are observing Pushpin consumes close to 100 KB of memory per WebSocket connection. Is that the expected behaviour ?
Major Configuration that might be helpful:
512 KB for client_buffer_size is huge. Each connection will have two of these buffers, one for receiving and one for sending, which would take 1 MB total. Try using much smaller buffers.