Added ‘reliable’ exponential backoff to the OPC UA connector; every client hit t=60s in sync and the server face-planted while the HMI blamed ‘network’. Who else learned to sprinkle jitter and circuit breakers only after watching 43 robots reconnect like it’s a flash mob?
I’d switch to decorrelated ‘full jitter’ with a cap and add a tiny server-side gate (e.g., only 10 concurrent SessionCreates) so reconnects queue instead of stampeding like Black Friday. Seed the jitter with a stable client ID so devices don’t re-sync after reboot — does that fit your OPC UA stack?
That ‘t=60s’ sync bit me — derive jitter from MAC hash and throttle SessionCreate with a token bucket; sound sane?
One tweak that saved us: add a randomized pre-delay before the first reconnect (0–5s) and a longer penalty when you see “Bad_TcpServerTooBusy”, plus stagger SecureChannel renewals to fire between about 60–80% of lifetime. @cwilson45 your token bucket + MAC hash is solid; did you also splay Subscription Publish intervals so a blip doesn’t turn into a kindergarten fire drill?
We fixed a similar herd by giving each client a different max backoff cap (say 40–80s) and persisting that seed, so they never all land on “t=60s” again — stagger the ceiling, not just the steps. @r_dowell34 if your gate is in place, this keeps the queue steady instead of clumping; can you assign a per-node seed at provisioning, like scheduling coffee breaks instead of a fire drill?
Swapped to decorrelated jitter and the t=60s stampede vanished; it keeps retries from clustering near the cap even after a power-cycle sync. Seed it off the client cert thumbprint so it persists across reboots; explainer: Exponential Backoff And Jitter | AWS Architecture Blog. @cwilson45 did you try decorrelated vs full jitter?