Honestly: this should not have happened. On Saturday, October 10, the reverse proxy on one of our edges crashed. The cause was a bug in Caddy itself, which its developers fixed shortly after. The real problem was not the crash. That one edge carried all the traffic, the proxy did not restart on its own, and nothing sent the traffic anywhere else. Until we stepped in, no measurements arrived.
For a monitoring tool, outages are particularly unpleasant.
Since Sunday we have asked one question for every station a measurement passes on its way from your shop into our database: what happens if exactly this fails right now?
How a measurement travels today
Your visitor's browser sends a measurement
on to the platform
on to processing
on to storage
The edge: an address that moves
The obvious fix would have been to switch DNS to another edge. That does not help everyone, though: some shops route our tracking through their own reverse proxy, which often looks the address up once at startup and keeps the result. A DNS change never reaches it.
So fastmon.site now points to a floating IP. That is an address we can move from one edge to another through an API, without changing anything in DNS. Our backend checks every edge every ten seconds with a health check. If the active one fails three times in a row while another one stays healthy throughout, the address moves.
Three smaller things add a lot on top: a crashed proxy restarts on its own after five seconds. All edges share their TLS certificates, so the replacement answers with a valid certificate right away. And before a planned update, an edge signs off first, so the address moves before the restart begins.
Redundant paths into the infrastructure
Behind the edge, every measurement used to take exactly one ingress into our infrastructure. Now the paths are redundant: traffic preferably goes through the ScaleCommerce ingress, and the ingress through smoxy stands by as the fallback.
The edge checks the preferred path every second. If it fails, the edge sends measurements over the fallback after about two seconds. A measurement that was already on its way and got no answer is sent there once more. If that makes it arrive twice, our backend folds the two back into one. A duplicate measurement costs us nothing. A lost one costs you a piece of your data.
So that a single slow request does not close a path right away, it takes several failures in quick succession. And even if no path is open for a moment, the edge waits briefly and tries again instead of giving up.
When a database wobbles
On October 13, the database holding accounts and settings switched over to its replica. For a moment, a few dozen measurements got an error back, because the collector could not reload an expired setting.
In that case it now keeps working with the last known state instead of rejecting. It then leaves the database alone for ten seconds before asking again, so nothing piles up at thousands of measurements per second. It also keeps a copy of the tracking script your shop loads in memory.
The measurement data itself lives in ClickHouse, replicated across several servers. Until now, though, every process only talked to one of them. Now every process switches to a healthy replica on its own as soon as its preferred one stops answering, and only returns once that one has caught up on everything it missed. There is no extra middleman in between that could fail in turn. We tested this with real replicas that we deliberately shut down one at a time and together.
What this means for you
If a single component fails on our side, ideally you notice nothing. Your measurements keep arriving, and your dashboard keeps showing what is happening in your shop.