Skip to content
Back to blog

Performance

Every deploy can wreck your performance. Would you notice?

Regressions almost always ride in on a deploy. Why the average hides them, how a release marker from your CI/CD makes them visible, and what you actually compare at the p75.

Kamil Adrian Czujowski

Kamil Adrian Czujowski

4 min read

Honestly: almost nobody deploys and thinks about their LCP. You merge the branch, the pipeline goes green, the feature is live. Done. That this exact deploy pushed your load time from 2.1 to 3.4 seconds shows up in no error log. Nothing is “broken”. It just got slower.

And that is the tricky thing about performance regressions: they throw no error. They slip in with a perfectly normal, successful deploy and stay until someone notices by accident. Usually weeks later, usually a customer.

Why regressions almost always ride in on a deploy

Your site does not get slower on its own overnight. It gets slower because something changed: a new library in the bundle, an image without dimensions, a third-party script, a font that now blocks, a query that pulls one more column. All of that arrives with a deploy.

So the most honest question for any regression is not “what is slow?” but “since when?”. And “since when” is almost always “since which deploy”. Know the moment and you have your suspect. Miss it and you are searching in fog.

Why you don’t see it in the average

Here is the part that fools most people. You open your performance dashboard, look at the last 7 days, and the line is… unremarkable. No spike, no alarm. So all good?

No. Two things hide the regression:

The average smooths it away. If a deploy slows half your pages but leaves the other half untouched, the mean barely moves. That is why web performance looks at the 75th percentile, p75 for short, not the average. p75 means: 75% of your visitors were at least this fast. The average flatters. p75 shows you the experience of the people it actually hits.

The long window dilutes it. A regression that went live two days ago drowns in five days of good data before it. Averaged over a week, the jump is a small bump. Over the 24 hours before and after, it is a wall.

Together they make sure that the very number showing the damage is the one almost nobody looks at: p75, in a short window, around the deploy.

The release marker: a moment, nothing more

For anyone to say “around the deploy”, the system has to know when the deploy was. That is all a release is: a named moment on your site. Nothing more behind it. A version (v2.4.1, a Git SHA, a date, whatever) and a timestamp. No deploy database, no changelog, just the marker.

The cleanest way is to set it automatically, at the end of your deploy job, after a successful deploy:

- name: Tag fastmon release
  if: success()
  env:
    FASTMON_TOKEN: ${{ secrets.FASTMON_TOKEN }}
    FASTMON_SITE_ID: ${{ vars.FASTMON_SITE_ID }}
  run: |
    curl -fsS https://api.fastmon.eu/v1/sites/$FASTMON_SITE_ID/releases \
      -H "Authorization: Bearer $FASTMON_TOKEN" \
      -H "Content-Type: application/json" \
      -d "{ \"version\": \"${{ github.sha }}\" }"

Four lines, set up once. From then on every successful deploy carries its own marker, with nobody having to remember when what went live.

What you compare afterwards

With the marker, “did this get worse?” becomes a single query. You pick the deploy in the release picker and the chart overlays before and after. Programmatically the lever is compare_to_release_id.

And now the important part, the one thing to understand once: the “before” window is exactly as long as your “after” window and ends precisely at the release. Query the 24 hours after the deploy and the system compares them against the 24 hours right before. Same length, same time of day, same weekday mix. Only the deploy sits in between.

Which leads to a simple rule: keep the window short. A 7-day query compares against the 7 days before and blurs the jump again. If you want to see a regression sharply, take 1 to 24 hours.

The three traps

Three things trip the comparison up in practice, and none of them is complicated:

  • The timestamp has to be right. Leave out released_at and the server sets “now”. Usually fine. But if your deploy job runs minutes before the actual go-live, say on a blue/green flip, date the marker to the moment the new version really gets traffic. Otherwise you cut at the wrong point.
  • Releases belong to a site. If a monorepo ships three sites, you need three release calls, even when the version string is identical.
  • A rollback is a release too. Tag it the same way, happily as rollback-v2.4.0. Then the chart shows all three transitions: the broken release live, the regression, and the rollback that catches the line again. You cannot document a bad day more honestly than that.

Note: The full flow, from API token through the release call in GitHub Actions, GitLab CI or Vercel to the comparison query, is in our docs on comparing releases. That link is exactly what your developers need.

Two nets, not one

The best time to catch a regression is before it goes live at all. That is what synthetic tests are for: a lab test under always-identical conditions that flags a jump before the deploy reaches real visitors. More about Synthetic Monitoring.

But the lab does not know every device and every network. Whatever slips through, you catch in the field: on real visitors, at the p75, in the short window around the release marker. More about Real User Monitoring. Two nets, one after the other. One before the deploy, one after.

What to take away

Regressions throw no error, they ride in on a deploy, and the weekly average hides them reliably. They only surface when three things come together: a marker on the deploy, the p75 instead of the average, and a short window around the marker.

The marker costs you four lines in CI. The query does the rest. And the next regression no longer gets a week’s head start.