Baseline vs Regression: Building a Repeatable Performance Benchmark for APIs

Release 412 “looked fine” in QA until p99 checkout crept 180ms over the envelope your team agreed on in March—because nobody reran the same k6 scenario with the same data and tags. A baseline is the accepted latency and error envelope for a scenario under a known environment contract. A regression is meaningful drift from that envelope after a change. Without frozen scenario DNA, you compare fiction.

In this guide you will learn what to lock in a baseline bundle, how to encode gates with k6 thresholds, when to reset baselines after infra change, and how to store comparable runs so engineer turnover does not erase memory.

Why baselines fail before thresholds do

Teams often copy a script, tweak VUs “to be safe,” and wonder why nightly charts disagree. Small drift invalidates comparisons:

• Executor mismatch (constant-vus vs arrival-rate) changes who waits in queues.
• Payload or auth drift alters CPU and payload size without a code change in the service under test.
• Environment fingerprint change (DB snapshot, cache warm, feature flags) shifts tails independently of your merge.

Industry practice ties service levels to SLIs users feel (Google SRE workbook); k6 encodes them as tagged thresholds on http_req_duration and http_req_failed. Review common load testing mistakes before you trust a red threshold—and wire smoke gates into load testing in CI/CD once the baseline is stable.

Pair baseline discipline with k6 thresholds examples when you add routes or dependencies.

Practical k6 implementation: frozen DNA, build tags, and gates

Treat the baseline script as a versioned artifact: scenario parameters, env vars, datasets, and threshold lines travel together.

Example script (illustrative—not production-ready). Fictional checkout API and envelope numbers.

What this example demonstrates:

• Env-driven fingerprint BUILD_ID on tags for report comparison across CI runs.
• Single golden scenario with constant arrival rate—stable request pressure week to week.
• Thresholds that fail the run when tails or errors breach the recorded baseline envelope.
• Abort-friendly duration for nightly smoke; extend duration for quarterly re-baselining.

import http from 'k6/http';
import { check, sleep } from 'k6';

const BASE = __ENV.API_BASE || 'https://staging.example.com';
const BUILD = __ENV.BUILD_ID || 'local';
const body = JSON.stringify({ sku: 'SKU-100', qty: 1, currency: 'USD' });

export const options = {
  scenarios: {
    baseline_checkout: {
      executor: 'constant-arrival-rate',
      rate: Number(__ENV.BASELINE_RPS || 25),
      timeUnit: '1s',
      duration: __ENV.BASELINE_DURATION || '10m',
      preAllocatedVUs: 25,
      maxVUs: 100,
      tags: { route: 'checkout', profile: 'baseline' },
      exec: 'checkout',
    },
  },
  thresholds: {
    // Envelope captured from build 408 green run — adjust only via explicit baseline renewal
    'http_req_duration{route:checkout}': ['p(95)<750', 'p(99)<1100'],
    http_req_failed: ['rate<0.008'],
  },
};

export function checkout() {
  const res = http.post(`${BASE}/v2/checkout`, body, {
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${__ENV.TOKEN}`,
    },
    tags: { route: 'checkout', build: BUILD, profile: 'baseline' },
  });
  check(res, { 'checkout 2xx': (r) => r.status >= 200 && r.status < 300 });
  sleep(0.35);
}

Patterns that work

• Store raw summary JSON + git SHA + env fingerprint beside every green baseline capture.
• Fail merges on threshold breach; use softer p95 delta review on weeklies with enough sample size—short smokes are noisy.
• Tag build on requests so Performate or k6 Cloud comparisons align runs (k6 observability metrics and tags).
• Document baseline renewal in the changelog when infra or intentional latency trades move the envelope.

Anti-patterns to avoid

• Changing RPS and duration in the same week you compare builds—pick one dimension per experiment.
• Resetting thresholds silently after a failed run “to go green.”
• Comparing staging baseline to production traffic shape without annotation.

Pro tip (example command): export JSON for archival and diff tooling.

k6 run baseline-checkout.js -e BUILD_ID=$CI_COMMIT_SHA -o baseline-summary.json

What this command demonstrates: machine-readable output attaches to the build record so regressions are diffable, not screenshot debates.

Decision framework: baseline capture vs regression gate vs reset

Capture a new baseline if you changed scenario DNA intentionally and stakeholders accept a new envelope.

Treat as regression if DNA is unchanged and tagged tails or errors cross thresholds—investigate before widening gates.

Reset dashboards only after explicit renewal notes—otherwise on-call chases ghosts from March numbers on July infra.

Observability, documentation, and next steps

Before the next release train:

• Freeze scenario DNA document: executor, duration, RPS, payload mix, think time, dataset version.
• Record environment fingerprint (DB snapshot id, feature flags, cache state procedure).
• Archive green summary JSON with git SHA when baselines are captured.
• Wire threshold failure to merge blocking in CI for the baseline scenario.
• Compare p95/p99 deltas between nightly runs with minimum sample rules.
• Log baseline renewal events with owner and reason when infra or product changes the envelope.

How Performate keeps baselines comparable across releases

Scenario definitions scattered across laptops do not survive turnover. Centralize the golden path and exports.

Example: checkout baseline tracked per build

1. Import the golden Postman flow (or OpenAPI operation) used for the March baseline capture. Problem solved: one definition for local, CI, and review meetings.
2. Set constant arrival rate to the recorded BASELINE_RPS (e.g. 25 req/s) and duration to 10m in the editor. Problem solved: scenario DNA lives in one workspace, not five forks.
3. Apply tags route:checkout, profile:baseline, and pass BUILD_ID from CI env in exported scripts. Problem solved: comparison views filter the same dimensions every run.
4. Enter thresholds matching the archived envelope (p95, p99, error rate). Problem solved: visual editing of gates without drift between hand-edited script copies.
5. Run on each release candidate and open historical comparison in the integrated report. Problem solved: product and engineering debate one export per build.
6. Export k6 for pipeline smoke at reduced duration/RPS with unchanged tags and threshold expressions—scale time, not shape.

That workflow delivers the post cta: store comparable runs and threshold outcomes between releases without apples-to-oranges charts.

Closing takeaway

API performance baseline discipline is frozen scenario DNA plus honest thresholds. Regression detection is only trustworthy when traffic shape, data, tags, and environment fingerprint match the captured green run—then let k6 fail the build when tails move.

Re-run your golden checkout scenario on the current candidate with BUILD_ID set—note whether you need a fix or a documented baseline renewal.

Try Performate free | Book a demo | k6 results output