Soak Tests and Long Runs: Using AI to Spot Patterns in Noisy Time Series

Your soak run finished at 3 a.m. with a flat error rate and a p95 line that barely moved—yet production still melted down on Tuesday from a connection pool that never released handles. Humans glaze over six hours of nearly identical charts; models can flag slope changes and periodic teeth in structured time series—if you export the right buckets and refuse hand-wavy summaries.

AI soak test analysis is not a replacement for heap dumps or query traces. It is a triage accelerator: turn long k6 evidence into candidate windows, then prove causes with infra metrics and controlled follow-up runs. In this guide you will learn what to export from soak tests, how to script k6 for interval-friendly metrics, which patterns models hunt well (and where they hallucinate), and how to close the loop with engineering tickets.

Why soak failures hide in noise—not spikes

Short load tests catch saturation and obvious misconfiguration. Soak tests catch slow modes: memory creep, pool churn, log-induced CPU, cache inefficiency, and replication lag that only appears after hours of steady traffic.

Under a long run, symptoms look boring:

• Gradual p99 lift with flat http_req_failed—queueing, GC, or fragmentation, not “the API is down.”
• Sawtooth latency aligned with cron, cache eviction, or batch jobs on shared staging.
• Error bursts that match connection pool timeouts or broker backpressure minutes after traffic looked healthy.

Pair soak discipline with the soak testing playbook for duration and abort rules. Use AI commentary (Gemini load test analysis) only on aggregated intervals—models need timestamps and values, not vague screenshots.

Define success before you start the clock

Document upfront: max error rate, max p99 growth vs hour-one baseline (or vs steady-state start at T+X), and when to abort so shared staging is not collateral damage. Hour one is often distorted by JIT warmups and cache fills—either exclude warm-up from SLO evaluation or mark “steady state starts at T+45m” on the run sheet.

Practical k6 implementation: soak shape, buckets, and drift tags

Soak scripts should hold arrival rate steady (not ramp-to-failure) and emit metrics you can bucket for AI and dashboards.

Example script (illustrative—not production-ready). Uses fictional URLs and SLO numbers. Adapt duration, rates, and thresholds to your risk profile.

What this example demonstrates:

• Long steady load with constant-arrival-rate so VU count does not mask pool exhaustion.
• Route tags on every request for per-endpoint drift in summaries.
• Custom trend metric soak_drift_ms sampled each iteration—optional hook for external aggregators; primary analysis still uses built-in http_req_duration percentiles per interval export.
• Strict error ceiling plus relaxed duration thresholds suited to multi-hour runs (tighten for smoke soaks).

import http from 'k6/http';
import { check, sleep, Trend } from 'k6';

const BASE = __ENV.API_BASE || 'https://staging.example.com';
const soakDrift = new Trend('soak_drift_ms', true);

export const options = {
  scenarios: {
    soak_checkout: {
      executor: 'constant-arrival-rate',
      rate: Number(__ENV.SOAK_RPS || 8),
      timeUnit: '1s',
      duration: __ENV.SOAK_DURATION || '4h',
      preAllocatedVUs: 15,
      maxVUs: 60,
      tags: { scenario: 'soak', route: 'checkout' },
      exec: 'checkout',
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.005'],
    // Soak SLO: tail may drift—gate on errors first; compare p99 buckets offline
    'http_req_duration{route:checkout}': ['p(99)<2000'],
  },
};

export function checkout() {
  const start = Date.now();
  const res = http.post(`${BASE}/v2/checkout`, JSON.stringify({ sku: 'SKU-100', qty: 1 }), {
    headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${__ENV.TOKEN}` },
    tags: { route: 'checkout', scenario: 'soak' },
  });
  soakDrift.add(Date.now() - start);
  check(res, { 'checkout 2xx': (r) => r.status >= 200 && r.status < 300 });
  sleep(Number(__ENV.THINK_SEC || 0.5));
}

Patterns that work

• Export 5-minute buckets: requests, errors, p95/p99, active VUs—small enough for AI, large enough to smooth noise.
• Attach infra overlays when policy allows: heap, GC time, DB connections—soak truth is rarely client-only.
• Annotate deployment markers (cache warm, config reload) in the run sheet so models do not blame your service for someone else’s batch job.
• Compare same-duration windows across builds with build_id columns in CSV exports.

Anti-patterns to avoid

• Feeding raw per-request CSVs with millions of rows—aggregate first, drill second.
• Treating a single spike on shared staging as your regression without checking diurnal noise (staging vs production load tests).
• Accepting AI text that says “degradation seems present” without cited timestamps and metric values.

Pro tip (example command): surface percentile trends in the summary for post-soak review.

k6 run soak-checkout.js --summary-trend-stats="p(95),p(99),avg" -o soak-summary.json

What this command demonstrates: JSON plus percentile stats give you a portable artifact to bucket externally or paste into bounded AI prompts alongside interval CSVs.

Decision framework: duration, exports, and AI vs manual review

Use AI triage if you have structured interval exports and a human owner who will run heap dumps, thread dumps, or query traces on flagged windows.

Use manual-only review if the run is short, the environment is too noisy, or you lack infra overlays—models will overfit to spikes.

Abort early if error rate crosses a red line or saturation breaks unrelated teams—soaks are learning tools, not denial-of-service drills.

Observability, documentation, and next steps

Before the next overnight soak:

• Document duration rationale beside the scenario name (why 4h vs 8h).
• Export 5-minute aggregates with build_id, git SHA, and environment fingerprint.
• Define steady-state start (exclude warm-up from SLO comparison).
• Post start/stop times to ops channels on shared clusters.
• Turn each AI-flagged window into a ticket: metric snippet, suspected subsystem, next experiment.
• Overlay last green soak vs candidate for the same window before merging perf fixes.

When long runs expose database lag, cross-check read replicas and eventual consistency load testing.

How Performate simplifies soak review with AI assistance

Long runs generate large artifacts; desktop workflow keeps imports, execution, and commentary in one place.

Example: four-hour checkout soak with AI-assisted pattern review

1. Import the Postman collection (or OpenAPI) for the checkout path you will soak. Problem solved: one source of truth for headers and payloads across reruns.
2. Create a constant-arrival-rate scenario at production-like RPS (e.g. 8 req/s) and set duration to 4h in the visual editor. Problem solved: no hand-editing executor blocks when product changes duration.
3. Apply tags scenario:soak and route:checkout on the scenario. Problem solved: reports slice latency the same way as the k6 example above.
4. Run the soak locally or against staging and capture the integrated report export. Problem solved: comparable summaries without juggling three spreadsheets.
5. Where your plan allows, use AI-assisted analysis on bounded summaries—require timestamps and values in the narrative, reject vague degradation language. Problem solved: faster candidate windows; you still verify with infra metrics.
6. Export the k6 script for CI nightly soaks at reduced rate/duration with the same tags and thresholds.

That workflow matches the post cta: ship faster on post-soak review without skipping validation or root-cause proof.

Closing takeaway

AI soak test analysis pairs disciplined k6 soaks with structured exports—models highlight candidate drift; engineers prove causes. Flat p95 lines can hide climbing tails; feed interval buckets, infra overlays, and build labels, then verify every AI-flagged window with experiments—not opinions.

Schedule this week’s soak with documented abort rules and steady-state boundaries—note which metric bucket diverged first from your last green run.

Try Performate free | Book a demo | k6 scenarios