Soak Testing Playbook: Memory Leaks, Connection Drift, and When to Stop the Run

Short load tests miss memory leaks and connection pool drift that need hours to surface. Soak tests hold moderate load long enough to watch slope—not peak RPS (Grafana soak guidance). The question is not "did we survive five minutes?" but "does latency or error rate creep while traffic stays flat?"

This playbook covers when to soak, how to staff abort decisions, a k6 scenario at ~70% peak RPS, and correlation with APM heap charts. Pair with quarterly health soak sample—full soak before longest uptime windows.

When to soak (and when to skip)

Soak tests cost time and staging occupancy. Run them when slow degradation causes real incidents:

• Before multi-day events—conferences, payroll windows, holiday shopping peaks.
• After JVM/Node/runtime upgrades—GC behavior changes expose leaks old baselines missed.
• When incidents show slow metric creep—p95 rises over hours without deploy or traffic change.
• After connection pool or ORM config changes—idle timeout mismatches show up as gradual exhaustion.

Skip full soak when you lack staffed APM watch, staging is shared with no reservation, or you have not passed a 10-minute steady load baseline—fix obvious regressions first (stress vs load).

Abort criteria before you start

Staff an abort owner with authority to stop the run without war room. Example tripwires:

• p95 slope rises >20% over 2 hours with flat k6 RPS.
• Error rate exceeds 1% for 10 consecutive minutes.
• Heap or connection count linear growth on APM with no plateau.
• DLQ or queue depth rises in linked async paths (queue testing).

Document tripwires in the ticket—on-call at hour four should not debate whether to stop.

k6 soak scenario

Illustrative—not production-ready. Production soak may run 2–4 hours; example uses 30m for lab-friendly iteration—extend duration before real events once slope monitoring works.

What this demonstrates:

• constant-arrival-rate at moderate RPS—~70% of documented peak, not hero load.
• Tags suite:soak filter metrics from short CI smoke in same service.
• Longer think time reduces artificial churn if route is read-heavy.
• Thresholds on duration percentiles—watch trends across time, not only final summary (export to Prometheus/Grafana for slope).

import http from 'k6/http';
import { check, sleep } from 'k6';

const BASE = __ENV.API_BASE || 'https://staging.example.com';

export const options = {
  scenarios: {
    soak: {
      executor: 'constant-arrival-rate',
      rate: Number(__ENV.SOAK_RPS || 15),
      timeUnit: '1s',
      duration: __ENV.SOAK_DURATION || '30m',
      preAllocatedVUs: 10,
      maxVUs: 50,
      tags: { suite: 'soak' },
    },
  },
  thresholds: {
    'http_req_duration{suite:soak}': ['p(95)<800'],
    http_req_failed: ['rate<0.01'],
  },
};

export default function () {
  const res = http.get(`${BASE}/work`, { tags: { route: 'work', suite: 'soak' } });
  check(res, { ok: (r) => r.status < 500 });
  sleep(Number(__ENV.THINK_SEC || 1));
}

Patterns that work

• Export k6 metrics to Prometheus/Grafana—plot p95 per minute, not only end summary.
• Correlate heap, GC, pool active count with k6 timeline—screenshot for leak tickets.
• Moderate RPS—soak finds drift; stress finds breaking point—different goals.
• Same synthetic data rules as shorter tests (GDPR data).

Anti-patterns to avoid

• Soak at 100% peak RPS—confuses capacity failure with drift.
• Unstaffed overnight run on shared staging—nobody aborts; teammates blame your test for their failures.
• Single final p95 number in executive slide—hides slope story.
• Ignoring async side effects—HTTP flat while queue depth climbs.

Pro tip (example command): push metrics to Grafana for slope charts.

k6 run soak.js -o experimental-prometheus-rw --tag suite=soak

What this command demonstrates: when Prometheus remote write is configured, p95 over time becomes visible—summary-only soak misses the creep this playbook targets. Adapt output flag to your observability stack.

Decision table: signals during soak

Stop early when slope is obvious—another two hours rarely changes the ticket priority.

Observability and pre-run checklist

• Moderate RPS (~70% of peak) documented with source.
• On-call or abort owner watching APM + k6 live—not "check in the morning."
• Post-run GC/log/thread dump procedure linked in runbook.
• Staging reservation communicated—avoid collision with CI load jobs.
• Synthetic data only; write paths idempotent if reruns overlap.
• Archive Grafana dashboard link + k6 summary JSON in same ticket.

How Performate supports soak testing

Below is a concrete workflow example for extending a steady scenario to soak—adapt route and duration to your event calendar.

Example: steady clone → overnight soak

1. Clone steady scenario from last green load run—same requests and tags. Problem solved: soak tests real user path, not synthetic one-off URL.
2. Lower rate to ~70% peak; extend duration in UI—30m lab, 2h+ pre-event. Problem solved: no hand-editing executor blocks for duration-only change.
3. Run overnight on staging with abort owner named in Slack channel. Problem solved: desktop export + live metrics without building Grafana glue first.
4. Export trend charts from integrated report for slope review. Problem solved: leak ticket attachment ready even if Prometheus export not wired yet.
5. File leak ticket if slope bad—link export + APM screenshot timestamp. Problem solved: dev team gets time-bounded evidence, not "perf felt worse."
6. Archive for quarterly health soak sample comparison. Problem solved: quarter-over-quarter drift visible in same workspace lineage.

That workflow maps directly to the cta in this post: runnable soak scenarios and shareable trend exports without glue code.

Closing takeaway

Soak answers does it degrade over time?—schedule one before your longest uptime window, staff abort tripwires, and plot slope—not only final percentiles.

Clone your steadiest load scenario, cut RPS to seventy percent of peak, run thirty minutes with someone watching APM, and stop early if the line climbs.

Try Performate free | Book a demo | k6 soak testing