Kubernetes API and Microservice Load Testing: Ingress, Timeouts, and HPA Signals

Your microservices pass k6 against localhost:8080—then production misses SLOs because ingress idle timeouts clip long polls, HPA adds pods sixty seconds after the queue backs up, and the mesh sidecar adds twelve milliseconds you never modeled. Kubernetes shifts failure modes; load tests must target customer-visible hostnames and correlate ramps with cluster signals, not pod IPs in a dev laptop port-forward.

Kubernetes adds layers—Ingress or Gateway API, Services, Endpoints, HPAs, and optional service mesh sidecars—each with timeouts, connection limits, and scaling lag. k6 remains the execution engine; the discipline is testing through the same front door users hit while watching kubectl top, HPA events, and saturation metrics alongside http_req_duration.

In this guide you will learn which K8s-specific bottlenecks dominate API latency stories, how to structure k6 scenarios against ingress hostnames, and which observables should block a release before you scale traffic into shared staging clusters.

Why Kubernetes changes performance—not just deployment

Functionally, your API behaves the same in a pod as on a VM. Under load through the cluster path, different limits appear:

• Ingress and gateway timeouts (proxy-read-timeout, proxy-send-timeout) terminate requests that outlive upstream slowness—k6 sees 502/504 bursts, not gradual latency growth.
• Connection reuse and keep-alive through L7 load balancers differ from direct pod tests—TLS handshakes and SNIs add cost at scale.
• HPA lag scales on CPU or custom metrics minutes after queue depth spikes—ramps that jump instantly hide autoscaling debt (capacity planning).
• Service mesh sidecars add CPU, memory, and hop latency—compare tagged runs with mesh:on vs mesh:off (observability tags).
• Noisy neighbors and limits on shared staging nodes skew results if you never record node pool and resource requests during the test.

Think of pod-local k6 as testing an engine on a lift; ingress-targeted k6 tests the whole car on the highway—including toll booths and on-ramps.

When in-cluster smoke passes but ingress fails at peak

A port-forward to one pod proves container health. It skips DNS, TLS termination, WAF rules, and cross-AZ routing. Teams discovered this gap when queue/worker pipelines looked healthy at the API while ingress returned 504s during fan-out—because the test never exceeded the controller's connection table.

Pair ingress tests with mock vs real dependencies consciously: mocks hide downstream saturation that HPA cannot fix.

Practical k6 implementation: ingress hostnames and ramp discipline

Target the external hostname and TLS config customers use. Ramp gradually so HPA and cluster autoscaler have time to react—unless you explicitly test cold-start failure modes. Below is an illustrative script; adapt hostnames, paths, and SLO numbers to your environment.

What this example demonstrates:

• External BASE URL: hits ingress hostname, not 127.0.0.1:8080.
• Gradual ramp: ramping-arrival-rate exposes HPA lag vs instant spikes.
• Mesh comparison tags: mesh:enabled env toggles headers or routes for A/B runs.
• Separate error and tail thresholds: ingress 502 bursts fail the run before average latency looks acceptable.

import http from 'k6/http';
import { check, sleep } from 'k6';

const BASE = __ENV.API_BASE || 'https://api.staging.example.com';
const MESH = __ENV.MESH_ENABLED === 'true' ? 'on' : 'off';

export const options = {
  scenarios: {
    ingress_ramp: {
      executor: 'ramping-arrival-rate',
      startRate: 5,
      timeUnit: '1s',
      preAllocatedVUs: 30,
      maxVUs: 200,
      stages: [
        { duration: '3m', target: 30 },
        { duration: '10m', target: 30 },
        { duration: '2m', target: 0 },
      ],
      tags: { route: 'search', mesh: MESH, target: 'ingress' },
      exec: 'search',
    },
  },
  thresholds: {
    'http_req_duration{route:search}': ['p(95)<500', 'p(99)<900'],
    'http_req_failed{route:search}': ['rate<0.02'],
  },
};

export function search() {
  const res = http.get(`${BASE}/v1/search?q=load-test`, {
    headers: {
      Authorization: `Bearer ${__ENV.TOKEN}`,
      'X-Mesh-Enabled': MESH === 'on' ? 'true' : 'false',
    },
    tags: { route: 'search', mesh: MESH, target: 'ingress' },
    timeout: '120s',
  });
  check(res, {
    'search not gateway timeout': (r) => r.status !== 504 && r.status !== 502,
    'search 2xx': (r) => r.status >= 200 && r.status < 300,
  });
  sleep(0.2);
}

Patterns that work

• Match ingress timeout annotations to k6 timeout values—know whether slowness or proxy kill fired.
• Watch HPA during steady stage: kubectl get hpa -w and correlate scale events with k6 RPS.
• Record node pool, pod count, and CPU throttling in the test runbook—not just k6 summaries.
• Run mesh on/off pairs at identical RPS before blaming application code (HTTP/2 and HTTP/3 tuning).

Anti-patterns to avoid

• Load-testing pod IPs while production traffic uses ingress—misleading SLO sign-off.
• Instant max RPS ramps on autoscaling workloads without documenting intentional cruelty.
• Ignoring staging cluster quotas—saturating shared ingress hurts neighboring teams.
• Single aggregate latency threshold when 502 bursts hide behind fast 200s from cache hits.

Pro tip (example command): tag runs with cluster context for postmortems that cite identical conditions.

k6 run ingress-search.js --tag cluster=staging-eu1 --tag git_sha=$CI_COMMIT_SHA --summary-export=summary.json

What this command demonstrates: exported summaries tie k6 results to cluster metadata so regressions compare the same ingress, mesh mode, and release.

Decision framework: what to test through Kubernetes layers

Test through ingress if customers never touch pod networks directly—which is almost always.

Test HPA reaction if autoscaling metrics drive your capacity story—hold steady RPS long enough for scale-out.

Test mesh variants if sidecars ship mTLS and retry policies that change tail latency.

Observability, documentation, and next steps

Kubernetes load tests fail postmortems when cluster state was not captured. Before you scale traffic:

• Document ingress hostname, TLS cert source, timeout annotations, and WAF rules in the runbook.
• Record start/end pod counts, HPA min/max, and CPU/memory limits for the target Deployment.
• Correlate k6 tags with APM and service mesh dashboards during the steady stage.
• Snapshot kubectl top pods and node pressure at peak RPS.
• Archive k6 summary JSON with cluster name, mesh mode, and git SHA per release.

How Performate simplifies Kubernetes-targeted load tests

Teams lose days wiring auth, hostnames, and export metadata every time ingress changes. Below is a concrete workflow example for the search API through staging ingress.

Example: ingress-realistic search scenario without port-forward hacks

1. Import OpenAPI or Postman for GET /v1/search with production-like query params. Problem solved: URLs and headers match what ingress routes—not localhost shortcuts.
2. Set API_BASE to the staging ingress hostname (https://api.staging.example.com) in the environment panel. Problem solved: TLS and DNS paths match customer traffic.
3. Create a ramping-arrival-rate scenario—5→30 req/s over fifteen minutes. Problem solved: visual ramp tuning while you watch HPA in another terminal.
4. Apply tags route:search, target:ingress, and mesh:on/off from env. Problem solved: reports isolate mesh overhead debates with data, not anecdotes.
5. Set thresholds on p99 and gateway error rate (502/504 checks in script). Problem solved: release gates catch proxy kills average latency hides.
6. Export k6 script and summary with scenario metadata for the postmortem folder. Problem solved: cluster snapshots and traffic mixes cite the same artifact.

That workflow maps directly to the cta in this post: runnable scenarios and shareable reports without losing days to glue code.

Closing takeaway

Kubernetes load testing is an ingress, timeout, and autoscaling problem—not a pod health check. Target external hostnames, ramp honestly, tag mesh modes, and correlate k6 ramps with HPA events before you sign off on capacity.

Run this week's search ramp through staging ingress—not port-forward—and note whether latency tails or gateway errors appear first when pods scale.

Try Performate free | Book a demo | k6 running in Kubernetes