Webhooks Under Fire: Retries, Idempotency, and Signature Verification at Scale

Webhooks fail under load when retries duplicate work, signatures burn CPU, and bursts exceed handler concurrency—not when single deliveries work in Postman. Stripe webhook guidance stresses idempotency keys and tolerant handlers—your load tests should hammer that reality before the payments provider retries a thousand payment.succeeded events into your staging cluster.

This guide models burst delivery, duplicate event_id replay, signature verification cost, and links to async backlog effects (queue pipelines). You get a k6 burst script with HMAC signing, decision routing by risk type, and abort criteria when 5xx or DLQ rates spike.

Scenarios to model

Webhook perf is not average POST latency—it is shape and duplication:

• Burst delivery—provider retry storm after your endpoint slowed for two minutes.
• Duplicate event_id payloads—same event delivered twice; handler must 200 or 409, not double-charge.
• Invalid signature path—sample low-rate 401/403 to ensure crypto failure is cheap vs success path.
• Downstream queue backlog—handler accepts fast but workers lag; HTTP green, business SLO red.
• Concurrent verification—HMAC per request at high RPS CPU-bound on small instances.

Stripe-shaped vs generic inbound

Adapt headers (Stripe-Signature, custom X-Signature) to your provider; k6 pattern stays: sign body, POST, check idempotent status codes. Never use prod signing secrets in git—k6 secrets only.

k6 burst + duplicate events

Illustrative—not production-ready. Include duplicate-event scenario in extended test plan; snippet focuses on signed burst.

What this demonstrates:

• ramping-arrival-rate spike simulates retry storm—10 → 200 req/s step.
• HMAC sign in script—models CPU cost of verification under load.
• Check accepts 200 or 409—409 as idempotent success for duplicate event_id.
• Tags route:webhook for filtering from REST scenarios in same suite.

import http from 'k6/http';
import { check, sleep } from 'k6';
import crypto from 'k6/crypto';

const secret = __ENV.WEBHOOK_SECRET || 'test_secret';
const BASE = __ENV.API_BASE || 'https://staging.example.com';

function sign(body) {
  return crypto.hmac('sha256', secret, body, 'hex');
}

export const options = {
  scenarios: {
    webhook_burst: {
      executor: 'ramping-arrival-rate',
      startRate: 10,
      timeUnit: '1s',
      preAllocatedVUs: 30,
      maxVUs: 150,
      stages: [
        { duration: '1m', target: 10 },
        { duration: '30s', target: 200 },
        { duration: '2m', target: 200 },
        { duration: '1m', target: 10 },
      ],
      tags: { suite: 'webhook_burst' },
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.02'],
    'http_req_duration{route:webhook}': ['p(95)<1500'],
  },
};

export default function () {
  const body = JSON.stringify({
    event_id: `evt-${__VU}-${__ITER}`,
    type: 'payment.succeeded',
  });
  const res = http.post(`${BASE}/webhooks/inbound`, body, {
    headers: {
      'Content-Type': 'application/json',
      'X-Signature': sign(body),
    },
    tags: { route: 'webhook', suite: 'webhook_burst' },
  });
  check(res, { accepted: (r) => r.status === 200 || r.status === 409 });
  sleep(0.05);
}

Patterns that work

• Duplicate subset—10% iterations reuse fixed event_id list—validates idempotency store sizing.
• Abort tripwire on 5xx > 1% during burst—stop before corrupting shared staging data.
• Monitor DLQ and idempotency DB during run—HTTP is necessary, not sufficient.
• Synthetic events only—no prod customer ids in payloads (GDPR).

Anti-patterns to avoid

• Load test against prod webhook URL without provider approval.
• Measuring only 200 rate—ignore 409 semantics and duplicate side effects.
• Ignoring async worker backlog—pair with queue lag metrics.
• Zero signature failures in test plan—crypto regressions hide until prod forgery attempts.

Pro tip (example command): export webhook-only stats for payments team.

k6 run webhook-burst.js --summary-trend-stats="p(95),p(99)" --tag route=webhook

What this command demonstrates: filtered summary attaches to payments review without REST noise from unrelated scenarios in the same repo.

Decision table: risk vs test

Observability and pre-run checklist

• Idempotency store sized for burst duplicates—document max event_id cardinality tested.
• DLQ monitoring during test with abort owner named.
• Synthetic events only; signing secret from env profile.
• Downstream workers scaled knowingly—interpret HTTP vs lag separately.
• Compare burst to spike test language in launch runbook.
• Archive export for payments/compliance if regulated flows touched.

How Performate supports webhook load testing

Below is a concrete workflow example for inbound payment webhooks—adapt signing header to your provider.

Example: Postman import → burst scenario → payments export

1. Import webhook request from Postman—body template preserved. Problem solved: signature payload matches what functional tests already use.
2. Add pre-request sign logic in exported k6—HMAC from env secret. Problem solved: crypto path included without manual reimplementation each sprint.
3. Configure burst stages in UI—30s spike to 200 req/s. Problem solved: visual stages reduce YAML errors before payments review.
4. Run with abort tripwire on 5xx—Slack owner tagged. Problem solved: shared staging protected during aggressive burst.
5. Export report for payments team—409 rate visible alongside 200. Problem solved: idempotency story told with numbers, not code walkthrough.
6. Promote single-delivery smoke to CI—burst stays manual pre-release. Problem solved: daily regression on wiring without daily retry storm.

That workflow maps directly to the cta in this post: runnable webhook scenarios and shareable reports without glue-code sprints.

Closing takeaway

Webhook perf is bursts + duplicates + crypto, not average POST latency. Schedule a retry-shaped spike before the next payments incident; validate 409 paths; watch DLQ while HTTP still looks fine.

Import your inbound webhook, add signing, run one burst on staging, and record where 5xx begins—that rate belongs in the payments runbook.

Follow the burst with a low-rate duplicate-event_id scenario the same day—idempotency store exhaustion often appears only after the spike when retries stack.

Capture DLQ depth at burst peak in the same ticket as the k6 export—payments reviews trust graphs that share a timestamp with the load window.

Treat 409 responses as first-class metrics in the report—not errors—so idempotent success does not look like failure to executives scanning red counts.

Try Performate free | Book a demo | Stripe webhooks