Ir para o conteúdo

Load-test Ultra @ 100k — runbook (issue #53 Task 1)

Validates that the Ultra-tier batch path (POST /batch_reports) sustains 100 000 receivers per submission within the published Ultra SLA (99.95% / per-batch deadline 1800 s) before raising the public Ultra max_batch_rows from 50 000 → 100 000.

This runbook covers Phase 1 (code/config prep, no infra cost) plus the checklist to run Phase 2/3 when GPU capacity is approved.

Phase 1 — what changed in code (already shipped)

  • TIER_LIMITS[*].max_batch_rows is now env-overridable per tier via MAX_BATCH_ROWS_{FREE,STARTER,PRO,BUSINESS,ENTERPRISE,ULTRA}. Default values are unchanged on production.
  • New optional staging-only Ultra demo key gated by two envs:
  • ENABLE_ULTRA_DEMO_KEY=true
  • ULTRA_DEMO_KEY=<opaque-token> (NEVER hardcoded; load from secret manager)

When both are set, the configured token authenticates as Tier.ULTRA. If ENABLE_ULTRA_DEMO_KEY=true but ULTRA_DEMO_KEY is empty, the API logs a warning and registers no key (fail-safe).

Production runtime is unaffected as long as ENABLE_ULTRA_DEMO_KEY is unset (default) and MAX_BATCH_ROWS_ULTRA is unset (defaults to 50 000).

Phase 2 — staging stand-up

Copy-pasteable commands (TD diff, SSM put, Batch CE scale, teardown) live in staging-gpu-standup.md. The summary below is the contract; that document is the execution.

Required envs on the staging task definition (NOT production):

Env Value
ENABLE_ULTRA_DEMO_KEY true
ULTRA_DEMO_KEY random 32-byte token from SSM /telecom-tower-power-staging/ULTRA_DEMO_KEY
MAX_BATCH_ROWS_ULTRA 100000
RATE_LIMIT_ULTRA keep 5000 (per-minute; the test never exceeds it)
ENABLE_DEMO_KEYS true

DNS: add an A/ALIAS record for api.staging.telecomtowerpower.com.br pointing at the staging ALB (or reuse the prod ALB with a new host rule).

Smoke before the real run:

export LOCUST_API_KEY="$(aws ssm get-parameter \
  --name /telecom-tower-power-staging/ULTRA_DEMO_KEY \
  --with-decryption --query Parameter.Value --output text)"
locust -f locustfile.py \
  --host https://api.staging.telecomtowerpower.com.br \
  --headless --tags smoke -u 5 -r 1 -t 30s

Expected: HTTP 200 on /health, no 401/403, sub-second p95.

Phase 3 — full Ultra batch test

First-ever Phase 3 execution? Do NOT run the 4-worker command block below. Follow load-test-ultra-phase3-first-run.md instead — opinionated 3-stage ramp (1×10k → 1×100k → 2×100k) with hard abort gates and mandatory teardown. The 4-worker plan in this section is the second run, scheduled only after one clean first-run execution.

GPU worker capacity should mirror the production target pool size for the SLA you intend to publish. Spin up the workers BEFORE submitting the first 100k payload (a cold pool will burn the SLA window on warmup).

# Mixed-API background traffic (parallel terminal A):
locust -f locustfile.py \
  --host https://api.staging.telecomtowerpower.com.br \
  --headless --tags smoke towers analyze batch repeater \
  -u 750 -r 50 -t 45m \
  --csv reports/ultra-mixed --html reports/ultra-mixed.html

# Ultra 100k batch driver (parallel terminal B):
export LOCUST_API_KEY="$(aws ssm get-parameter \
  --name /telecom-tower-power-staging/ULTRA_DEMO_KEY \
  --with-decryption --query Parameter.Value --output text)"
export LOCUST_ULTRA_ROWS=100000
export LOCUST_ULTRA_DEADLINE_S=1800
locust -f locustfile.py \
  --host https://api.staging.telecomtowerpower.com.br \
  --headless --tags ultra -u 4 -r 1 -t 45m \
  --csv reports/ultra-batch --html reports/ultra-batch.html

Metrics to capture

End-to-end (from locust CSV / HTML):

  • p50, p95, p99 latency on /batch_reports?ultra=100000
  • p50, p95, p99 polling time on /jobs/[ultra_job_id]
  • Failure rate (any class above 0.05% breaches Ultra SLA)
  • Total successful 100k batches / total attempted

Infrastructure (CloudWatch, Grafana):

  • GPU utilization peak + sustained avg per worker
  • Worker container memory peak (OOM watermark check)
  • Postgres CPU + connections (RDS metrics)
  • Redis CPU + commands/sec (ElastiCache, if applicable)
  • SQS queue depth peak + age-of-oldest-message
  • ALB target 5xx count
  • Per-batch cost = (GPU instance $/h × wall-clock h) + (egress + storage)

Decision gate

Result Action
p99 within SLA window (1800 s) and failure rate ≤ 0.05% Raise public Ultra max_batch_rows 50 000 → 100 000 in Pricing.jsx, rate-limits.md, README.md. Update MAX_BATCH_ROWS_ULTRA default to 100000 in code.
Any breach Keep Ultra at 50 000 publicly. Document the validated ceiling internally in this runbook's "Run log" section.

Archive

After every full run (Phase 3), append a "Run log" entry below with:

  • Date, staging task-def revision, GPU worker count
  • Locust HTML/CSV report path (commit under docs-site/docs/operations/load-test-reports/<date>/)
  • Outcome vs SLA
  • Decision taken

Run log

(no full runs yet — Phase 1 only, 2026-05-25)

Safety / non-goals

  • DO NOT run Phase 3 against api.telecomtowerpower.com.br (production) while real customers are live — even 4 concurrent 100k batches will saturate the GPU pool and breach other customers' SLA.
  • DO NOT publish a hard SLA number (Task 3 of #53) until at least 3 consecutive Phase 3 runs land inside SLA across a 7-day window.
  • DO NOT leave MAX_BATCH_ROWS_ULTRA=100000 or ENABLE_ULTRA_DEMO_KEY=true set on the production task definition. They are staging-only knobs.