Load-test Ultra @ 100k + NOC runbook

Operational artifact for issue #53 — Task 1 (Ultra batch validation) and Task 2 (24/7 NOC runbook template). Task 3 (publishing a hard Carrier SLA number) is deliberately out of scope until Tasks 1 and 2 are signed off.

Status: runbook authored, awaiting first execution against staging. Update §6 with measured numbers after each run and link the resulting CSV/HTML artifacts.

1. Scope and target environment

Field	Value
Target	https://api.staging.telecomtowerpower.com.br (ECS service telecom-tower-power in sa-east-1)
Profile (mixed surface)	100k requests total, ramp to ~500 RPS sustained for 10 min
Profile (Ultra batch)	4 concurrent 100k-receiver batch submitters, each polled for ≤ 30 min
Worker pool under test	batch_gpu_interference_worker.py at production replica count
Datastores	Staging RDS Postgres, Elasticache Redis, S3/MinIO bucket for batch outputs
Tooling	Locust (locustfile.py), Grafana (telecom-tower-power.json), CloudWatch Container Insights

2. SLOs being validated

Endpoint / surface	p50	p95	p99	Error budget
GET /health	≤ 50 ms	≤ 150 ms	≤ 300 ms	0.05%
GET /towers, GET /towers/nearest	≤ 150 ms	≤ 500 ms	≤ 1 s	0.1%
POST /analyze	≤ 600 ms	≤ 2 s	≤ 3 s	0.5%
POST /batch_reports (sync ≤ 100 rows)	≤ 1 s	≤ 3 s	≤ 5 s	0.5%
POST /batch_reports (async, Ultra 100k)	end-to-end ≤ 30 min wall time, ≥ 99.95% completion
GET /jobs/{id}, /jobs/{id}/download	≤ 200 ms	≤ 800 ms	≤ 1.5 s	0.1%

Overall mixed-surface 5xx rate must stay ≤ 0.1% across the 10-minute sustain window.

3. Pre-flight checklist

• [ ] Staging ECS service desired count matches production scaling profile (see fargate_autoscaling.py).
• [ ] Staging RDS instance class and Redis node count match production.
• [ ] GPU worker pool (ttp-gpu-batch-worker ECS service) is at production replica count.
• [ ] Test API key (LOCUST_API_KEY) is a dedicated Carrier-tier staging key — never reuse a production tenant key.
• [ ] Staging Alertmanager routes are switched to the load-test receiver (silent channel) so the run does not page the on-call.
• [ ] Baseline Grafana snapshot captured before ramp-up (telecom-tower-power dashboard at T-5min).
• [ ] reports/ directory exists locally (mkdir -p reports).

4. Execution

Run the two profiles in parallel terminals. The Ultra batch users have very long wait_time (10–30 s) and 30-min job polls, so co-running them with the mixed surface in a single Locust process distorts the RPS metric — keep them split.

# Terminal A — mixed surface (~500 RPS sustained 10 min)
locust -f locustfile.py \
  --host https://api.staging.telecomtowerpower.com.br \
  --headless --tags smoke towers analyze batch repeater \
  -u 750 -r 50 -t 10m \
  --csv reports/ultra-mixed --html reports/ultra-mixed.html

# Terminal B — Ultra batch (4 concurrent 100k-receiver jobs, up to 45 min)
LOCUST_ULTRA_ROWS=100000 LOCUST_ULTRA_DEADLINE_S=1800 \
locust -f locustfile.py \
  --host https://api.staging.telecomtowerpower.com.br \
  --headless --tags ultra \
  -u 4 -r 1 -t 45m \
  --csv reports/ultra-batch --html reports/ultra-batch.html

The mixed run achieves ~500 RPS once 750 users are spawned (each does ~0.66 task/s given wait_time = between(0.5, 2.0)). Adjust -u if observed RPS at minute 2 is below 450 or above 550.

5. Metrics to capture

Snapshot the following at T+0 (ramp start), T+5 (mid-sustain), T+10 (end of sustain), T+30 (after Ultra batches drain):

• Locust CSV stats (reports/ultra-mixed_stats.csv, reports/ultra-batch_stats.csv) — automatic.
• Grafana: telecom-tower-power dashboard panels — request rate, latency percentiles, residual booster size, fallback rate.
• CloudWatch Container Insights: ECS task CPU/memory at the API service and the GPU worker service. Watch for MemoryUtilization ≥ 80% or sustained CPU ≥ 75%.
• RDS: DatabaseConnections, CPUUtilization, ReadIOPS/WriteIOPS, FreeableMemory.
• Elasticache Redis: CurrConnections, CPUUtilization, Evictions, CacheHits/CacheMisses.
• GPU worker pool (if NVIDIA DCGM exporter scraped): GPU utilization %, GPU memory used.
• Cost per batch = (GPU worker minutes × on-demand rate) + (API task-seconds × Fargate rate) + (S3 PUT count × $0.005/1k). Record per 100k batch.

6. Results

Fill in after each run. Link the archived reports/ artifacts.

Run date	Commit	p50 mixed	p95 mixed	p99 mixed	Ultra p50 e2e	Ultra p95 e2e	5xx rate	Cost / 100k	Verdict
pending

7. Decision gate

After the first clean run:

Outcome	Action
All SLOs in §2 met AND Ultra p99 e2e ≤ 30 min AND 5xx ≤ 0.1%	Raise public Ultra batch limit 50k → 100k in Pricing.jsx, docs-site/docs/.../rate-limits.md, README.md. Open the PR with this runbook linked as the validation artifact.
Any SLO violated	Keep public Ultra at 50k. File an issue per failing component with the captured metrics. Do not publish a hard Carrier SLA number (Task 3 stays parked).
Inconclusive (single-run anomaly)	Re-run after 24h. Two consecutive passes required before raising public limits.

8. Triage matrix

Symptom → likely cause → first mitigation.

Symptom during load-test	Most likely cause	First action
POST /analyze p95 jumps to > 5 s	API task CPU pegged (single-tenant batch reused worker thread)	Scale API service +2 tasks; check fargate_autoscaling.py target tracking thresholds
POST /batch_reports returns 503	SQS publish failure or queue depth alarm	aws sqs get-queue-attributes for queue depth; check Lambda dead-letter
GET /jobs/{id} p99 > 3 s	RDS DatabaseConnections near max	Add RDS read replica routing OR raise pool size in db.py
Ultra batch stays in processing past 25 min	GPU worker pool starved	Check GPU worker ECS DesiredCount vs RunningCount; inspect dead-letter for poisoned messages
residual_booster_bytes flat at 0 mid-run	Booster failed to load (over ceiling or sidecar mismatch)	Already alerted by ResidualBoosterOversized; trainer-side gate caught it. No mitigation needed during load-test — flag for follow-up
5xx burst correlated with Redis Evictions ≥ 0	Redis memory exhausted by SRTM tile cache	Bump Elasticache node size; check SRTM_REDIS_URL TTL settings
Locust shows ConnectionError spikes	ALB target deregistration during deploy or scale-in	Confirm no concurrent deploy; check ECS deployment events

9. NOC paging matrix (Tier-1 Carrier go-live template)

Promoted to a standalone document. See carrier-noc-runbook.md for:

• RTO / RPO per component
• Alertmanager → NOC routing (already wired in alertmanager/entrypoint.sh; inert until NOC_CARRIER_WEBHOOK_URL is set)
• Maintenance and change-window policy
• Per-customer template (filled out under docs-site/docs/operations/carrier/<customer-slug>.md, not committed publicly)
• Pre-countersign sign-off checklist

That document is the activation gate for any Tier-1 Carrier contract.

10. Out of scope

• Changes to Free / Starter / Pro / Business / Enterprise tier limits.
• Any "unlimited" wording — banned for the same reason it was banned in the Carrier PR.
• Publishing a hard Carrier SLA number — that is Task 3 of #53 and gated on two consecutive clean runs of §4 plus a signed-off §9 deployment.