Phase 3 — First Run Execution Checklist (issue #53 Task 1)

Opinionated, cost-bounded checklist for the first Ultra @ 100k Phase 3 run. Designed to fail loudly, cheaply, and reversibly. Once we have one clean run on file, subsequent runs follow load-test-ultra-100k.md §Phase 3 verbatim with no special gating.

Driver script: scripts/run_phase3_first_staging.sh. The script enforces the core automation gates (health, CE/job checks, stage execution, teardown); checklist items that depend on business context remain operator-owned.

Why this exists

The generic Phase 3 plan jumps straight to 4 concurrent 100k batches on 4 GPU workers. For a first-ever execution that is too many independent failure modes at once. This checklist walks a 3-stage ramp (A → B → C) on shared infrastructure (staging API + Batch CE), with explicit go/no-go gates so a single bad signal aborts the run before the budget is spent.

---

Hard guarantees before any AWS mutation

These must all be true. If any is missing, stop.

• [ ] Written cost approval for ≥ USD 20 for this run, linked in the run log entry below (the §1 estimate is ~$10 + buffer; first run carries a tighter floor due to teardown-failure tail risk).
• [ ] On-call ack: name + start time + ETA logged in #ops (or equivalent). The on-call MUST stay reachable for the full window plus 30 min teardown buffer.
• [ ] Production side empty of paying customers in real time: confirm via GET /admin/active_sessions (or Grafana → Live Sessions) that no Ultra/Business customer has a batch running. The shared ce-sionna-rt-g5 CE is the production GPU pool; if a real customer batch is in flight the test will starve it.
• [ ] Staging API is healthy (current task-def, /health returns 200):

  aws ecs describe-services --cluster telecom-tower-power \
    --services telecom-tower-power-staging --region sa-east-1 \
    --query 'services[0].{td:taskDefinition,r:runningCount,d:desiredCount,deps:length(deployments)}' \
    --output table
  curl -sS https://api.staging.telecomtowerpower.com.br/health
  

• [ ] Batch CE quiescent: desiredvCpus == 0 AND no running EC2 instance in its ASG. Anything else means a prior run was not torn down — abort and clean up before proceeding.
• [ ] AWS account-level vCPU quota for g family ≥ 16. (aws service-quotas get-service-quota --service-code ec2 --quota-code L-3819A6DF --region sa-east-1.) The shared production deploy uses 6 of the Fargate quota already; the GPU pool runs on a separate quota — confirm both.
• [ ] Most recent staging task-def has the §Phase 2 envs (ENABLE_ULTRA_DEMO_KEY=true, MAX_BATCH_ROWS_ULTRA=100000, etc.) and the corresponding SSM parameter exists. staging-gpu-standup.md §3–4 are the source of truth for those steps.

run_phase3_first_staging.sh precheck enforces the infrastructure/runtime items in this list; business/process acknowledgements remain manual go/no-go items.

---

Ramp-up sequence

Three stages, in order, with a hard pause between each. Do not collapse stages even if Stage A succeeds in 90 seconds. The pause is the abort window.

Stage	Workers	Concurrent 100k batches	Locust rows	Wall ETA	$ ceiling	Purpose
A	1 (maxvCpus=4)	0	1 × 10 000	8 min	$2	Validates the whole pipeline (auth → submit → enqueue → GPU pull → result → callback). Failure here means the staging task-def is wrong, not the workload.
B	1 (maxvCpus=4)	1	1 × 100 000	25 min	$5	First real 100k. Validates the actual target on a quiescent pool. Failure here is workload-level (GPU OOM, queue starvation, RDS-saturated).
C	2 (maxvCpus=8)	2	2 × 100 000 concurrent	30 min	$8	First multi-tenant pressure. Anything that only breaks under concurrency (Redis lock contention, SQS visibility-timeout, ALB target slots) surfaces here.

A clean run = A pass + B pass + C pass. Anything else: stop, write the run log, hand off to follow-up issue.

The 4-worker / 4-concurrent case from load-test-ultra-100k.md §Phase 3 is the second run, scheduled only after this checklist exits clean.

---

Terminal layout

Four terminals (or four tmux panes). Keep them visible side-by-side; do not multiplex.

Pane	Purpose	Persistent command
T1 — driver	Submit batches via locust. Single command per stage, foreground.	(commands below per stage)
T2 — pool watch	Batch CE + job queue + workers.	watch -n 5 'aws batch describe-compute-environments --compute-environments ce-sionna-rt-g5 --region sa-east-1 --query "computeEnvironments[0].{state:state,desired:computeResources.desiredvCpus,min:computeResources.minvCpus,max:computeResources.maxvCpus}" --output table && aws batch list-jobs --job-queue jq-sionna-rt --job-status RUNNING --region sa-east-1 --query "jobSummaryList[].{name:jobName,started:startedAt,status:status}" --output table'
T3 — service watch	ALB target health, ECS staging deployments, RDS connections.	watch -n 10 '/tmp/phase3-svc-watch.sh' (script generated by run_phase3_first_staging.sh precheck).
T4 — cost + abort meter	Live operator notes panel for "should we abort?" decisions based on the threshold table below.	tail -f /tmp/phase3-abort-meter.log

The driver script writes the T3 helper script into /tmp/ during precheck and initializes /tmp/phase3-abort-meter.log for T4; if those files don't exist, precheck was skipped — go back.

---

Abort thresholds

Hit any of these for the sustained window and the operator should halt the current stage and skip subsequent stages.

Metric	Source	Threshold	Sustained for	Action
Locust failure rate	locust *_stats.csv	> 1.0 % overall or any 30 s window > 5 %	30 s	abort stage + skip remaining stages
p99 latency on /batch_reports polling	locust HTML	> 2400 s (133 % of SLA)	one batch	abort stage + skip remaining stages
Batch job FAILED count on jq-sionna-rt	aws batch list-jobs --job-status FAILED	> 0 attributable to this run	immediate	abort stage + capture job logs + skip remaining stages
GPU pool warm-up time	T2 watch — desiredvCpus rising	> 8 min from first SUBMITTED to first RUNNING	8 min	abort stage; investigate quota / instance availability
ALB 5xx on staging target group	CloudWatch HTTPCode_Target_5XX_Count	> 5 / min	60 s	abort stage + dump logs
RDS connections on shared cluster	CloudWatch DatabaseConnections	> 80 % of max_connections	60 s	abort stage immediately — this is the only metric that risks production impact
ElastiCache (Redis) CPU	CloudWatch EngineCPUUtilization	> 75 %	60 s	abort stage
SQS ApproximateAgeOfOldestMessage on Sionna queue	CloudWatch	> 600 s (10 min)	one sample	abort stage — workers can't keep up
Live burn rate (extrapolated $)	abort meter	> 1.5× stage $ ceiling	one sample	abort stage

Abort = pkill -INT -P $$ locust (graceful) + trap exit teardown_guard triggered by the driver script. If the script is wedged, run bash scripts/run_phase3_first_staging.sh emergency_teardown manually — it is idempotent.

---

Stage A — 1 worker × 10 000 rows

# T1 driver (foreground; the script wraps this)
export LOCUST_API_KEY="$(aws ssm get-parameter \
  --name /telecom-tower-power-staging/ULTRA_DEMO_KEY \
  --with-decryption --query Parameter.Value --output text --region sa-east-1)"
export LOCUST_ULTRA_ROWS=10000
export LOCUST_ULTRA_DEADLINE_S=300

mkdir -p reports/$(date +%F)-phase3-first/stage-a
locust -f locustfile.py \
  --host https://api.staging.telecomtowerpower.com.br \
  --headless --tags ultra -u 1 -r 1 -t 10m \
  --csv reports/$(date +%F)-phase3-first/stage-a/ultra \
  --html reports/$(date +%F)-phase3-first/stage-a/ultra.html

Exit criteria (all required):

• locust HTML reports 0 failures (or < 1 %)
• a single Batch job ran in SUCCEEDED state for the duration
• p99 < 300 s
• No abort threshold tripped

Go/no-go pause: 5 minutes minimum. Eyeball T2/T3/T4. Confirm CE returned to desired=0 (auto-scale-down). Only then proceed.

---

Stage B — 1 worker × 100 000 rows

export LOCUST_ULTRA_ROWS=100000
export LOCUST_ULTRA_DEADLINE_S=1800

mkdir -p reports/$(date +%F)-phase3-first/stage-b
locust -f locustfile.py \
  --host https://api.staging.telecomtowerpower.com.br \
  --headless --tags ultra -u 1 -r 1 -t 35m \
  --csv reports/$(date +%F)-phase3-first/stage-b/ultra \
  --html reports/$(date +%F)-phase3-first/stage-b/ultra.html

Exit criteria (all required):

• 1 batch SUCCEEDED end-to-end
• p99 < 1800 s (Ultra SLA)
• failure rate ≤ 0.05 %
• GPU memory peak < 90 % of instance (CloudWatch GPUMemoryUtilization)
• No abort threshold tripped

Go/no-go pause: 10 minutes. CE must scale back to desired=0 before Stage C begins. If it hasn't, abort the run (parked GPU = burning budget).

---

Stage C — 2 workers × 2 concurrent 100 000 rows

Bumps the CE ceiling, then runs two locust drivers in parallel (T1 + a fifth pane T1B), each driving one batch.

# pre-stage: lift CE ceiling
aws batch update-compute-environment \
  --compute-environment ce-sionna-rt-g5 \
  --compute-resources minvCpus=0,maxvCpus=8,desiredvCpus=0 \
  --region sa-east-1

# T1 driver (one Ultra batch)
locust -f locustfile.py --host https://api.staging.telecomtowerpower.com.br \
  --headless --tags ultra -u 1 -r 1 -t 35m \
  --csv reports/$(date +%F)-phase3-first/stage-c/ultra-a \
  --html reports/$(date +%F)-phase3-first/stage-c/ultra-a.html &

# T1B driver (second Ultra batch, started 60 s later to expose race conditions)
sleep 60
locust -f locustfile.py --host https://api.staging.telecomtowerpower.com.br \
  --headless --tags ultra -u 1 -r 1 -t 35m \
  --csv reports/$(date +%F)-phase3-first/stage-c/ultra-b \
  --html reports/$(date +%F)-phase3-first/stage-c/ultra-b.html &

wait

Exit criteria (all required):

• 2 batches SUCCEEDED, both within 1800 s
• combined failure rate ≤ 0.05 %
• ALB 5xx total < 10
• RDS connections never crossed 80 %
• No abort threshold tripped

Stage C is the gate for Task 3 of #53 (raising the public Ultra ceiling). A passing run does not flip the env immediately; the decision gate in load-test-ultra-100k.md requires three consecutive clean runs across a 7-day window.

---

Mandatory teardown

Triggered by the driver script's trap EXIT — runs whether the stages passed, failed, or the script was Ctrl-C'd. Manual fallback:

bash scripts/run_phase3_first_staging.sh emergency_teardown

Teardown does, in order:

1. aws batch update-compute-environment ... --compute-resources minvCpus=0,maxvCpus=8,desiredvCpus=0 (revert CE ceiling).
2. Roll staging ECS service back to the previous task-def revision (so ENABLE_ULTRA_DEMO_KEY is unset).
3. Delete /telecom-tower-power-staging/ULTRA_DEMO_KEY from SSM.
4. Verify checklist (same as staging-gpu-standup.md §8).
5. Tail of last 50 events on the staging ECS service for operator auditability.

On DRY_RUN=false, required teardown calls are treated as hard requirements; failures keep teardown in failed state and should be triaged before any next run.

---

Artifact collection

Everything goes under reports/<YYYY-MM-DD>-phase3-first/ and gets committed to a follow-up PR chore(53): phase3 first run artifacts <date>. Required contents:

reports/2026-06-XX-phase3-first/
├── stage-a/
│   ├── ultra_stats.csv
│   ├── ultra_stats_history.csv
│   ├── ultra_failures.csv         # may be empty
│   └── ultra.html
├── stage-b/   (same shape)
├── stage-c/
│   ├── ultra-a*.csv  ultra-a.html
│   └── ultra-b*.csv  ultra-b.html
├── cloudwatch/
│   ├── gpu_util.json              # `aws cloudwatch get-metric-data` export
│   ├── rds_connections.json
│   ├── alb_5xx.json
│   └── sqs_age.json
├── batch-jobs.json                # `aws batch describe-jobs` for every job id
├── teardown-verify.txt            # output of the verify block in teardown
├── abort-meter.log                # full T4 log
└── RUN_LOG.md                     # decision summary; copy into load-test-ultra-100k.md

CloudWatch JSON exports are operator-collected artifacts for now (the driver script does not auto-export them yet).

---

Run log entry template

After teardown, append to load-test-ultra-100k.md §Run log (verbatim block — copy from RUN_LOG.md produced by the script):

### <YYYY-MM-DD> — first Phase 3 run

- Operator: <name>
- On-call: <name>
- Cost approval: <link>
- Staging task-def revision used: telecom-tower-power-staging:<n>
- Stages completed: A=<pass|fail>, B=<pass|fail>, C=<pass|fail>
- Worst p99 across stages: <s> (SLA: 1800 s)
- Worst failure rate: <%>
- Abort thresholds tripped: <list, or "none">
- Cost actual (AWS Cost Explorer, +2 days lag): USD <amount>
- Artifacts: docs-site/docs/operations/load-test-reports/<date>/
- Decision: <hold at 50k publicly | escalate to second run | raise ceiling>
- Follow-up issues opened: <#nnn list>

If any abort tripped, also open a follow-up issue scoped to that root cause. Do not retry the run on the same day; the follow-up needs code/config change first, otherwise we burn money on the same failure.

---

Out of scope for the first run

• Touching production task-def, production CE limits, or any production secret.
• Skipping any stage even if the previous one was instantaneous.
• Setting ENABLE_ULTRA_DEMO_KEY=true on the prod task-def "for parity" — see load-test-ultra-100k.md §Safety.
• Persisting the SSM ULTRA_DEMO_KEY past teardown.
• Combining the first run with any other test (drive-test smoke, GA validation, etc.).