Staging stand-up for Ultra @ 100k load-test (issue #53 Task 1, Phase 2)¶

Concrete, copy-pasteable runbook to bring the staging surface to the state required by load-test-ultra-100k.md Phase 2/3. Authored ahead of cost approval so the actual run is one approval away.

This document is dry-run by default. Every command that incurs cost is gated by DRY_RUN=true; running the same command with DRY_RUN=false (and only after written cost approval) is what actually mutates infra.

0. Verified topology (snapshot 2026-05-26)¶

Resource	Name / id	Notes
ECS cluster	`telecom-tower-power` (sa-east-1)	Single cluster hosts prod and staging services.
Staging API service	`telecom-tower-power-staging`	Already exists. Task-def family `telecom-tower-power-staging`.
Prod API service	`telecom-tower-power`	Do not touch during the test.
GPU compute	AWS Batch CE `ce-sionna-rt-g5` (managed, EC2, instance types `g5`,`g4dn`, current minv=0 maxv=8 desired=8)	Workers join queue `jq-sionna-rt` via job def `sionna-rt-worker` (rev 4).
SSM parameter prefix	`/telecom-tower-power-staging/`	Test-specific secrets land here.

The numbers above were captured live on 2026-05-26 via aws ecs list-services, aws batch describe-compute-environments, and aws batch describe-job-queues. Re-verify before each run; the CE maxvCpus=8 is the hard ceiling on concurrent GPU work and must be at least what the test demands.

1. Cost ceiling estimate¶

The numbers below are upper-bound estimates for one full Phase-3 run (≤ 60 min of GPU time + ALB + RDS).

Item	Quantity	Unit cost (sa-east-1, on-demand)	Subtotal
g5.xlarge GPU instance	4 instances × 1.5 h	~ USD 1.20 / h	USD 7.20
ALB LCU (staging)	≤ 25 LCU-h	USD 0.008 / LCU-h	USD 0.20
RDS staging (db.t3.medium, only if newly started)	2 h	USD 0.082 / h	USD 0.16
S3 PUT for batch output (4 × 100k rows ≈ 200 MB)	200 MB + 400 PUTs	negligible	< USD 0.05
CloudWatch ingestion	≤ 1 GB	USD 0.67 / GB	USD 0.67
Total per full run			≈ USD 8.30

These are AWS sticker prices as of 2026-05-26 sa-east-1; confirm against the AWS pricing pages before approving. Add 25 % buffer for retries and idle ramp \u2192 budget ≈ USD 10 per attempt.

2. Pre-flight (idempotent, no cost)¶

# 1. confirm staging API task definition can be rendered with the Ultra envs
aws ecs describe-task-definition \
  --task-definition telecom-tower-power-staging \
  --region sa-east-1 \
  --query 'taskDefinition.containerDefinitions[0].{env:environment,secrets:secrets}' \
  --output json

# 2. confirm RDS staging endpoint resolves and is reachable (peek via SSM session)
aws rds describe-db-instances \
  --region sa-east-1 \
  --query 'DBInstances[?contains(DBInstanceIdentifier,`staging`)].{id:DBInstanceIdentifier,endpoint:Endpoint.Address,status:DBInstanceStatus}' \
  --output table

# 3. confirm Batch CE is healthy and at the expected ceiling
aws batch describe-compute-environments --region sa-east-1 \
  --query 'computeEnvironments[?computeEnvironmentName==`ce-sionna-rt-g5`].{state:state,status:status,minv:computeResources.minvCpus,maxv:computeResources.maxvCpus}' \
  --output table

All three must succeed before continuing.

3. Provision SSM parameters¶

# Generate a random 32-byte token (locally; never reuse a prod key)
TOKEN=$(openssl rand -base64 32 | tr -d '=+/' | head -c 40)

# Store as SecureString
aws ssm put-parameter \
  --name /telecom-tower-power-staging/ULTRA_DEMO_KEY \
  --type SecureString \
  --value "$TOKEN" \
  --region sa-east-1 \
  --overwrite

# Verify length and that the value retrieves correctly
aws ssm get-parameter \
  --name /telecom-tower-power-staging/ULTRA_DEMO_KEY \
  --with-decryption \
  --region sa-east-1 \
  --query 'Parameter.Value' --output text | wc -c
# expected: 41 (40 chars + trailing newline from the CLI)

unset TOKEN

ULTRA_DEMO_KEY is staging only. Setting it on the production task definition is a deployment policy violation \u2014 the load-test-ultra-100k.md "Safety / non-goals" section calls this out explicitly.

4. Roll the staging task definition (DRY-RUN first)¶

# Pull the current staging TD, mutate envs, write a new revision, but DO NOT
# update the service yet.
DRY_RUN=${DRY_RUN:-true}

aws ecs describe-task-definition \
  --task-definition telecom-tower-power-staging \
  --region sa-east-1 \
  --query taskDefinition \
  --output json > /tmp/td-current.json

python3 - <<'PY' > /tmp/td-next.json
import json, sys
td = json.load(open('/tmp/td-current.json'))
# Strip read-only fields before re-registering
for k in ('taskDefinitionArn','revision','status','requiresAttributes',
          'compatibilities','registeredAt','registeredBy'):
    td.pop(k, None)
# Inject Ultra envs into the api container
for c in td['containerDefinitions']:
    if c['name'] in ('api','app','telecom-tower-power','telecom-tower-power-staging'):
        env = {e['name']: e['value'] for e in c.get('environment', [])}
        env.update({
            'ENABLE_ULTRA_DEMO_KEY': 'true',
            'ENABLE_DEMO_KEYS': 'true',
            'MAX_BATCH_ROWS_ULTRA': '100000',
            'RATE_LIMIT_ULTRA': '5000',
        })
        c['environment'] = [{'name':k,'value':v} for k,v in sorted(env.items())]
        secs = {s['name']: s['valueFrom'] for s in c.get('secrets', [])}
        secs['ULTRA_DEMO_KEY'] = 'arn:aws:ssm:sa-east-1:490083271496:parameter/telecom-tower-power-staging/ULTRA_DEMO_KEY'
        c['secrets'] = [{'name':k,'valueFrom':v} for k,v in sorted(secs.items())]
json.dump(td, sys.stdout, indent=2)
PY

# Diff what's changing
diff <(jq -S '.containerDefinitions[]|{name,environment,secrets}' /tmp/td-current.json) \
     <(jq -S '.containerDefinitions[]|{name,environment,secrets}' /tmp/td-next.json) || true

if [ "$DRY_RUN" = "false" ]; then
  aws ecs register-task-definition \
    --cli-input-json file:///tmp/td-next.json \
    --region sa-east-1 \
    --query 'taskDefinition.{family:family,rev:revision}' --output table
else
  echo "DRY_RUN=true \u2014 skipping register-task-definition"
fi

After registering, deploy with circuit-breaker armed:

NEW_TD_REV=$(aws ecs describe-task-definition \
  --task-definition telecom-tower-power-staging \
  --region sa-east-1 --query 'taskDefinition.revision' --output text)

if [ "$DRY_RUN" = "false" ]; then
  aws ecs update-service \
    --cluster telecom-tower-power \
    --service telecom-tower-power-staging \
    --task-definition telecom-tower-power-staging:$NEW_TD_REV \
    --deployment-configuration 'deploymentCircuitBreaker={enable=true,rollback=true},maximumPercent=200,minimumHealthyPercent=100' \
    --force-new-deployment \
    --region sa-east-1 \
    --query 'service.{name:serviceName,td:taskDefinition,desired:desiredCount}' --output table

  aws ecs wait services-stable \
    --cluster telecom-tower-power \
    --services telecom-tower-power-staging \
    --region sa-east-1
fi

5. Smoke (cheap, validates auth + env)¶

Always run before the heavy locust profile.

export LOCUST_API_KEY=$(aws ssm get-parameter \
  --name /telecom-tower-power-staging/ULTRA_DEMO_KEY \
  --with-decryption --region sa-east-1 \
  --query 'Parameter.Value' --output text)

curl -sS -o /dev/null -w 'health=%{http_code} t=%{time_total}s\n' \
  https://api.staging.telecomtowerpower.com.br/health

# 5 users, 30 s, smoke tag only \u2014 no Ultra batches dispatched
locust -f locustfile.py \
  --host https://api.staging.telecomtowerpower.com.br \
  --headless --tags smoke -u 5 -r 1 -t 30s

Pass criteria: zero non-200, p95 < 1 s on /health and /towers/nearest.

6. Scale the GPU pool for Phase 3¶

The Batch CE currently sits at minv=0 maxv=8 desired=8. For the four-concurrent-Ultra-batch profile, the test plan in load-test-ultra-100k.md calls for 4 GPU instances active during the run. The maxvCpus=8 already covers this on g5.xlarge (4 vCPU per instance = 16 vCPU at peak; if g5.xlarge actually charges 4 vCPU per node Batch will provision 2 nodes, so we lift the ceiling to be safe):

if [ "$DRY_RUN" = "false" ]; then
  aws batch update-compute-environment \
    --compute-environment ce-sionna-rt-g5 \
    --compute-resources minvCpus=0,maxvCpus=32,desiredvCpus=0 \
    --region sa-east-1 \
    --query 'computeEnvironmentArn' --output text
fi

desiredvCpus=0 lets Batch scale up only on demand from the job queue, then back to zero after drain \u2014 critical for keeping the cost ceiling.

After the run, revert:

if [ "$DRY_RUN" = "false" ]; then
  aws batch update-compute-environment \
    --compute-environment ce-sionna-rt-g5 \
    --compute-resources minvCpus=0,maxvCpus=8,desiredvCpus=0 \
    --region sa-east-1
fi

Failing to revert leaves a paying ceiling parked. Don't skip.

7. Run Phase 3¶

Once §2–6 are green:

First-ever Phase 3 run → follow load-test-ultra-phase3-first-run.md. Use scripts/run_phase3_first_staging.sh as the driver — it wraps §6 ceiling lifts and the teardown in §8 below into an idempotent trap EXIT harness.
Subsequent runs (after one clean first-run on file) → follow load-test-ultra-100k.md § Phase 3 verbatim.

Capture all artifacts under reports/<YYYY-MM-DD>-phase3-<first|nominal>/.

8. Teardown (mandatory)¶

if [ "$DRY_RUN" = "false" ]; then
  # Roll the staging task definition back to its previous revision
  PREV_REV=$(aws ecs describe-task-definition \
    --task-definition telecom-tower-power-staging:$(($NEW_TD_REV - 1)) \
    --region sa-east-1 --query 'taskDefinition.revision' --output text)
  aws ecs update-service \
    --cluster telecom-tower-power \
    --service telecom-tower-power-staging \
    --task-definition telecom-tower-power-staging:$PREV_REV \
    --region sa-east-1

  # Delete the staging-only Ultra key
  aws ssm delete-parameter \
    --name /telecom-tower-power-staging/ULTRA_DEMO_KEY \
    --region sa-east-1

  # Revert CE ceiling (see \u00a76)
fi

Teardown verification checklist:

[ ] telecom-tower-power-staging task def latest revision has ENABLE_ULTRA_DEMO_KEY unset.
[ ] /telecom-tower-power-staging/ULTRA_DEMO_KEY returns ParameterNotFound.
[ ] ce-sionna-rt-g5 is back at maxvCpus=8 desiredvCpus=0.
[ ] No EC2 instance in the ce-sionna-rt-g5 ASG is still running.
[ ] Daily AWS cost-explorer reading for the test day matches the §1 estimate ± 25 %.

9. Out of scope¶

Touching production task definition, production CE limits, or production secrets.
Persisting ENABLE_ULTRA_DEMO_KEY=true past the test window (it is a fail-safe, but leaving it set after teardown defeats the purpose of the gate).
Skipping teardown to "just leave it ready for next time" \u2014 the on-demand g5/g4dn pricing makes a parked ceiling expensive.