Skip to content

Carrier 24/7 NOC runbook

Operational artifact for issue #53 — Task 2. This runbook is the activation gate for Tier-1 Carrier customers: no Carrier contract may be countersigned until every item in §6 is signed off.

Status: template authored, awaiting first Carrier contract for live execution. The Alertmanager wiring in §3 is shipped but inert until NOC_CARRIER_WEBHOOK_URL is set.

Task 1 (Ultra @ 100k load-test) is tracked in load-test-ultra-runbook.md. Task 3 (publishing a hard Carrier SLA number) is deliberately deferred until two consecutive clean runs of Task 1 and a signed §6 in this document.

1. Scope

Carrier-tier customers receive a negotiated, contract-specific SLA. To honor that SLA operationally (not just in print), the response surface below must be in place before the contract takes effect:

  • 24/7 paging with a named on-call rotation (internal or contracted vendor).
  • RTO / RPO targets per component, with restore drills exercised in the last 12 months.
  • Documented maintenance and change-window policy.
  • Per-customer dashboard and alert routing scoped by tenant tier.
  • Disaster-recovery walk-through executed with the customer.

2. RTO / RPO per component

Component RTO target RPO target Recovery mechanism
API (telecom-tower-power ECS service) ≤ 5 min 0 ALB health checks + ECS service auto-recovery; deployment circuit breaker rolls back failed rollouts.
GPU batch worker pool (batch_gpu_interference_worker.py) ≤ 15 min 0 Jobs are idempotent via SQS visibility-timeout; a failed batch restarts from job_id without data loss.
Postgres (RDS telecom-tower-power-db) ≤ 30 min ≤ 5 min Multi-AZ failover; PITR enabled with 5-minute log shipping. Nightly logical dumps to S3 with set -o pipefail + size sanity check.
Redis (Elasticache) ≤ 10 min ≤ 60 s Cluster mode with one replica per shard. Cache loss is degraded service, not data loss; physics-only path remains live.
Object storage (S3 batch output) ≤ 5 min 0 Cross-region replication; versioning + MFA delete enabled.
Audit log (KMS-encrypted) ≤ 30 min 0 Immutable append log; restored from S3 + KMS-encrypted backups. No data-loss tolerance.

Restore drills must be executed quarterly and recorded under docs-site/docs/operations/drills/drill-YYYY-MM-DD-<scenario-slug>.md. Each drill report has its own file; an index lives in docs-site/docs/operations/drills/README.md. The report must include: scenario, start/end timestamps, RTO/RPO actuals vs target, operator names, deviations from procedure, and follow-up actions filed as GitHub issues. A Carrier customer's per-customer copy (§5) references the two most recent drills relevant to that customer.

3. Alertmanager → NOC routing

The wiring is already present in alertmanager/entrypoint.sh (route + receiver block) — the runtime config is rendered from it at container start; there is no static reference copy in the repo (#2192). Activation is gated on environment variables — leaving both unset keeps the receiver inert and changes nothing for non-Carrier tenants.

Variable Effect
NOC_CARRIER_WEBHOOK_URL When set to an https:// URL, emits a noc-carrier route+receiver. Without it, the route is omitted and the receiver is not generated.
CARRIER_NOC_EMAIL_TO Optional second channel. Without it, only the webhook fires.

Pages are routed when an alert carries tenant_tier="carrier". The route uses group_wait: 0s, group_interval: 1m, repeat_interval: 1h, with continue: true so the existing severity-based routes still fire in parallel.

Rules expected to label tenant_tier="carrier" (non-exhaustive — see prometheus_alert_rules.yml for the canonical set):

  • ApiHttp5xxRateHigh (severity: page, category: api)
  • BatchJobBacklogHigh (severity: page, category: batch)
  • RDSConnectionsNearMax (severity: page, category: data)
  • ResidualEngineFallbackRateHigh (severity: warning, category: ml-ops)
  • ResidualBoosterOversized (severity: warning, category: ml-ops)

3.1 Activating the NOC receiver

Activation checklist — every box must be ticked, in order, before the carrier sign-off in §6 is valid.

  • [ ] 1. Vendor / rotation provisioned. PagerDuty, Opsgenie, Squadcast, or internal scheduler. Inbound incident-create webhook URL captured (NOT a chat-channel webhook).
  • [ ] 2. Secret installed. NOC_CARRIER_WEBHOOK_URL set in the runtime .env or via Docker secret /run/secrets/noc_carrier_webhook_url. Optional CARRIER_NOC_EMAIL_TO set if a second channel is required by the contract.
  • [ ] 3. Alertmanager re-rendered. Restart the Alertmanager container and confirm the route+receiver appeared: 
    docker compose restart alertmanager
docker compose exec alertmanager amtool config routes show \
  --config.file=/etc/alertmanager/alertmanager.yml | grep -A2 noc-carrier
docker compose exec alertmanager amtool config receivers \
  --config.file=/etc/alertmanager/alertmanager.yml | grep noc-carrier
    Both greps must print a non-empty match. If they don't, the secret was not picked up — check docker compose config and the entrypoint render step before continuing.
  • [ ] 4. Synthetic page fired and acknowledged. 
    docker compose exec alertmanager amtool alert add \
  tenant_tier=carrier severity=page alertname=NOCSmokeTest \
  --alertmanager.url=http://localhost:9093 \
  --annotation=summary="Carrier NOC activation smoke"
    Confirm the incident appears in the vendor UI within RTO and is acknowledged by the primary on-call. Record the ack timestamp in the per-customer copy of §5.
  • [ ] 5. Resolve the smoke. 
    docker compose exec alertmanager amtool alert add \
  tenant_tier=carrier severity=page alertname=NOCSmokeTest \
  --end=$(date -u +%FT%TZ)
    Vendor UI must transition the incident to resolved. A page that cannot be auto-resolved is an activation failure, not a pass.
  • [ ] 6. Quarterly drill scheduled. Calendar entry created for the next §3.2 drill cadence; entry references this checklist.

3.2 On-call rotation mechanics

The noc-carrier route is only as good as the rotation behind the webhook. The rotation MUST satisfy:

  • Tooling. Either PagerDuty, Opsgenie, Squadcast, or an internal rotation backed by a scheduler with deterministic on-call lookup (no chat-channel "who has the pager today?"). The vendor URL configured in NOC_CARRIER_WEBHOOK_URL must hit the rotation's incident-create endpoint, not a generic channel webhook.
  • Coverage. 24/7/365 with two independent layers (primary + backup). A single named human is not a rotation — vacation/illness MUST be covered by automatic schedule override, not ad-hoc Slack pings.
  • Acknowledgement SLO. Primary acks within 5 min; otherwise auto-escalates to backup at 5 min and to engineering at 15 min. These thresholds are encoded in the vendor's escalation policy, not in this document.
  • Handoff. Daily 09:00 BRT handoff with a written one-liner (incidents touched, dashboards trending, open follow-ups). Stored in the rotation tool's shift-notes feature or, if absent, in a dedicated channel with daily timestamped messages.
  • Drill cadence. Once per quarter the on-call answers a synthetic page (see §3.1 step 4); the timestamp and ack latency are recorded under drills/.
  • Vendor / internal switch. Switching providers requires updating NOC_CARRIER_WEBHOOK_URL, re-running the §3.1 smoke, and notifying each active Carrier customer in writing ≥ 7 days in advance.

4. Maintenance windows

  • Weekly — Tue 03:00–05:00 BRT — minor deploys, image rebuilds, restart of stateless workers. No schema migration.
  • Monthly — first Sun 02:00–06:00 BRT — schema migrations, PITR snapshot taken at T−15 min, RDS minor-version patching.
  • Quarterly — published 30 days in advance — DR drills (RDS PITR restore, region failover dry-run, S3 replication validation).
  • Emergency change — must satisfy all of:
  • Two engineers signed off on the PR (one is the deployer).
  • NOC notified ≥ 15 min before deploy with a one-line summary and rollback plan.
  • Post-change dashboard review within 30 min of deploy; signed off in the change log.
  • Circuit-breaker rollback armed (deploymentCircuitBreaker={enable=true,rollback=true}).

4.1 Approval and communication SLAs

Window type Approver (TTP side) Customer notice Channel
Weekly (Tue 03:00 BRT) Engineering lead None per-occurrence; calendar publicly listed in this runbook n/a
Monthly (first Sun 02:00 BRT) Engineering lead + account owner ≥ 14 calendar days before window start Email + per-customer calendar invite
Quarterly (DR drill) Engineering lead + account owner ≥ 30 calendar days before window start Email + calendar invite + Carrier-tier Grafana banner
Emergency Two engineers (one is deployer) + on-call account owner notified ≥ 15 min before deploy with rollback plan; full post-mortem within 5 business days Email + NOC page

The published calendar of upcoming windows lives at maintenance-calendar.md and is the single source of truth so the customer email, calendar invite, and runbook never disagree. Cancelling a window inside its notice horizon requires the same approver who scheduled it plus a written notice to every active Carrier customer no later than 6 hours before original start.

Carrier customers receive a maintenance calendar invite 14 days in advance for the monthly window. Weekly windows are not announced individually but are listed in this runbook.

5. Carrier customer template

The extracted template lives at carrier-noc-runbook-template.md. Filled out per customer before sign-off — live copies live at docs-site/docs/operations/carrier/<customer-slug>.md (not committed to the public repo; see carrier/README.md).

The template block below is kept inline for reference; the canonical copy is the standalone template file.

# Carrier customer: <Legal Name>

- **Tenant ID:** <UUID>
- **API key prefix:** <6+ chars>
- **Tier:** carrier
- **Contract effective:** YYYY-MM-DD
- **Contract SLA (negotiated):** <e.g., 99.95% monthly availability, p99 ≤ 3 s on /analyze>
- **Negotiated batch ceiling:** <e.g., Ultra @ 100k, 4 concurrent>

## NOC contacts

| Role | Name | Phone | Email | Escalation order |
|---|---|---|---|---|
| Primary on-call | | | | 1 |
| Backup on-call | | | | 2 |
| Engineering escalation | | | | 3 |
| Account owner (TTP side) | | | | 4 |

## Alert channels

- Webhook: `<vendor URL or internal rotation endpoint>`
- Slack: `#noc-<customer-slug>`
- Email: `<noc@customer.example.com>`

## Per-tenant overrides

- AI quota override (`/memories/repo/ai-quota-per-tenant-override.md`): <value or n/a>
- Batch concurrency override: <value or n/a>
- Custom dashboard URL: <Grafana link>

## DR drill history

| Date | Scenario | Outcome | Notes |
|---|---|---|---|
| YYYY-MM-DD | RDS PITR restore to T−1 h | pass / fail | |
| YYYY-MM-DD | Region failover dry-run | pass / fail | |

## Change log

| Date | Change | Approver | Post-deploy verification |
|---|---|---|---|

6. Pre-countersign sign-off checklist

To be completed before any Carrier contract is countersigned. Each box requires an initial and date in the per-customer copy.

  • [ ] Customer-specific API key issued via key_store_db.py with tenant tier carrier.
  • [ ] Per-tenant quota override applied via PUT /admin/api-keys/{prefix}/quota (TOTP step-up required; audited as admin.tenant.ai_quota.set). Override semantics: NULL → tier default, 0 → hard block on /bedrock/*, >0 → replaces tier ai_per_month. Record the override value in §5 of the per-customer copy.
  • [ ] NOC_CARRIER_WEBHOOK_URL configured and a smoke alert fired and acknowledged within RTO (amtool alert add tenant_tier=carrier severity=page alertname=NOCSmokeTest). Record the ack timestamp in the per-customer copy.
  • [ ] On-call rotation in §3.2 verified end-to-end (vendor URL hits incident-create endpoint, primary + backup layers configured, escalation policy seeded with 5/15-min thresholds). Drill timestamp filed under drills/.
  • [ ] Customer's NOC contact list (primary + escalation) filled in §5 of the per-customer copy.
  • [ ] Latest Ultra @ 100k load-test report (load-test-ultra-100k.md §Run log and load-test-ultra-runbook.md §6) ≤ 90 days old AND meeting all SLOs for the negotiated batch ceiling (not just 50k).
  • [ ] Production safety knobs verified: ENABLE_ULTRA_DEMO_KEY is unset on the production task definition; MAX_BATCH_ROWS_ULTRA is unset (defaults to 50000) unless the customer's negotiated ceiling has been raised through a separate signed change-window. Confirm via aws ecs describe-task-definition --task-definition telecom-tower-power | jq '.taskDefinition.containerDefinitions[].environment[] | select(.name|test("ULTRA_DEMO_KEY|MAX_BATCH_ROWS_ULTRA"))'.
  • [ ] Customer counter-signed copies of DPA-LGPD.md and EULA.md.
  • [ ] DR walk-through executed with the customer in the last 12 months (RDS PITR restore + region failover dry-run); both recorded in §5 and the underlying drill reports linked from drills/.
  • [ ] Carrier-tier dashboard provisioned (per-tenant Grafana view) and shared read-only with customer.
  • [ ] Account owner (TTP side) acknowledged 24/7 NOC reachability for the next 12 months.

7. Out of scope

  • Changes to Free / Starter / Pro / Business / Enterprise tier limits.
  • Any "unlimited" wording — banned for the same reason it was banned in the Carrier PR.
  • Publishing a hard Carrier SLA number — that is Task 3 of #53 and gated on two consecutive clean Task-1 runs plus a signed-off §6 of this document.