AWS tenant standup and rebuild (AWS only)

This skill drives a full AWS Shifter tenant from a clean (or dirty) account to a healthy, verified tenant with a live POLARIS range. It exists so a fresh session does not repeat the deploy-docs archaeology and does not re-hit the fresh-environment gotchas already fixed on dev. GCP is out of scope.

Ground rules

• All Shifter AWS resources live in us-east-2 unless the operator says otherwise.
• The canonical GitHub repository is Brad-Edwards/shifter (see AGENTS.md).
• Never add AI or Claude attribution to any commit, PR, issue, or file.
• Do not merge branches. Merges are operator-driven. Fixes land as PRs to dev.
• Run long-running Terraform and deploys in the background. A foreground tool timeout can kill a Terraform provider plugin mid-apply and corrupt progress.
• Do not stack pushes to an environment branch while its CI or deploy is in flight.
• Verify with the CLI before stating a fact. Do not fabricate on a failed step.
• Fix root causes and PR them to dev. Do not jury-rig around a real bug.

Parameters

Read these from the invocation arguments if supplied (natural language or key=value pairs). Ask the operator for any that are missing before doing anything destructive.

• profile: the AWS CLI profile to use (for example proof or aws-dev).
• region: the AWS region. Default us-east-2; confirm rather than assume.
• mode: one of
  • teardown: rip out everything, including AMIs, and re-bake fresh images.
  • preserve-amis: tear down the infrastructure but keep existing AMIs and skip the bake phase (faster restart when the images are still good).

Derive the environment from the profile and confirm it with the operator: proof maps to Terraform env proof and deploy input aws-proof; aws-dev or dev maps to Terraform env dev and deploy input aws-dev. The deploy.yml workflow_dispatch environment options are aws-dev and aws-proof.

Authoritative sources (read these first)

Read these before acting. They are the real process. Do not follow docs/technical/platform_infrastructure/manual-deployment.md as the primary path; it is a legacy fallback, not the deploy pipeline.

• docs/dev/deploy-secrets.md: the single authoritative checklist of every secret a fresh AWS environment needs, plus the fresh-account bootstrap order.
• docs/dev/aws-teardown-runbook.md: the teardown procedure and the tag-scoped account-recovery sweep.
• docs/technical/dev/secrets.md: where runtime secrets live and how tfvars reach Terraform (real values come from TF_VARS_<ENV>_* GitHub secrets rendered to a gitignored local.auto.tfvars, not committed tfvars).
• .github/workflows/deploy.yml: the real deploy path (workflow_dispatch with an environment input; the branch you dispatch from is the code that deploys).
• .github/workflows/packer.yml: the image bake pipeline.

Phase 0: preflight

1. Confirm the account identity and record the account id: aws --profile <profile> sts get-caller-identity.
2. Confirm the local overlay exists for each stack: platform/terraform/environments/<env>/local.auto.tfvars (core), .../<env>/range/local.auto.tfvars, and .../<env>/portal/local.auto.tfvars. These are gitignored real values. If any is missing, the values must already be in the matching TF_VARS_<ENV>_* GitHub secret. Keep the overlay and the secret in step with scripts/sync-deploy-secrets.sh (it never prints contents). You may read and edit the local overlay; never paste its contents into chat, a PR, a commit, or an issue.
3. Run the deploy preflight to confirm required secrets are present: python3 scripts/bootstrap/preflight.py (see deploy-secrets.md for the required set).

Phase 1: teardown

Follow docs/dev/aws-teardown-runbook.md. Key points learned in practice:

• Destroy all three Terraform roots: portal, then range, then core (platform/terraform/environments/<env>/{portal,range,}). The portal root declares cloud_provider; pass -var="cloud_provider=aws" there. The range and core roots do not declare it, and Terraform 1.15+ errors on an undeclared -var, so do not pass it to those.
• Empty versioned buckets (logs, ECR, Terraform state) before destroy, including all object versions and delete markers, or the destroy fails on non-empty buckets.
• The account-recovery sweep is tag-scoped (Project=shifter plus the environment tag). Run it to catch anything Terraform missed.
• If mode=preserve-amis: before teardown, capture the AMI ids you will reuse (ec2_ami_id, ctfd_ami_id in the portal overlay; the /shifter/ami/* SSM parameters; polaris-vm and polaris-dc; shifter/packer/dc-amis.json). Do not deregister those AMIs, and skip Phase 3.

Phase 2: bootstrap and secrets

1. Bootstrap the fresh account bits (OIDC role, state bucket, global IAM): python3 scripts/bootstrap/deploy.py bootstrap --env <env>. This sets AWS_ROLE_ARN_<ENV> and TF_INFRA_STATE_BUCKET_<ENV>.
2. Ensure the runners are registered per the runbook.
3. Ensure the deploy secrets exist. Assume the same starting secrets as before. Push local overlays with scripts/sync-deploy-secrets.sh --env <env> (add --stack config --shifter-config <path> for the SHIFTER_CONFIG_* payload).

Phase 3: image bakes (skip when preserving AMIs)

Bake base images (kali, ubuntu, windows, dc) and the POLARIS images (polaris-vm, polaris-dc) via packer.yml.

• Bakes need per-account PACKER_VERIFY_{SUBNET,SG,INSTANCE_PROFILE}_ID_<ENV> variables plus a <env>.pkrvars.hcl, both derived from the freshly rebuilt VPC. There is a post-build DNS validation gate.
• After baking, update ec2_ami_id and ctfd_ami_id in the portal overlay, update shifter/packer/dc-amis.json, update the /shifter/ami/* SSM parameters, then re-sync the portal secret with sync-deploy-secrets.sh.
• Known design issue: baking must not depend on a branch merge (tracked in issue #1680). If a bake seems to require merging first, stop and flag it.

Phase 4: deploy

Deploy via the real pipeline, dispatched from the code branch you want live (usually dev for a standard standup):

gh workflow run deploy.yml --repo Brad-Edwards/shifter \
  -f environment=aws-<env> --ref <code-branch>

The stages run Core, then Range, then Engine, then Platform. Monitor the run to a terminal state. On the first deploy of a fresh account, watch the portal migration step: if it fails because there is no in-service ASG instance yet, or because the runtime DB user does not exist, run the migration by hand over SSM as the DB master:

scripts/portal-deploy/deploy_portal.sh --migrate-only   # via SSM on the portal EC2

The entrypoint switches to the RDS-IAM runtime user (portal_runtime), which is created by a migration, so on a fresh DB the migrate must run with DB_IAM_AUTH_RUNTIME=false (the --migrate-only path sets this).

Phase 5: health verification

• ALB target health: proof-portal-tg (portal app) and proof-portal-guacamole should be all healthy (substitute the env prefix).
• ACM certificate validated for the portal domain.
• Portal responds: /health/ returns 200 and / returns 200. If Cloudflare DNS is not yet pointed at the new ALB, verify against the raw ALB with a host header or curl --resolve <domain>:443:<alb-ip>.

Phase 6: base-range smoke

Run the post-deploy smoke against a base range to prove provisioning works end to end. See scripts/post_deploy_smoke/README.md for the entrypoint. This exercises subnet allocation, guest launch, and bootstrap. Do this before POLARIS to avoid range contention.

Phase 7: POLARIS range walkthrough

Provision a POLARIS range and confirm it reaches READY.

1. Provision via the portal Django shell over SSM. Get the portal instance, docker exec the portal container, re-export the app environment from /proc/1/environ (the shell needs DJANGO_SECRET_KEY and friends), then call cms_create_range(user, 'polaris', {}, False).
2. Poll the range status to READY or FAILED. The provisioner launches as an ECS task off the latest ACTIVE <env>-portal-pulumi-provisioner task definition.
3. On READY, confirm: two guests running (kali attacker on polaris-vm, dc01 on polaris-dc), the DC promoted (boreas.local), the full container stack up on the kali host (17 containers including a14-kali, the Bedrock agent), and the agent verify checks pass (identity via credential_process plus regional STS, and a Bedrock invoke-model).

POLARIS Bedrock requirements:

• The account must have Bedrock model access for the configured agent models. Verify with an actual bedrock-runtime invoke-model, not list-foundation-models (the latter shows region availability, not account access). See reference_proof_bedrock_model_access in operator notes: the proof account can invoke only sonnet-4-6; all haiku variants are Marketplace gated.
• The aws_polaris_agent_* variables must be set in the portal overlay and the TF_VARS_<ENV>_PORTAL secret. Empty values make POLARIS fail at variable build. Point both the main and small models at a model the account can invoke.

Phase 8: report and DNS handoff

Report the outcome and the DNS records the operator must set in Cloudflare (DNS is external; the agent does not hold those credentials):

• <env>.shifter.keplerops.com CNAME to the portal ALB DNS name.
• polaris.keplerops.com A record to the CTFd EIP.

Note that a running POLARIS range costs money until it is destroyed. State clearly whether the range was left running.

Known pitfalls (already fixed on dev; verify, do not re-hit)

These were the fresh-environment failures found during the proof rebuild. They are fixed on dev. On a fresh account, confirm the fix is present rather than rediscovering the failure.

• Provisioner image must copy the whole shared package, not a subset, or range provisioning fails with ModuleNotFoundError: shared.range_instantiation_policy (issue #1678). Broke all range provisioning in every environment.
• The CI permissions boundary must carry the per-range POLARIS agent carve-out and the IAM read permissions the provider needs after role creation (iam:ListAttachedRolePolicies, iam:ListInstanceProfilesForRole) (issue #1683, fixes #1681).
• First-deploy migrate race: the deploy waits for an in-service ASG instance before migrating (issue #1677). If it still fails, run the manual migrate from Phase 4.
• POLARIS agent verify runs host-side because the a14-kali container has no aws CLI, and the container aws-config needs sts_regional_endpoints = regional plus an STS host pin (issue #1688).
• Environment parity: a new import-time environment variable must be added to user_data.sh, the portal SSM parameters, and deploy_portal.sh common env, or a fresh AWS deploy fails at migrate.
• Range guests can fail SSM when the VPC DNS resolver is not reachable; see the guest DNS operator note.

Progress and traceability

• Track phases with a task list and give the operator running status. Do not run the whole thing silently in a subagent.
• File an issue for every new bug found, fix the root cause, and PR it to dev.
• Keep a durable scratch record of parameters, decisions, resource ids, and PRs so the run survives a context reset.