Infrastructure as Code - Terraform & Terragrunt

Comprehensive guidance for infrastructure as code using Terraform and Terragrunt, from development through production deployment.

When to Use This Skill

Use this skill when:

• Writing or refactoring Terraform configurations
• Creating reusable Terraform modules
• Troubleshooting Terraform/Terragrunt errors
• Managing Terraform state
• Implementing IaC best practices
• Setting up Terragrunt project structure
• Reviewing infrastructure code
• Debugging plan/apply issues

Core Workflows

1. New Infrastructure Development

Workflow Decision Tree:

Is this reusable across environments/projects?
├─ Yes → Create a Terraform module
│   └─ See "Creating Terraform Modules" below
└─ No → Create environment-specific configuration
    └─ See "Environment Configuration" below

Creating Terraform Modules

When building reusable infrastructure:

1. Scaffold new module with script:

python3 scripts/init_module.py my-module-name

This automatically creates:

• Standard module file structure
• Template files with proper formatting
• Examples directory
• README with documentation

1. Use module template structure:

   • See assets/templates/MODULE_TEMPLATE.md for complete structure
   • Required files: main.tf, variables.tf, outputs.tf, versions.tf, README.md
   • Recommended: examples/ directory with working examples

2. Follow module best practices:

   • Single responsibility - one module, one purpose
   • Sensible defaults for optional variables
   • Complete descriptions for all variables and outputs
   • Input validation using validation blocks
   • Mark sensitive values with sensitive = true

3. Validate module:

python3 scripts/validate_module.py /path/to/module

This checks for:

• Required files present
• Variables have descriptions and types
• Outputs have descriptions
• README exists and is complete
• Naming conventions followed
• Sensitive values properly marked

1. Test module:

cd examples/complete
terraform init
terraform plan

1. Document module:
   • Use terraform-docs to auto-generate: terraform-docs markdown . > README.md
   • Include usage examples
   • Document all inputs and outputs

Key Module Patterns:

See references/best_practices.md "Module Design" section for:

• Composability patterns
• Variable organization
• Output design
• Module versioning strategies

Environment Configuration

For environment-specific infrastructure:

1. Structure by environment:

environments/
├── dev/
├── staging/
└── prod/

1. Use consistent file organization:

environment/
├── main.tf           # Resource definitions
├── variables.tf      # Variable declarations
├── terraform.tfvars  # Default values (committed)
├── secrets.auto.tfvars  # Sensitive values (.gitignore)
├── backend.tf        # State configuration
├── outputs.tf        # Output values
└── versions.tf       # Version constraints

1. Reference modules:

module "vpc" {
  source = "git::https://github.com/company/terraform-modules.git//vpc?ref=v1.2.0"
  
  name        = "${var.environment}-vpc"
  vpc_cidr    = var.vpc_cidr
  environment = var.environment
}

2. State Management & Inspection

When to inspect state:

• Before major changes
• Investigating drift
• Debugging resource issues
• Auditing infrastructure

Inspect state and check health:

python3 scripts/inspect_state.py /path/to/terraform/directory

Check for drift:

python3 scripts/inspect_state.py /path/to/terraform/directory --check-drift

The script provides:

• Resource count and types
• Backend configuration
• Provider versions
• Issues with resources (tainted, etc.)
• Drift detection (if requested)

Manual state operations:

# List all resources
terraform state list

# Show specific resource
terraform state show aws_instance.web

# Remove from state (doesn't destroy)
terraform state rm aws_instance.web

# Move/rename resource
terraform state mv aws_instance.web aws_instance.web_server

# Import existing resource
terraform import aws_instance.web i-1234567890abcdef0

State best practices: See references/best_practices.md "State Management" section for:

• Remote backend setup (S3 + DynamoDB)
• State file organization strategies
• Encryption and security
• Backup and recovery procedures

3. Standard Terraform Workflow

# 1. Initialize (first time or after module changes)
terraform init

# 2. Format code
terraform fmt -recursive

# 3. Validate syntax
terraform validate

# 4. Plan changes (always review!)
terraform plan -out=tfplan

# 5. Apply changes
terraform apply tfplan

# 6. Verify outputs
terraform output

With Terragrunt:

# Run for single module
terragrunt plan
terragrunt apply

# Run for all modules in directory tree
terragrunt run-all plan
terragrunt run-all apply

4. Troubleshooting Issues

When encountering errors:

1. Read the complete error message - Don't skip details

2. Check common issues: See references/troubleshooting.md for:

   • State lock errors
   • State drift/corruption
   • Provider authentication failures
   • Resource errors (already exists, dependency errors, timeouts)
   • Module source issues
   • Terragrunt-specific issues (dependency cycles, hooks)
   • Performance problems

3. Enable debug logging if needed:

export TF_LOG=DEBUG
export TF_LOG_PATH=terraform-debug.log
terraform plan

1. Isolate the problem:

# Test specific resource
terraform plan -target=aws_instance.web
terraform apply -target=aws_instance.web

1. Common quick fixes:

State locked:

# Verify no one else running, then:
terraform force-unlock <lock-id>

Provider cache issues:

rm -rf .terraform
terraform init -upgrade

Module cache issues:

rm -rf .terraform/modules
terraform init

5. Code Review & Quality

Before committing:

1. Format code:

terraform fmt -recursive

1. Validate syntax:

terraform validate

1. Lint with tflint:

tflint --module

1. Security scan with checkov:

checkov -d .

1. Validate modules:

python3 scripts/validate_module.py modules/vpc

1. Generate documentation:

terraform-docs markdown modules/vpc > modules/vpc/README.md

Review checklist:

• All variables have descriptions
• Sensitive values marked as sensitive
• Outputs have descriptions
• Resources follow naming conventions
• No hardcoded values (use variables)
• README is complete and current
• Examples directory exists and works
• Version constraints specified
• Security best practices followed

See references/best_practices.md for comprehensive guidelines.

Terragrunt Patterns

Project Structure

terragrunt-project/
├── terragrunt.hcl              # Root config
├── account.hcl                 # Account-level vars
├── region.hcl                  # Region-level vars
└── environments/
    ├── dev/
    │   ├── env.hcl            # Environment vars
    │   └── us-east-1/
    │       ├── vpc/
    │       │   └── terragrunt.hcl
    │       └── eks/
    │           └── terragrunt.hcl
    └── prod/
        └── us-east-1/
            ├── vpc/
            └── eks/

Dependency Management

# In eks/terragrunt.hcl
dependency "vpc" {
  config_path = "../vpc"
  
  # Mock outputs for plan/validate
  mock_outputs = {
    vpc_id         = "vpc-mock"
    subnet_ids     = ["subnet-mock"]
  }
  mock_outputs_allowed_terraform_commands = ["validate", "plan"]
}

inputs = {
  vpc_id     = dependency.vpc.outputs.vpc_id
  subnet_ids = dependency.vpc.outputs.private_subnet_ids
}

Common Patterns

See assets/templates/MODULE_TEMPLATE.md for complete Terragrunt configuration templates including:

• Root terragrunt.hcl with provider generation
• Remote state configuration
• Module-level terragrunt.hcl patterns
• Dependency handling

Reference Documentation

references/best_practices.md

Comprehensive best practices covering:

• Project Structure - Recommended directory layouts
• State Management - Remote state, locking, organization
• Module Design - Single responsibility, composability, versioning
• Variable Management - Declarations, files hierarchy, secrets
• Resource Naming - Conventions and standards
• Security Practices - Least privilege, encryption, secret management
• Testing & Validation - Tools and approaches
• CI/CD Integration - Pipeline patterns

Read this when:

• Setting up new Terraform projects
• Establishing team standards
• Designing reusable modules
• Implementing security controls
• Setting up CI/CD pipelines

references/troubleshooting.md

Detailed troubleshooting guide for:

• State Issues - Lock errors, drift, corruption
• Provider Issues - Version conflicts, authentication
• Resource Errors - Already exists, dependencies, timeouts
• Module Issues - Source not found, version conflicts
• Terragrunt Specific - Dependency cycles, hooks
• Performance Issues - Slow plans, optimization strategies

Read this when:

• Encountering specific error messages
• Investigating unexpected behavior
• Debugging failed deployments
• Performance tuning

Each issue includes:

• Symptom description
• Common causes
• Step-by-step resolution
• Prevention strategies

references/cost_optimization.md

Cloud cost optimization strategies for Terraform-managed infrastructure:

• Right-Sizing Resources - Compute, database, and storage optimization
• Spot and Reserved Instances - Cost-effective instance strategies
• Storage Optimization - S3 lifecycle policies, EBS volume types
• Networking Costs - VPC endpoints, data transfer optimization
• Resource Lifecycle - Scheduled shutdown, cleanup automation
• Cost Tagging - Comprehensive tagging for cost allocation
• Monitoring and Alerts - Budget alerts, anomaly detection
• Multi-Cloud - Azure, GCP cost optimization patterns

Read this when:

• Planning infrastructure to minimize costs
• Conducting cost reviews or optimization initiatives
• Implementing auto-scaling and scheduling
• Setting up cost monitoring and alerts
• Designing cost-effective architectures

CI/CD Workflows

Ready-to-use CI/CD pipeline templates in assets/workflows/:

github-actions-terraform.yml

Complete GitHub Actions workflow including:

• Terraform validation and formatting checks
• TFLint linting
• Checkov security scanning
• Terraform plan on PRs with comment posting
• Terraform apply on main branch with approval
• OIDC authentication support

github-actions-terragrunt.yml

Terragrunt-specific workflow featuring:

• Changed module detection
• Multi-module parallel planning
• Run-all commands
• Dependency-aware apply ordering
• Manual workflow dispatch with environment selection

gitlab-ci-terraform.yml

GitLab CI/CD pipeline with:

• Multi-stage pipeline (validate, lint, security, plan, apply)
• Artifact management
• Manual deployment gates
• Multi-environment configuration examples

Use these templates as starting points for your CI/CD pipelines. Customize based on your:

• Cloud provider and authentication method
• Repository structure
• Team approval workflows
• Environment promotion strategy

Scripts

init_module.py

Scaffolds a new Terraform module with proper structure and template files.

Usage:

# Create module in current directory
python3 scripts/init_module.py my-vpc

# Create in specific path
python3 scripts/init_module.py my-vpc --path ./modules

# Get JSON output
python3 scripts/init_module.py my-vpc --json

Creates:

• main.tf - Resource definitions with TODO placeholders
• variables.tf - Input variables with validation examples
• outputs.tf - Output values with descriptions
• versions.tf - Terraform and provider version constraints
• README.md - Module documentation template
• examples/complete/ - Complete usage example

Use when:

• Starting a new Terraform module
• Ensuring consistent module structure across team
• Quickly bootstrapping module development
• Teaching module best practices

inspect_state.py

Comprehensive state inspection and health check.

Usage:

# Basic inspection
python3 scripts/inspect_state.py /path/to/terraform

# Include drift detection
python3 scripts/inspect_state.py /path/to/terraform --check-drift

Provides:

• State health status
• Resource counts and types
• Provider versions
• Backend configuration
• Resource issues (tainted, etc.)
• Configuration drift detection (optional)
• Actionable recommendations

Use when:

• Before major infrastructure changes
• Investigating resource issues
• Auditing infrastructure state
• Detecting configuration drift

validate_module.py

Validates Terraform modules against best practices.

Usage:

python3 scripts/validate_module.py /path/to/module

Checks:

• Required files present (main.tf, variables.tf, outputs.tf)
• Variable descriptions and types
• Output descriptions
• Sensitive value handling
• README completeness
• Version constraints
• Example configurations
• Naming conventions
• Hard-coded values that should be variables

Returns:

• Issues (must fix)
• Warnings (should fix)
• Suggestions (consider)

Use when:

• Creating new modules
• Reviewing module code
• Before releasing module versions
• Establishing quality standards

Assets

templates/MODULE_TEMPLATE.md

Complete Terraform module template including:

• File-by-file structure and examples
• main.tf patterns
• variables.tf with validation
• outputs.tf best practices
• versions.tf constraints
• README.md template
• Example usage configurations
• Terragrunt configuration templates

Use this when:

• Creating new modules from scratch
• Standardizing module structure
• Onboarding team members
• Establishing module conventions

Quick Reference

Essential Commands

# Initialize
terraform init
terraform init -upgrade  # Update providers

# Validate
terraform validate
terraform fmt -recursive

# Plan
terraform plan
terraform plan -out=tfplan

# Apply
terraform apply
terraform apply tfplan
terraform apply -auto-approve  # CI/CD only

# State
terraform state list
terraform state show <resource>
terraform state rm <resource>
terraform state mv <old> <new>

# Import
terraform import <resource_address> <resource_id>

# Destroy
terraform destroy
terraform destroy -target=<resource>

# Outputs
terraform output
terraform output <output_name>

Terragrunt Commands

# Single module
terragrunt init
terragrunt plan
terragrunt apply

# All modules
terragrunt run-all plan
terragrunt run-all apply
terragrunt run-all destroy

# With specific modules
terragrunt run-all apply --terragrunt-include-dir vpc --terragrunt-include-dir eks

Best Practices Summary

Always:

• Use remote state with locking
• Plan before apply (review changes)
• Pin Terraform and provider versions
• Use modules for reusable components
• Mark sensitive values as sensitive
• Document everything
• Test in non-production first

Never:

• Commit secrets or credentials
• Manually edit state files
• Use root AWS credentials
• Skip code review for production changes
• Deploy without testing
• Ignore security scan findings

Key Principles:

• Infrastructure as code (everything in version control)
• DRY (Don't Repeat Yourself) - use modules
• Immutable infrastructure
• Environment parity (dev/staging/prod similar)
• Security by default
• Document for future you