Skip to content

deployment.md

Latest commit

 

History

History
353 lines (261 loc) · 7.75 KB

deployment.md

File metadata and controls

353 lines (261 loc) · 7.75 KB
Raw

Deployment Guide

This guide provides step-by-step instructions for deploying the portfolio data processing solution.

Prerequisites

Before deploying, ensure you have:

  1. AWS Account

    • Active AWS account with billing enabled
    • IAM user with AdministratorAccess or appropriate permissions

  2. AWS CLI

    # Install AWS CLI
pip install awscli

# Configure credentials
aws configure

  3. Terraform

    # MacOS
brew install terraform

  4. Python 3.11+

    python3 --version

Deployment Steps

1. Clone and Setup

git clone <repository_url>

# Install Python dependencies (optional, for local testing)
make install
# or
pip install -r requirements.txt

2. Configure Terraform Variables

Create a infrastructure/terraform.tfvars file:

aws_region   = "us-east-1"
environment  = "staging"
bucket_name  = "portfolio-data-staging-YOUR-UNIQUE-SUFFIX"

Important: S3 bucket names must be globally unique. Replace YOUR-UNIQUE-SUFFIX with something unique.

3. Initialize Terraform

cd infrastructure
terraform init

Expected output:

Initializing the backend...
Initializing provider plugins...
Terraform has been successfully initialized!

4. Review Infrastructure Plan

terraform plan --var-file="staging.tfvars"

This shows what resources will be created:

  • 1 S3 bucket
  • 2 Lambda functions
  • 1 DynamoDB table
  • 2 SQS queues
  • IAM roles and policies
  • CloudWatch log groups
  • S3 event notifications

5. Deploy Infrastructure

terraform apply --var-file="staging.tfvars"

Type yes when prompted to confirm.

Deployment takes approximately 2-3 minutes.

6. Save Outputs

After successful deployment, note the outputs:

terraform output

Save these values:

  • s3_bucket_name - For uploading CSV files
  • sqs_queue_url - For reading messages
  • coordinator_lambda_arn - For monitoring
  • processor_lambda_arn - For monitoring

Example:

s3_bucket_name = "staging-portfolio-data-abc123"
sqs_queue_url = "https://sqs.us-east-1.amazonaws.com/123456789/staging-portfolio-messages"

Testing the Deployment

1. Upload Test Data

Use the Makefile command:

make upload-test-data

Or manually:

BUCKET_NAME=$(terraform -chdir=infrastructure output -raw s3_bucket_name)

aws s3 cp example_data/clients_20200130.csv s3://$BUCKET_NAME/
aws s3 cp example_data/portfolios_20200130.csv s3://$BUCKET_NAME/
aws s3 cp example_data/accounts_20200130.csv s3://$BUCKET_NAME/
aws s3 cp example_data/transactions_20200130.csv s3://$BUCKET_NAME/

2. Monitor Lambda Execution

Watch the coordinator Lambda logs:

aws logs tail /aws/lambda/staging-csv-coordinator --follow

Watch the processor Lambda logs:

aws logs tail /aws/lambda/staging-csv-processor --follow

3. Verify Messages in SQS

QUEUE_URL=$(terraform -chdir=infrastructure output -raw sqs_queue_url)

aws sqs receive-message \
  --queue-url $QUEUE_URL \
  --max-number-of-messages 10 \
  --attribute-names All

Expected output includes messages like:

{
  "type": "client_message",
  "client_reference": "9e40659b-8b9f-4fc4-814b-5a7b5a23b64",
  "tax_free_allowance": 801,
  "taxes_paid": 0
}

4. Check DynamoDB Tracking Table

aws dynamodb scan \
  --table-name staging-file-tracking \
  --output table

Troubleshooting

Lambda Function Not Triggered

Symptom: Files uploaded but no messages in SQS

Solutions:

  1. Check S3 event notification configuration:

    aws s3api get-bucket-notification-configuration \
  --bucket $BUCKET_NAME

  2. Verify Lambda has S3 invoke permission:

    aws lambda get-policy \
  --function-name staging-csv-coordinator

  3. Check CloudWatch logs for errors

Terraform Apply Fails

Symptom: "Error creating S3 bucket: BucketAlreadyExists"

Solution: Change bucket_name in terraform.tfvars to a unique value

Symptom: "Error creating Lambda function: InvalidParameterValueException"

Solution: Ensure Python source files exist in src/ directory

No Messages in SQS

Symptom: Lambda executes successfully but no messages

Solutions:

  1. Check processor Lambda logs for errors
  2. Verify all 4 CSV files were uploaded for the same date
  3. Check DynamoDB table to confirm all files marked present
  4. Ensure CSV files have correct format and headers

Permission Errors

Symptom: "AccessDenied" errors in Lambda logs

Solutions:

  1. Verify IAM roles have correct policies attached
  2. Check bucket policy allows Lambda access
  3. Ensure SQS queue policy allows SendMessage

Monitoring and Maintenance

CloudWatch Dashboards

Create a CloudWatch dashboard to monitor:

  • Lambda invocation count and errors
  • Lambda duration and throttles
  • SQS message count and age
  • DynamoDB read/write capacity

Alarms

Set up CloudWatch alarms for:

# Lambda errors
aws cloudwatch put-metric-alarm \
  --alarm-name staging-processor-errors \
  --alarm-description "Alert on Lambda errors" \
  --metric-name Errors \
  --namespace AWS/Lambda \
  --statistic Sum \
  --period 300 \
  --threshold 1 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 1 \
  --dimensions Name=FunctionName,Value=staging-csv-processor

Log Retention

Logs are retained for 7 days by default. To change:

# In infrastructure/main.tf
resource "aws_cloudwatch_log_group" "processor_logs" {
  retention_in_days = 30  # Change to 30 days
}

Updating the Deployment

Update Lambda Code

  1. Modify Python files in src/
  2. Run: 
    terraform apply

Terraform detects code changes and updates Lambda functions.

Update Infrastructure

  1. Modify Terraform files
  2. Run: 
    terraform plan  # Review changes
terraform apply

Cleanup

To remove all resources and stop charges:

# Delete all objects from S3 bucket first
aws s3 rm s3://$BUCKET_NAME --recursive

# Destroy infrastructure
cd infrastructure
terraform destroy

Type yes to confirm.

Warning: This permanently deletes all data and resources.

Cost Estimation

Estimated monthly costs for moderate usage (assuming 30 days, 1 upload/day):

Service Usage Estimated Cost
Lambda 60 invocations, 1GB-sec $0.00 (Free Tier)
S3 120 files, 10MB each $0.03
SQS 120 requests $0.00 (Free Tier)
DynamoDB 120 writes, 240 reads $0.00 (Free Tier)
CloudWatch 1GB logs $0.50
Total ~$0.53/month

Note: Costs increase with:

  • Higher upload frequency
  • Larger file sizes
  • More downstream consumers reading from SQS
  • Longer log retention

Production Readiness Checklist

Before deploying to production:

  • Change environment variable to "prod"
  • Use unique bucket name with "prod" prefix
  • Enable S3 bucket logging
  • Enable AWS CloudTrail
  • Configure CloudWatch alarms
  • Set up AWS Backup for DynamoDB
  • Implement VPC endpoints for private networking
  • Add KMS encryption for S3 and SQS
  • Configure reserved Lambda concurrency
  • Set up AWS Config for compliance
  • Create runbook for incident response
  • Test disaster recovery procedures
  • Review and tighten IAM policies
  • Enable AWS GuardDuty for threat detection

Support

For issues or questions:

  1. Check CloudWatch Logs for error messages
  2. Review this deployment guide
  3. Consult the main README.md
  4. Check AWS Service Health Dashboard

Additional Resources