Study Notes for AWS Certified DevOps Engineer – Professional (DOP-C02)
- AWS Certified DevOps Engineer Professional Exam Guide -- high level overview
- Toward the Cloud - Study Guide -- more indepth DOP-C02 overview (compared official guide)
- Tutorials Dojo DOP-C02 Study Materials -- useful overview with sample questions and scenarios
- AWS Devops Engineer Professional -- including really good summary on CloudFormation -- very through study notes in general -- albeit for v1 of Exam (not v2)
- Adrian Cantrill Study Guide -- paid guide / excellent -- best paid guide out there ~70hrs
- AWS Introduction to DevOps -- Solid overview / good intro to DOP-C02
- AWS DevOps Guidance -- not very technical -- but good leadership style doc on driving adoption (DevOps) . Must read White Paper in General for Driving DevOps Adoption.
- Infrastructure as Code - AWS Whitepaper -- must read !
- Practicing CI/CD on AWS -- solid CI/CD fundamentals overview
- DevOps on AWS: Best Practices
- AWS Well-Architected Framework
- AWS Deployment Pipeline Reference Architecture -- good tips / best practices on geenral CI/CD pipeline design
- AWS Automating safe, hands-off deployments
- AWS CodeBuild build spec reference
- Running Containerized Microservices on AWS
- Microservices on AWS
- Disaster Recovery of Workloads on AWS: Recovery in the Cloud
- AWS Security White Paper -- dated but good overview
- AWS The CI/CD litmus test: Is your pipeline fully CI/CD?
- AWS EventBridge
Building a solid CI/CD pipeline is crucial for DevOps success. It ensures fast, reliable, and scalable software delivery across environments. AWS offers best practices and guidelines to help teams implement effective pipelines and optimize processes.
- Begin with a minimum viable pipeline to kick off continuous delivery.
- Start simple: implement processes like code style checks or a single unit test.
- Ensure fast feedback during early stages with lightweight builds that handle pushes and basic tests efficiently.
- In the commit stage, focus on minimizing delays to speed up development.
-
Consistency:
- Build and package artifacts once: Create a single, consistent artifact for deployment.
- Store artifacts in a registry with metadata, ready for deployment to any environment.
- Ensure versioning and traceability of the pipeline output, linking it to source code and business requirements.
- Promote artifacts through environments automatically—avoid rebuilding.
-
Small Batches:
- Encourage small, frequent releases by minimizing manual processes.
- Automate testing and deployment to eliminate toil and speed up feedback loops.
- Avoid long-lived branches—favor trunk-based development where changes are merged and deployed daily.
-
Measured Approach:
- Provide real-time metrics for areas like:
- Code quality (e.g., static analysis).
- Speed (e.g., deployment frequency, lead time).
- Reliability (e.g., failure rate, time to recover).
- Use dashboards to track metrics and optimize processes.
- When instrumentation is unavailable, use surveys to estimate metrics.
- Provide real-time metrics for areas like:
- Average time for changes to reach production: Identify delays between commits and deployments.
- Average build time: Ensure builds are fast and reliable to speed up feedback.
- Pipeline stage time: Measure how long it takes for changes to progress through each pipeline stage.
- Avoid long-lived branches and complicated merges; favor short-lived branches or trunk-based development.
- Replace manual tests with automated testing frameworks to streamline testing.
- Reduce manual approvals; implement automated gates based on test results and metrics.
- Automate code reviews and security checks to avoid delays and errors.
- Pipeline Expectations: Consider models like the AWS Deployment Pipeline Reference Architecture. Don't reinvent the wheel if you don't have to.
- Evolutionary Design: Requirements change—avoid locking in detailed designs early. Adopt a flexible approach where the pipeline evolves with the project. There's no such thing as a perfect pipeline.
- In large organizations, teams may require different tools to solve unique challenges.
- Encourage teams to use the best tools for their specific needs (e.g., different databases, programming languages).
- Balance decentralized governance with central standards to ensure flexibility without sacrificing alignment.
-
Anti-patterns for Leader Sponsorship:
- Diluted leadership focus: Ensure the single-threaded leader responsible for DevOps isn’t juggling too many priorities. DevOps adoption requires dedicated attention.
- Forcing DevOps adoption: Leaders should support, not enforce, DevOps. Imposing DevOps hierarchically leads to resistance. Create a collaborative environment for smoother transitions.
-
Pair Programming:
- Pair programming serves as an alternative to PR-based reviews. It allows earlier code reviews, reducing the time taken to identify and fix issues.
- Teams may opt for trunk-based development, using pair programming and post-commit processes instead of PRs, depending on team preference.
-
Sequence build actions strategically for prompt feedback. (i.e. you don't need one slow sequentual pipeline).
- Run long-duration tasks earlier in parallel with faster tasks to prevent bottlenecks.
- Continuously review and adjust action sequencing to optimize for early detection of issues.
-
Refine pipelines with metrics:
- Use key metrics like deployment frequency, lead time for changes, and failure rates to optimize the continuous integration process.
- Embed observability in your pipeline—transform logs into actionable metrics and make them visible to all teams for proactive monitoring and improvement.
- Pipeline stability: Track the percentage of build failures due to infrastructure issues, not code errors. Regularly audit the pipeline to ensure stable builds.
- Mean Time to Build (MTTB): Track the average time for a complete successful build. Improve by reducing dependencies, leveraging caching, and running tasks in parallel.
- Run tests in parallel: For faster feedback, run multiple tests concurrently, especially as the number of test cases grows in microservice architectures.
- Over-indexing on coverage metrics: Don’t rely solely on high code coverage.
- Excessive data collection: Avoid over-instrumentation (especially logging). Prioritize relevant data and implement retention policies to reduce costs.
- Noisy and unactionable alarms/errors: Continuously evaluate alarms to ensure they are actionable and significant. Mute false positives to avoid alert fatigue. (see Broken Window Fallacy)
- Practicing Continuous Integration and Continuous Delivery on AWS
- Running Containerized Microservices on AWS
- What is DevOps? AWS DevOps Page
- AWS Deployment Pipeline Reference Architecture
- AWS DevOps Guidance
This version incorporates your additional insights, organizes it more tightly, and uses the right level of detail for quick reference. Let me know if any further adjustments are needed!
- AWS CodePipeline: Automates the end-to-end release process.
- AWS CodeBuild: Compiles source code, runs tests, and produces software packages.
- AWS CodeCommit: Managed Git service for version control.
- AWS CodeDeploy: Automates deployments to Amazon EC2, Lambda, and on-premises servers.
- AWS Secrets Manager / AWS SSM Parameter Store: Manages secrets and configuration data in CI/CD pipelines.
-
Continuous Integration and Continuous Deployment (CI/CD):
- Automate pipelines to integrate builds, tests, and deployments.
- Learn to set up and manage complex CI/CD workflows across environments and accounts.
-
Automated Testing:
- Integrate unit, integration, and security tests in CI/CD.
- Automate testing phases in pipelines using tools like CodeBuild.
-
Artifact Management:
- Create and manage artifact repositories (AWS CodeArtifact, ECR).
- Automate artifact storage and distribution (Amazon S3, EC2 Image Builder).
-
Deployment Strategies:
- Understand deployment methods like blue/green, canary, and rolling updates.
- Implement these strategies for EC2, Lambda, and container-based deployments (ECS, EKS).
- AWS CloudFormation: Primary IaC tool for defining and managing AWS infrastructure.
- AWS CDK: High-level framework for defining AWS infrastructure with code.
- AWS Systems Manager: Centralized management service for patching and configuration.
- AWS OpsWorks: Configuration management using Chef or Puppet.
-
Infrastructure as Code (IaC):
- Learn to write and manage CloudFormation templates and AWS CDK.
- Deploy reusable infrastructure components with AWS Service Catalog.
-
Configuration Management:
- Automate patching, configuration, and compliance using AWS Systems Manager and AWS OpsWorks.
-
Multi-Account Setup:
- Master account management services like AWS Organizations and Control Tower.
- Implement governance controls using Service Control Policies (SCPs) and AWS Config Rules.
- Amazon Route 53: Highly available and scalable DNS web service.
- Amazon RDS / Aurora: Multi-AZ database deployments for high availability.
- Amazon CloudFront: Global content delivery with automatic failover.
- Elastic Load Balancing (ELB): Load balancing across multiple AZs and Regions.
-
High Availability:
- Design multi-AZ and multi-Region architectures for EC2, RDS, DynamoDB, and S3.
- Understand load balancing and failover techniques.
-
Scalability:
- Implement auto-scaling for EC2, ECS, and RDS based on CloudWatch metrics.
- Design distributed, loosely coupled architectures.
-
Disaster Recovery:
- Define RTO and RPO requirements.
- Master backup and recovery techniques (AWS Backup, cross-Region replication).
- Amazon CloudWatch: Metrics and logs monitoring for AWS resources.
- AWS X-Ray: Distributed tracing for microservices and serverless applications.
- AWS CloudTrail: Tracks API calls and account activity for audit purposes.
- AWS Config: Tracks AWS resource configuration changes.
-
Log Aggregation:
- Collect, aggregate, and store logs using CloudWatch Logs, Kinesis Data Firehose, and Amazon S3.
- Implement log retention and lifecycle policies for log data.
-
Real-Time Monitoring:
- Create CloudWatch metrics, alarms, and dashboards for proactive monitoring.
- Set up anomaly detection using CloudWatch alarms and custom metrics.
-
Event-Driven Monitoring:
- Configure EventBridge for event-driven architectures and alert notifications.
- Integrate auto-scaling events with CloudWatch and SNS for dynamic scaling.
- Amazon EventBridge: Event-driven architecture for capturing and processing events.
- AWS Health Dashboard: Monitors AWS service health and notifies of incidents.
- Amazon SNS: Notifies teams about critical events and incidents.
- Amazon SQS: Message queuing for decoupling systems and handling event workflows.
-
Event-Driven Response:
- Capture and process AWS events using EventBridge, CloudWatch, and Lambda.
- Build automation workflows for incident response using Step Functions and SNS.
-
Incident Management:
- Monitor system health using AWS Health and CloudWatch metrics.
- Implement auto-remediation for common incidents using AWS Systems Manager and Lambda.
-
Root Cause Analysis:
- Use CloudWatch and X-Ray for debugging application failures.
- Perform post-incident analysis to identify and fix root causes in CI/CD pipelines or auto-scaling configurations.
- AWS Identity and Access Management (IAM): Centralized control for user permissions and access.
- AWS Secrets Manager: Securely manage and rotate secrets.
- AWS Key Management Service (KMS): Encryption and key management service for securing data.
- Amazon GuardDuty: Threat detection and continuous security monitoring.
-
Identity and Access Management (IAM):
- Implement fine-grained access control using IAM roles, policies, and permission boundaries.
- Master identity federation and multi-factor authentication (MFA) for secure access.
-
Security Automation:
- Automate security best practices using AWS Config, AWS Security Hub, and AWS Systems Manager.
- Apply encryption and data protection using KMS, ACM, and AWS WAF.
-
Compliance and Auditing:
- Enable auditing and monitoring with CloudTrail, Config, and GuardDuty.
- Ensure compliance by continuously monitoring configurations using AWS Config rules and CloudFormation drift detection.
1. AWS Control Tower
-
Purpose: Simplifies setting up and governing a secure, multi-account AWS environment based on best practices.
-
Core Concepts:
- Landing Zone: A multi-account environment that Control Tower sets up using AWS best practices for security and compliance.
- Guardrails: Automated rules that help enforce compliance with organizational policies.
- Preventive guardrails: Prevent actions that could cause non-compliance.
- Detective guardrails: Continuously monitor for compliance violations.
- Account Factory: Automates the creation and configuration of new accounts with predefined settings.
-
Use Cases:
- Enterprises needing a quick and secure way to establish a multi-account environment.
- Ensuring that each AWS account follows the same security, logging, and monitoring policies.
-
Key Features:
- Automates deployment of AWS Organizations, AWS Config, AWS CloudTrail, and Service Control Policies (SCPs).
- Integrated with AWS Organizations to manage account hierarchy and permissions.
- Centralized dashboard for account management and monitoring compliance.
-
Best Practices:
- Organizational Units (OUs): Use OUs to group accounts based on function (e.g., production, development) and apply different policies.
- Enable all available guardrails to enforce security best practices and compliance (e.g., ensuring S3 bucket encryption, preventing root access).
- Utilize Account Factory for creating standardized accounts automatically with predefined baseline configurations.
2. AWS Organizations
-
Purpose: Enables centralized management of multiple AWS accounts under a single organization.
-
Core Concepts:
- Organization: The root account and child accounts within an AWS organization.
- Organizational Units (OUs): Group AWS accounts into OUs to apply different management policies (e.g., separate development from production environments).
- Service Control Policies (SCPs): Policies that specify the maximum allowed permissions for accounts within an OU. SCPs don’t grant permissions but restrict what accounts can do.
-
Use Cases:
- Simplifying multi-account management by grouping accounts into OUs.
- Enforcing global security policies across accounts using SCPs.
- Managing billing for all accounts in one place through consolidated billing.
-
Key Features:
- Consolidated Billing: View and pay invoices for all AWS accounts under the organization.
- SCPs: Enforce policies at the organizational level (e.g., block users from launching resources in specific regions, or enforce encryption policies).
- Cross-Account Role Delegation: Enable administrators to manage multiple accounts using IAM roles with cross-account permissions.
-
Best Practices:
- Design your OU structure based on business units, environments, or security requirements (e.g., production, sandbox).
- Use SCPs for least privilege enforcement across accounts. For example, apply SCPs to deny root account access or limit permissions for certain AWS services across all accounts.
- Integrate with CloudFormation StackSets to deploy resources across multiple accounts/regions from a central point.
3. AWS CloudFormation StackSets
-
Purpose: Enables deployment of CloudFormation stacks across multiple AWS accounts and regions from a single, central management account.
-
Core Concepts:
- StackSet: A set of CloudFormation stacks that can be deployed across multiple accounts and regions.
- Administrator Account: The account that creates and manages the StackSet.
- Target Accounts: Accounts where StackSet stacks are deployed.
- Execution Role: IAM role that StackSets assumes in target accounts to perform stack actions.
-
Use Cases:
- Deploying infrastructure consistently across multiple AWS accounts and regions (e.g., setting up networking, IAM roles, or logging configurations).
- Centralized management of global resources, such as AWS Config rules, CloudTrail, or logging solutions like VPC Flow Logs.
-
Key Features:
- Cross-Account Stacks: Deploy and update stacks across multiple accounts in an AWS organization.
- Cross-Region Stacks: Automatically deploy stacks to multiple regions from a single template.
- Drift Detection: Check whether resources in your stacks have been modified outside of CloudFormation.
-
Best Practices:
- Use StackSets for large-scale, multi-account environments where infrastructure consistency is critical.
- Combine with SCPs and AWS Config to enforce security and compliance policies across regions and accounts.
- Ensure proper role delegation by setting up IAM Execution Roles in each target account to enable cross-account deployments.
1. AWS Config
-
Purpose: AWS Config continuously monitors and records the configurations of your AWS resources, enabling compliance checks and governance.
-
Core Concepts:
- Configuration Recorder: Tracks configuration changes to AWS resources.
- Config Rules: Evaluate whether AWS resources comply with the desired configuration.
- AWS Managed Rules: Predefined rules for common scenarios (e.g., S3 bucket must be encrypted).
- Custom Rules: User-defined rules that use AWS Lambda to enforce specific business logic or security requirements.
- Conformance Packs: Group of Config Rules that adhere to a specific compliance framework (e.g., PCI-DSS, CIS Benchmarks).
-
Use Cases:
- Auditing and ensuring compliance with security or operational best practices.
- Enforcing resource configuration policies (e.g., ensuring EC2 instances are within certain instance types, or RDS instances are multi-AZ).
- Tracking changes to resource configurations over time for troubleshooting or compliance reports.
-
Key Features:
- Snapshot of Resource Configurations: Provides detailed history of configuration changes.
- Drift Detection: Detects when resource configurations deviate from your defined compliance policies.
- Multi-account Support: You can enable AWS Config across all accounts in an AWS Organization.
-
Best Practices:
- Use Conformance Packs to quickly implement industry-standard compliance frameworks (e.g., NIST, CIS).
- Combine with SCPs and CloudTrail to ensure a holistic security and compliance strategy.
- Enable Config Aggregators to centralize compliance reporting for multiple accounts and regions.
2. Service Control Policies (SCPs)
-
Purpose: SCPs are policies used in AWS Organizations to define permission boundaries for AWS accounts. They restrict the maximum set of permissions allowed for accounts or OUs but do not grant permissions themselves.
-
Core Concepts:
- Deny/Allow Actions: SCPs can explicitly allow or deny specific actions across accounts.
- Inherited Policies: Accounts within OUs inherit SCPs from parent OUs and the root organization.
-
Use Cases:
- Enforce security best practices globally (e.g., prevent usage of certain services or restrict operations in certain regions).
- Block actions that are potentially harmful, like deleting critical infrastructure or disabling logging.
-
Key Features:
- Granular Control: SCPs apply organization-wide and can enforce strict security and operational policies.
- Preventive Measure: SCPs can prevent users or roles from bypassing critical security controls (e.g., prevent root account access or unapproved regions).
-
Best Practices:
- Apply SCPs at the OU level to manage permissions consistently across similar accounts.
- Deny usage of services that are not needed in your organization (e.g., prevent creating unencrypted S3 buckets or running EC2 in unapproved regions).
- Combine with AWS IAM for more granular permission management (IAM focuses on user-level permissions, SCPs focus on account-level boundaries).
3. Governance Using CloudFormation
-
Purpose: CloudFormation is key to enforcing standardized infrastructure and governance by defining AWS resource configurations as code.
-
Key Governance Features:
- CloudFormation Stack Policies: Control what actions (e.g., delete, update) can be performed on resources in a stack to prevent accidental deletions.
- Modules and Nested Stacks: Allow for reusable templates, enabling the enforcement of consistent security and operational policies.
- Drift Detection: Identifies discrepancies between the actual configuration of AWS resources and the template-defined configuration.
-
Best Practices:
- Use CloudFormation StackSets to enforce resource compliance (e.g., ensuring all accounts have logging enabled, IAM roles created with least privilege access).
- Incorporate security best practices into CloudFormation templates (e.g., require encryption for EBS, S3, RDS).
- Version-control templates and integrate them with CI/CD pipelines for safe, automated deployments.
- AWS Control Tower simplifies multi-account management with pre-configured security and compliance guardrails.
- AWS Organizations and SCPs provide governance at scale by enforcing organizational-wide policies and permissions across multiple accounts.
- CloudFormation StackSets enables consistent resource deployment across multiple accounts/regions from a central location.
- AWS Config ensures compliance by continuously monitoring AWS resource configurations and enforcing security best practices.
- Combine SCPs, **AWS
Config**, and CloudFormation StackSets to ensure consistent, compliant, and secure infrastructure across multi-account environments.
-
AWS CodeDeploy automates the deployment of applications to a variety of targets, including:
- EC2 Instances
- On-Premises Servers
- Lambda Functions
- Amazon ECS Services
-
Deployment Models:
- In-Place Deployments: Deploys the new application version on existing instances.
- Blue/Green Deployments: Spins up a new environment (green) and switches traffic over from the old environment (blue).
| Deployment Type | Description | Pros | Cons |
|---|---|---|---|
| In-Place | Updates the existing instances with the new revision. | Simple, no new infrastructure needed. | Risk of downtime, possible inconsistencies. |
| Blue/Green | New instances are launched with the new revision and traffic is shifted to these instances when ready. | Zero downtime, easy rollback. | Higher cost, requires managing two environments. |
- Definition: The AppSpec file defines the deployment instructions and hooks for CodeDeploy.
- Supported Formats: YAML or JSON.
- Location: Must be located in the root directory of the application bundle (in S3, GitHub, etc.).
Key Sections:
- version: Specifies the version of the AppSpec file.
- os: The target operating system (
linuxorwindows). - files: Describes the source and destination paths for files to be copied during deployment.
- hooks: Defines lifecycle event hooks to run scripts or commands during the deployment.
| Hook | Description |
|---|---|
| ApplicationStop | Stops the currently running application before the deployment. |
| BeforeInstall | Pre-installation steps like taking backups or stopping services. |
| AfterInstall | Post-installation steps like configuring the environment or setting permissions. |
| ApplicationStart | Starts the application after the new version is installed. |
| ValidateService | Verifies that the application is running correctly (e.g., smoke tests). |
| BeforeAllowTraffic | Runs before traffic is routed to the new version, typically used for final checks or validations. |
| AfterAllowTraffic | Runs after traffic is routed to the new version, used for cleanup or monitoring tasks. |
- CodeDeploy can automatically roll back to the previous revision if:
- Deployment fails (e.g., lifecycle event script fails).
- Instances fail health checks.
- You configure CloudWatch Alarms to trigger a rollback.
- OneAtATime: Deploys to one instance at a time.
- HalfAtATime: Deploys to 50% of instances at a time.
- AllAtOnce: Deploys to all instances at once (fast, but risky for production).
- For EC2/On-Premises Deployments, CodeDeploy requires an agent running on the instance.
- The agent communicates with the CodeDeploy service and executes the deployment instructions as per the AppSpec file.
- CloudWatch Logs: Capture logs for each lifecycle event and deployment.
- SNS Notifications: Can notify via email/SMS if a deployment fails or succeeds.
- CloudWatch Alarms: Use alarms to trigger rollbacks or notifications based on deployment health metrics.
- Ensure that:
- CodeDeploy Service Role has the correct permissions to interact with other AWS services (e.g., EC2, Lambda).
- EC2 Role allows the instance to communicate with the CodeDeploy service (for pulling the deployment bundle, running scripts, etc.).
version: 0.0
os: linux
files:
- source: /app/source/
destination: /var/www/myapp
- source: /config/settings.conf
destination: /etc/myapp/settings.conf
hooks:
ApplicationStop:
- location: scripts/stop_server.sh
timeout: 300
runas: root
BeforeInstall:
- location: scripts/install_dependencies.sh
timeout: 600
runas: root
AfterInstall:
- location: scripts/change_permissions.sh
timeout: 600
runas: root
ApplicationStart:
- location: scripts/start_server.sh
timeout: 300
runas: root
ValidateService:
- location: scripts/validate_service.sh
timeout: 300
runas: root- version: Specifies the version of the AppSpec file (
0.0is the required value). - os: Specifies the operating system (in this case,
linux). - files: Specifies the source paths in the deployment bundle and the destination paths on the target instance.
- source: Refers to a directory or file inside the build artifact (e.g., uploaded to S3).
- destination: Specifies the path where the files should be copied on the target instance.
- hooks: Defines the deployment lifecycle event hooks and associated scripts.
- Example: The
ApplicationStophook runs thestop_server.shscript before stopping the current version of the app.
- Example: The
- AppSpec File Errors: Ensure proper syntax (YAML/JSON) and that paths and hooks are correct.
- Agent Issues: Ensure the CodeDeploy agent is installed and running on EC2/On-Premises instances.
- Permission Errors: Verify IAM roles are correctly assigned to CodeDeploy and EC2 instances.
- Script Timeouts: Make sure your lifecycle event scripts are correctly configured with reasonable timeouts.
Here's a shorter, optimized CodeBuild Study Sheet in Markdown with a concise sample buildspec.yml that covers all the key concepts, without redundancy.
- AWS CodeBuild is a fully managed service for compiling source code, running tests, and producing artifacts.
- It integrates with CodePipeline, S3, CodeDeploy, and other AWS services for automating CI/CD workflows.
- Source: Code is pulled from sources like CodeCommit, GitHub, Bitbucket, or S3.
- Buildspec File: YAML file defining the build phases, environment variables, artifacts, caching, and reports.
- Artifacts: Output from the build process, stored in S3 or used by other services.
- Reports: Test reports (e.g., JUnit, Cucumber) that can be viewed in the CodeBuild console.
- Caching: Reuse dependencies across builds (e.g.,
npm,maven), reducing build time. - Batch Builds: Run multiple builds in parallel or sequentially, useful for multi-environment or multi-version workflows.
- version: Specifies the buildspec version (
0.2recommended). - env: Defines environment variables, secrets, and proxies.
- Secrets Manager: Injects sensitive data (e.g.,
DB_PASSWORD) from AWS Secrets Manager. - Parameter Store: Retrieves parameters (e.g.,
API_KEY) from AWS Systems Manager Parameter Store.
- Secrets Manager: Injects sensitive data (e.g.,
- phases: The build lifecycle is split into these phases:
- install: Install necessary tools or dependencies.
- pre_build: Setup before the build (e.g., Docker login).
- build: Main build step (e.g., compiling code, running tests).
- post_build: Tasks after build (e.g., packaging artifacts).
- finally: Runs even if earlier phases fail (used for cleanup).
- artifacts: Specifies files to store after build (e.g., upload to S3).
- cache: Defines paths to cache to speed up subsequent builds (e.g.,
maven,npm). - reports: Generate and upload test results for analysis.
- IAM Permissions: CodeBuild needs roles to access services (e.g., S3, Secrets Manager, CodeCommit).
- Exported Variables: Environment variables can be defined in one phase and reused in subsequent phases using the
exportcommand. - Proxy Support: If the build is behind a proxy, configure HTTP/HTTPS proxies via environment variables.
- Batch Builds: Execute multiple builds in parallel or sequentially within a single project, ideal for complex workflows.
version: 0.2
# Environment variables, including secrets and parameter store.
env:
variables:
ENVIRONMENT: production # Custom environment variable.
secrets-manager:
DB_PASSWORD: my-database-password # Securely fetch from AWS Secrets Manager.
parameter-store:
API_KEY: /myapp/api/key # Fetch from Parameter Store.
proxy:
HTTP_PROXY: http://proxy.example.com:8080 # Set proxy for build environment.
# Build lifecycle phases.
phases:
install:
commands:
- echo "Installing dependencies..."
- apt-get update && apt-get install -y awscli
pre_build:
commands:
- echo "Logging into Docker..."
- aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin <your_ecr_url>
build:
commands:
- echo "Building the project..."
- export BUILD_ID=$(uuidgen) # Exported variable for reuse.
- mvn clean install # Example: Build Java Maven project.
post_build:
commands:
- echo "Packaging build..."
- zip -r myapp.zip target/ # Package build output.
finally:
commands:
- echo "Cleaning up resources..."
- aws s3 rm s3://my-bucket/temp --recursive # Example cleanup action.
# Artifacts to upload after the build.
artifacts:
files:
- myapp.zip # Package to upload.
base-directory: target # Where the output files are stored.
# Reports configuration for test results.
reports:
test-reports:
files:
- '**/*' # Report files pattern.
base-directory: test-reports # Directory with test results.
# Caching configuration to speed up future builds.
cache:
paths:
- /root/.m2/**/* # Cache Maven dependencies.
- /root/.npm/**/* # Cache npm modules.- Phases: Organize your build process into logical steps with install, pre_build, build, post_build, and finally.
- Environment Variables: Leverage Secrets Manager and Parameter Store for sensitive data, and use exported variables across phases.
- Caching: Speed up subsequent builds by caching dependencies.
- Artifacts & Reports: Define what gets saved (e.g., build outputs) and generate test reports for visibility.
Here’s a CodePipeline Cheat Sheet with key concepts relevant for the AWS Certified DevOps Engineer – Professional (DOP-C02) exam, along with an example CodePipeline YAML template that includes all the key concepts.
- AWS CodePipeline is a fully managed continuous integration and continuous delivery (CI/CD) service.
- Automates the build, test, and deploy phases of your release process.
- Integrates with multiple services: CodeBuild, CodeDeploy, S3, Lambda, ECS, CloudFormation, etc.
- A pipeline consists of stages, and each stage has actions.
- Source Stage: Fetches code or artifacts (e.g., from S3, CodeCommit, GitHub).
- Build Stage: Compiles, runs tests, and produces artifacts (typically using CodeBuild).
- Deploy Stage: Deploys applications (e.g., using CodeDeploy, CloudFormation, ECS).
- Approval Stage: Manual approvals before proceeding to the next stage.
- Actions are tasks within a stage. Common action types:
- Source: Pulls code from a repository (e.g., S3, GitHub, CodeCommit).
- Build: Executes a build job (e.g., CodeBuild, Jenkins).
- Test: Runs test suites.
- Deploy: Deploys artifacts (e.g., CodeDeploy, CloudFormation, ECS).
- Invoke: Calls Lambda functions.
- Approval: Requires manual intervention to approve or reject.
- Artifacts are the output from one action that can be passed to another action (e.g., build output from CodeBuild used by a CodeDeploy action).
- Pipelines can be triggered by events:
- Source Change: Automatically start when new code is pushed (e.g., to CodeCommit, GitHub).
- Manual Start: Users can manually start a pipeline.
- Schedule: Start the pipeline at specific intervals.
- Cross-account roles allow CodePipeline to interact with resources in another AWS account.
- Requires setting up appropriate IAM roles with trust policies.
- Artifacts and the pipeline can be encrypted using AWS KMS (Key Management Service).
- CodePipeline retries failed actions automatically. The default behavior is:
- 1 retry for most action types.
- Pipelines can include manual approval steps to require human intervention before continuing.
- SNS notifications can be sent when an approval is pending.
- CodeDeploy and CloudFormation actions can include pre- and post-deployment lifecycle hooks.
- Example: A CodeDeploy action can trigger
BeforeInstall,AfterInstall, andValidateServicehooks.
- Example: A CodeDeploy action can trigger
- Use Multiple Stages: Break the pipeline into multiple stages to isolate steps (source, build, deploy).
- Manual Approvals: Implement approvals for critical environments (e.g., production).
- Monitoring and Alerts: Use CloudWatch Events or SNS to monitor pipeline executions.
- Cross-Account Pipelines: Use cross-account pipelines to manage deployments across multiple AWS accounts.
This template sets up a basic CI/CD pipeline with CodeCommit as the source, CodeBuild for the build process, and CodeDeploy to deploy the application.
AWSTemplateFormatVersion: '2010-09-09'
Description: A basic AWS CodePipeline setup with CodeCommit, CodeBuild, and CodeDeploy.
Resources:
# IAM Role for CodePipeline with permissions to interact with CodeCommit, CodeBuild, CodeDeploy
PipelineServiceRole:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Principal:
Service: "codepipeline.amazonaws.com"
Action: "sts:AssumeRole"
Policies:
- PolicyName: "CodePipelinePermissions"
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: "Allow"
Action:
- "codecommit:*"
- "codebuild:*"
- "codedeploy:*"
- "s3:*"
- "iam:PassRole"
Resource: "*"
# S3 Bucket to store build artifacts between stages
CodePipelineArtifactStore:
Type: "AWS::S3::Bucket"
Properties:
VersioningConfiguration:
Status: "Enabled"
# CodePipeline Definition
MyPipeline:
Type: "AWS::CodePipeline::Pipeline"
Properties:
RoleArn: !GetAtt PipelineServiceRole.Arn
ArtifactStore:
Type: S3
Location: !Ref CodePipelineArtifactStore
Stages:
# Source Stage: Pulls source code from CodeCommit
- Name: "Source"
Actions:
- Name: "SourceAction"
ActionTypeId:
Category: "Source"
Owner: "AWS"
Provider: "CodeCommit"
Version: "1"
OutputArtifacts:
- Name: "SourceOutput"
Configuration:
RepositoryName: "MyCodeRepo"
BranchName: "main"
RunOrder: 1
# Build Stage: Runs CodeBuild to compile the application
- Name: "Build"
Actions:
- Name: "BuildAction"
ActionTypeId:
Category: "Build"
Owner: "AWS"
Provider: "CodeBuild"
Version: "1"
InputArtifacts:
- Name: "SourceOutput"
OutputArtifacts:
- Name: "BuildOutput"
Configuration:
ProjectName: "MyCodeBuildProject"
RunOrder: 1
# Deploy Stage: Deploys using CodeDeploy
- Name: "Deploy"
Actions:
- Name: "DeployAction"
ActionTypeId:
Category: "Deploy"
Owner: "AWS"
Provider: "CodeDeploy"
Version: "1"
InputArtifacts:
- Name: "BuildOutput"
Configuration:
ApplicationName: "MyCodeDeployApplication"
DeploymentGroupName: "MyDeploymentGroup"
RunOrder: 1
# CodeBuild Project Definition
MyCodeBuildProject:
Type: "AWS::CodeBuild::Project"
Properties:
Name: "MyCodeBuildProject"
ServiceRole: !GetAtt PipelineServiceRole.Arn
Artifacts:
Type: "CODEPIPELINE"
Source:
Type: "CODEPIPELINE"
Environment:
ComputeType: "BUILD_GENERAL1_SMALL"
Image: "aws/codebuild/standard:5.0"
Type: "LINUX_CONTAINER"
TimeoutInMinutes: 30
# CodeDeploy Application and Deployment Group
MyCodeDeployApplication:
Type: "AWS::CodeDeploy::Application"
Properties:
ApplicationName: "MyCodeDeployApplication"
MyCodeDeployDeploymentGroup:
Type: "AWS::CodeDeploy::DeploymentGroup"
Properties:
ApplicationName: !Ref MyCodeDeployApplication
DeploymentGroupName: "MyDeploymentGroup"
ServiceRoleArn: !GetAtt PipelineServiceRole.Arn
AutoScalingGroups:
- "MyAutoScalingGroup" # Example: Deploy to EC2 ASG
DeploymentConfigName: "CodeDeployDefault.OneAtATime"
Ec2TagFilters:
- Key: "Name"
Value: "MyEC2Instance"
Type: "KEY_AND_VALUE"
Outputs:
PipelineId:
Description: "The ID of the created pipeline"
Value: !Ref MyPipeline-
Stages: The pipeline has three stages:
- Source Stage: Pulls code from a CodeCommit repository.
- Build Stage: Uses CodeBuild to build and produce the output artifact.
- Deploy Stage: Uses CodeDeploy to deploy the build artifact to EC2 instances (via Auto Scaling Group).
-
IAM Roles:
- PipelineServiceRole: Grants permissions for CodePipeline to interact with CodeCommit, CodeBuild, and CodeDeploy.
-
Artifacts:
- An S3 bucket is defined as the artifact store to hold intermediate artifacts between pipeline stages.
-
Action Configuration:
- SourceAction: Fetches the code from CodeCommit.
- BuildAction: Runs the build in CodeBuild using a project called
MyCodeBuildProject. - DeployAction: Deploys the output using CodeDeploy to a deployment group targeting an Auto Scaling Group.
-
Cross-Service Integration:
- CodePipeline integrates with **
CodeCommit**, CodeBuild, and CodeDeploy, demonstrating multi-service orchestration in a CI/CD pipeline.
- Outputs: Returns the ID of the created pipeline as an output value for reference or further automation.
- Stages and Actions: Understand how to define and configure multiple stages and their respective actions in a pipeline.
- Cross-Service Integration: Know how to integrate CodeCommit, CodeBuild, CodeDeploy, S3, and other services.
- IAM Role Setup: Ensure proper IAM permissions are granted to CodePipeline to interact with other services.
- Manual Approvals: Use approval actions to introduce manual approval stages.
- Artifact Management: Configure S3 buckets or CodeBuild for managing and storing build artifacts.
- Notifications: Use SNS for pipeline success, failure, and approval notifications.
- AWS CodeArtifact is a fully managed artifact repository service for securely storing, sharing, and publishing software packages.
- Supports popular package formats like npm, Maven, PyPI, and NuGet.
- It integrates with CI/CD tools like CodeBuild and CodePipeline to manage dependencies and package distribution in the development process.
- Domains: A domain aggregates repositories, allowing sharing of resources and policies.
- Repositories: Storage for package versions; each repository can contain multiple package formats.
- Upstream Repositories: CodeArtifact repositories can proxy other repositories like public registries (e.g., npmjs.com) to pull in external dependencies.
- Package Versioning: Supports version control for different packages, ensuring consistency in builds and deployments.
- Access Control with IAM: Define fine-grained access using IAM roles and policies. Only authorized users or services can publish, retrieve, or modify packages.
- Encryption: Packages are encrypted at rest using AWS KMS.
- Dependency Management: CodeArtifact simplifies managing dependencies for projects, particularly in complex, multi-team environments.
- Integration with CodeBuild: CodeArtifact can be directly accessed within CodeBuild projects, streamlining dependency resolution during the build process.
- Private Package Hosting: Host private versions of packages internally and control who can access them.
- Dependency Caching: Use CodeArtifact as a proxy to external repositories, reducing dependency fetch times and mitigating external repository downtimes.
- Security and Governance: Track and control access to software packages, ensuring only approved dependencies are used in production environments.
- Publish Packages:
npm publish,mvn deploy, orpip uploaddirectly from your CI/CD pipeline to CodeArtifact. - Consume Packages: Use aws codeartifact login to authenticate package managers (npm, Maven, pip) with CodeArtifact.
- Create Repository:
aws codeartifact create-repositoryto create new repositories for storing artifacts. - Associate Upstream: Link to external or public repositories using
aws codeartifact associate-external-connection.
# Authenticate npm with CodeArtifact
aws codeartifact login --tool npm --repository my-repo --domain my-domain --domain-owner 123456789012
# Add npm dependencies from CodeArtifact
npm install <package-name>
# Publish package to CodeArtifact
npm publish- Repository Management: Organize your package storage with Domains and Repositories for easy access and sharing across teams.
- Upstream Repositories: Proxy external repositories to avoid network issues and ensure faster, reliable dependency resolution.
- Access Control & Security: Use IAM policies for secure access, and encrypt packages with AWS KMS.
- Integration: Seamlessly integrates with AWS CodeBuild and CodePipeline for dependency resolution and artifact publishing in CI/CD workflows.
- AWS CloudFormation automates the creation, update, and management of AWS resources through infrastructure-as-code.
- You define resources in JSON or YAML templates.
- Parameters allow you to pass values (like environment, instance type) into the template, making it reusable and configurable.
- Pseudo Parameters are pre-defined by CloudFormation, and provide contextual information.
- Examples:
AWS::Region,AWS::AccountId,AWS::StackName.
- Examples:
- Conditions determine whether certain resources or properties are created/assigned based on parameter values or pseudo parameters.
- Intrinsic Functions allow dynamic configuration inside the template:
- Fn::Sub: String substitution for variables (e.g.,
"MyApp-${Environment}"). - Fn::Join: Concatenate values with a delimiter.
- Fn::GetAtt: Fetch attributes (like ARN or DNS) from resources.
- Fn::If: Conditional logic inside the template.
- Fn::FindInMap: Retrieve values from a
Mappingssection.
- Fn::Sub: String substitution for variables (e.g.,
- Mappings store key-value pairs used for region-specific values or configuration.
- Outputs return information after the stack is created (e.g., instance ID, public DNS).
- DeletionPolicy: Retain or create snapshots of resources before deletion.
- Use Parameters: Make templates flexible and reusable.
- Use Conditions: Dynamically create resources based on input or environment.
- Use Outputs: Share information like instance IDs and ARNs across stacks.
- Use
Mappings: For region or environment-specific configurations. - Stack Policies & Change Sets: Control updates and safely preview changes.
Here’s a single CloudFormation template in YAML that includes parameters, conditions, intrinsic functions, pseudo parameters, mappings, outputs, and more.
AWSTemplateFormatVersion: '2010-09-09'
Description: >
CloudFormation template demonstrating key concepts for AWS Certified DevOps Engineer exam.
# Parameters section defines inputs from the user.
Parameters:
Environment:
Description: "Specify the environment (prod or dev)"
Type: String
Default: dev
AllowedValues:
- prod
- dev
ConstraintDescription: "Must be either prod or dev."
InstanceType:
Description: "EC2 instance type"
Type: String
Default: t2.micro
AllowedValues:
- t2.micro
- t2.small
- t2.medium
ConstraintDescription: "Must be a valid EC2 instance type."
# Mappings section for region-specific AMI IDs.
Mappings:
RegionMap:
us-east-1:
AMI: "ami-0abcdef1234567890"
us-west-2:
AMI: "ami-0987654321abcdef0"
# Conditions section to create resources based on input values.
Conditions:
IsProdEnvironment: !Equals [!Ref Environment, "prod"]
# Resources section defines the AWS infrastructure.
Resources:
MyEC2Instance:
Type: "AWS::EC2::Instance"
Properties:
InstanceType: !Ref InstanceType # Referencing the InstanceType parameter
ImageId: !FindInMap [RegionMap, !Ref AWS::Region, AMI] # Lookup AMI based on region
SecurityGroupIds:
- !Ref MySecurityGroup # Attach the security group
Tags:
- Key: "Name"
Value: !Sub "MyInstance-${Environment}" # String substitution for dynamic name
MySecurityGroup:
Type: "AWS::EC2::SecurityGroup"
Properties:
GroupDescription: "Allow SSH access"
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 22
ToPort: 22
CidrIp: 0.0.0.0/0 # Open port 22 (SSH) to the world
ProdEC2Instance:
Condition: IsProdEnvironment # Only create this resource if the environment is "prod"
Type: "AWS::EC2::Instance"
Properties:
InstanceType: !Ref InstanceType
ImageId: !FindInMap [RegionMap, !Ref AWS::Region, AMI]
SecurityGroupIds:
- !Ref MySecurityGroup
# Outputs section returns useful information like the instance ID and public DNS.
Outputs:
EC2InstanceID:
Description: "ID of the created EC2 instance"
Value: !Ref MyEC2Instance
InstancePublicDNS:
Description: "Public DNS of the EC2 instance"
Value: !GetAtt MyEC2Instance.PublicDnsName-
Parameters:
Environment(withprodanddevoptions) andInstanceTypefor user input.
-
Mappings:
RegionMapfor region-specific AMIs (e.g., different AMIs forus-east-1andus-west-2).
-
Conditions:
IsProdEnvironmentcondition ensures that certain resources (likeProdEC2Instance) are only created in the production environment.
-
Intrinsic Functions:
!Refreferences parameter values (e.g.,InstanceType).!FindInMapretrieves region-specific values from theRegionMap.!Subperforms string substitution (e.g., dynamic naming for EC2 instances).!GetAttretrieves the Public DNS of the EC2 instance.
-
Outputs:
- Instance ID and Public DNS are returned as outputs.
-
Conditions with Resources:
ProdEC2Instanceis created only when theEnvironmentisprod, showing how to use conditions to manage infrastructure per environment.
- Intrinsic Functions: Learn how to use
!Ref,!GetAtt,!Sub,!FindInMap, and conditionals like!If. - Parameters: Make templates flexible with user input for instance types, environment, etc.
- Conditions: Dynamically create resources based on environment (e.g.,
prod,dev). - Mappings: Static region-specific or environment-specific configuration (e.g., AMIs).
- Outputs: Return useful information like instance IDs, public DNS, or ARNs.
- Pseudo Parameters: Use built-in values like
AWS::Region,AWS::AccountId, andAWS::StackName.
High Availability ensures that your system is always accessible, minimizing downtime by distributing resources across multiple failure domains (like Availability Zones).
- Availability Zones (AZs): Independent data centers within an AWS region. Spreading workloads across multiple AZs improves HA.
- Multi-AZ Deployments: Resources are deployed across multiple AZs to ensure availability in case one AZ goes down.
- Elasticity & Scalability: Auto Scaling helps to automatically adjust the number of resources (EC2 instances, for example) based on demand, ensuring that your system can handle varying traffic loads.
- Use Elastic Load Balancing (ELB) to distribute incoming traffic across multiple instances.
- Deploy RDS in Multi-AZ mode to automatically fail over to a standby replica in case of failure.
- Use Amazon S3 with versioning and cross-region replication (CRR) to improve availability and data durability.
- Ensure Stateless Services where possible, so that instances can be replaced easily without affecting application state.
- Elastic Load Balancer (ELB): Distributes incoming traffic across multiple instances.
- Amazon Route 53: Provides DNS-based routing and health checks to route traffic to healthy resources.
- Amazon RDS (Multi-AZ): Ensures a standby database instance in another AZ for failover.
- Auto Scaling: Dynamically adjusts resources to meet changing demand.
Fault Tolerance is the ability of a system to continue functioning even if some components fail. In AWS, fault tolerance is achieved by building systems with redundancy at every layer.
- Redundancy: Ensure that resources (instances, databases, etc.) have backup instances to automatically take over if a failure occurs.
- Failover Mechanisms: Automatic redirection of traffic to standby resources (via Route 53, Auto Scaling, etc.) when a failure occurs.
- Stateful Services: Requires careful planning to ensure state replication or synchronization (e.g., using Amazon DynamoDB or ElastiCache with replication).
- Use Multi-AZ setups across critical resources such as databases, application servers, and storage.
- Implement Auto Healing with Auto Scaling or Elastic Beanstalk to replace failed instances automatically.
- Use DynamoDB Global Tables to replicate data across regions, providing both availability and fault tolerance.
- Ensure that your VPC is designed with redundancy, with NAT gateways or instances in multiple AZs.
- Amazon EC2 Auto Scaling: Automatically replaces unhealthy instances.
- Amazon RDS Multi-AZ: Provides failover capability for your database.
- Route 53 Failover Routing: Directs traffic to healthy endpoints based on health checks.
- Amazon DynamoDB Global Tables: Ensures fault tolerance across multiple regions with global data replication.
Disaster Recovery (DR) focuses on how quickly and efficiently a system can recover from a significant failure or disaster, such as a regional outage. It aims to minimize RPO (Recovery Point Objective) and RTO (Recovery Time Objective).
- RPO (Recovery Point Objective): The acceptable amount of data loss in time (how much data you are willing to lose).
- RTO (Recovery Time Objective): The acceptable downtime (how quickly systems must recover after failure).
- DR Tiers: AWS supports different DR strategies based on business requirements, ranging from Backup and Restore to Hot Standby (multi-site).
| DR Strategy | Description | RPO | RTO |
|---|---|---|---|
| Backup & Restore | Regular backups are taken and stored in a different location. System is rebuilt and data is restored. | High | High |
| Pilot Light | Critical infrastructure runs in a second region at minimal capacity. In a disaster, systems are scaled up. | Moderate | Moderate |
| Warm Standby | A scaled-down version of the production environment running in another region, ready to scale up. | Low | Moderate to Low |
| Hot Standby (Multi-Site) | A fully operational duplicate environment running in a different region. Traffic is routed instantly. | Very Low | Very Low |
- Implement Cross-Region Replication (CRR) for S3, DynamoDB, RDS, and other data stores.
- Use AWS Backup to automate backups for key AWS services like EC2, RDS, DynamoDB, EFS, etc.
- Use CloudFormation or Terraform to automate environment rebuilding, reducing RTO.
- Configure Route 53 Failover Routing to switch traffic to backup resources in case of primary resource failure.
- AWS Backup: Centralized backup for AWS services like EC2, RDS, DynamoDB, and EFS.
- Amazon S3 Cross-Region Replication: Ensures data is replicated to another AWS region.
- AWS Elastic Disaster Recovery (DRS): Helps replicate on-premises or cloud resources to AWS, making recovery easier and faster.
- Amazon RDS Read Replicas: Can be promoted to a master instance in another region in case of a regional failure.
Multi-region architectures help in improving both availability and fault tolerance by spreading resources across multiple AWS regions.
- Active-Active: Resources are actively serving traffic in multiple regions. Traffic is distributed via Route 53.
- Active-Passive: Only one region is actively serving traffic, while the other regions act as failover.
- Data Replication: Use Amazon DynamoDB Global Tables, Amazon RDS with cross-region replication, and Amazon S3 CRR to replicate data across regions.
- Use Route 53 for global load balancing and failover across multiple regions.
- DynamoDB Global Tables for multi-region, globally distributed databases.
- Cross-Region VPC Peering for network connectivity between regions.
- Read Replicas for databases like RDS to ensure low-latency reads across regions.
- Amazon Route 53: Provides DNS-based global traffic routing and failover.
- Amazon DynamoDB Global Tables: Enable multi-region, globally distributed data stores.
- Amazon RDS Cross-Region Read Replicas: Create read replicas in other AWS regions for disaster recovery and read scaling.
- S3 Cross-Region Replication (CRR): Automatically replicates data to another region for durability and disaster recovery.
Proactive monitoring and alerting help maintain HA and FT, and improve recovery in the event of failure or disaster.
- Health Checks: AWS offers health checks via Route 53, ELB, and Auto Scaling to ensure resources are functioning correctly.
- CloudWatch Alarms: Set up CloudWatch alarms to monitor the health of resources and alert on anomalies.
- Event-Driven Actions: Use CloudWatch Events and AWS Lambda to automate recovery actions based on events or failures.
- Use Route 53 Health Checks to detect unhealthy resources and automatically reroute traffic.
- Configure CloudWatch Logs and CloudWatch Alarms to monitor performance and trigger actions (like scaling, failover, etc.).
- Use AWS Personal Health Dashboard to get personalized alerts and status updates on service availability and health.
- Amazon CloudWatch: For logs, metrics, alarms, and dashboarding.
- AWS CloudTrail: Logs API calls and events across AWS accounts.
- Route 53 Health Checks: Monitors the health of your endpoints to ensure only healthy resources receive traffic.
- RPO (Recovery Point Objective): Measures how much data loss can be tolerated during a disaster.
- RTO (Recovery Time Objective): Measures the time it takes to restore services after a failure.
- MTTF (Mean Time to Failure): The average time before a system component is expected to fail.
- MTTR (Mean Time to Repair): The average time to recover after a failure occurs.
- SLA (Service Level Agreement): The agreed-upon uptime and availability for a service.
| Category | AWS Service |
|---|---|
| High Availability | Elastic Load Balancer (ELB), RDS Multi-AZ, Auto Scaling, S3 CRR |
| Fault Tolerance | Route 53 Failover, DynamoDB Global Tables, Auto Scaling |
| Disaster Recovery | AWS Backup, RDS Read Replicas, Elastic Disaster Recovery (DRS) |
| Monitoring & Alerts | CloudWatch, Route 53 Health Checks, CloudTrail, Personal Health Dashboard |
| Method | Description | Advantages | Challenges | Use Cases | AWS Services (related) |
|---|---|---|---|---|---|
| Deploy in Place | Deploys new code directly onto existing instances or infrastructure. | Simple, no need for extra infrastructure. | Risk of downtime, inconsistent environments during deployment. | Smaller applications with minimal traffic. | Elastic Beanstalk, CodeDeploy (in-place) |
| Rolling Deployment | Gradually replaces instances with new ones, while keeping some of the old ones running. | Minimizes downtime, supports graceful replacement. | Complexity in managing partial deployments, slow rollout. | Medium- to large-scale applications that need some availability during deployment. | CodeDeploy, Elastic Beanstalk, Auto Scaling |
| Immutable Deployment | New instances are created with the new version of the application, and old instances are terminated after traffic is routed to the new ones. | Reduces risk of configuration drift, ensures consistency. | Requires more infrastructure, higher cost. | Highly critical applications needing consistent environments. | CodeDeploy (immutable), Auto Scaling, Elastic Beanstalk |
| Traffic Splitting | Diverts a small percentage of traffic to the new version before full rollout. | Minimizes risk, allows testing with live traffic. | Complex to implement and monitor, needs good rollback mechanisms. | Testing new features with minimal impact, A/B testing. | AWS App Mesh, Lambda Traffic Shifting, API Gateway, CodeDeploy |
| Blue/Green Deployment | Two separate environments (Blue and Green) are maintained. Traffic is switched to the new environment (Green) when ready. | Zero downtime, easy rollback. | High infrastructure cost, complex environment management. | Mission-critical systems needing zero downtime during deployment. | Elastic Beanstalk, CodeDeploy, Route 53, ELB, Auto Scaling |
Let's address your questions and update the table accordingly.
AWS OpsWorks and AWS CloudFormation are both configuration management tools, but they serve different purposes and work in distinct ways.
-
AWS OpsWorks is designed for managing application stacks and automating deployments using Chef or Puppet. It focuses more on configuration management and ensuring that servers maintain a consistent state.
-
AWS CloudFormation is an infrastructure as code tool that automates the provisioning of AWS resources. It provides a declarative way to define and manage infrastructure, such as EC2 instances, VPCs, and databases.
OpsWorks is not a direct alternative to CloudFormation; in fact, they can be complementary. You would use CloudFormation for resource provisioning and infrastructure management, while OpsWorks would be used for managing application configurations and deployments.
When it comes to storing secrets, AWS offers several services:
- AWS Secrets Manager
- AWS Systems Manager Parameter Store (SecureString)
- AWS Key Management Service (KMS)
Each service has its own use cases and pricing model:
-
AWS Secrets Manager: This is a fully managed service specifically for storing secrets such as database credentials, API keys, etc. It supports automatic rotation of secrets, integrates with RDS and other AWS services, and is generally more expensive because of its rich feature set.
-
AWS Systems Manager Parameter Store: Parameter Store can be used to store both plaintext parameters and SecureString (encrypted) parameters. It’s cheaper than Secrets Manager but does not have built-in rotation of secrets. Suitable for lightweight use cases where automatic rotation is not required.
-
AWS KMS: KMS is primarily used to create and manage encryption keys. It integrates with both Secrets Manager and Parameter Store to encrypt stored secrets. It’s more appropriate for managing encryption keys rather than directly storing secrets.
| Service | Purpose | Key Features | When to Use |
|---|---|---|---|
| AWS OpsWorks | Configuration management service with Chef and Puppet support. | - Managed Chef/Puppet environments. - Automates deployments and configuration management. - Supports hybrid environments (on-prem & cloud). - Not a direct alternative to CloudFormation; can be used alongside CloudFormation for application stack management. |
- You prefer using Chef or Puppet for configuration management. - You have a hybrid cloud setup. - You need infrastructure-as-code for automated deployments. - Use CloudFormation for resource provisioning and OpsWorks for managing the configuration of those resources. |
| AWS Systems Manager | Centralized service for operational data and management across resources. | - State Manager: Maintains instance configurations. - Parameter Store (can store secrets, both plaintext and encrypted). - Run Command: Executes commands remotely. - Patch Manager: Automates patching. - Automation: Workflows for deployments. - Parameter Store: Can store encrypted SecureString parameters (cheaper than Secrets Manager). |
- You need centralized operational management across AWS and on-premises. - You want to automate config, patching, and maintenance tasks at scale. - You need basic secret storage without rotation using Parameter Store. - You require secure parameter management for secrets or configs. |
| AWS Config | Tracks and audits AWS resource configurations and compliance. | - Continuous configuration tracking. - Rules and conformance packs for compliance checks. - Configuration history of resources. |
- You need to audit and track configuration changes over time. - You want to ensure compliance with internal policies or industry standards (e.g., PCI-DSS). |
| AWS AppConfig | Manages and deploys dynamic application configurations without redeployment. | - Feature flagging for applications. - Gradual configuration rollouts with monitoring. - Automatic rollback on failure. |
- You need dynamic app configurations without redeploying. - You want feature flagging and controlled rollout of config changes. - You need real-time config updates with safe rollback options. |
| AWS Secrets Manager | Fully managed service for storing and managing sensitive information (secrets). | - Automatic rotation of secrets (e.g., for RDS, Redshift, etc.). - Secure integration with AWS services. - Fine-grained access control using IAM. - Higher cost than Parameter Store. - Supports key/value pairs for secrets. |
- You need automatic rotation of secrets. - You are managing sensitive credentials like database passwords, API keys, and want tight integration with AWS services. - You're okay with higher cost in exchange for features like secret rotation. |
| AWS KMS (Key Management Service) | Service for managing and controlling encryption keys. | - Create and manage encryption keys. - Used with Secrets Manager and Parameter Store to encrypt secrets. - Supports encryption at rest for S3, EBS, etc. - Provides fine-grained key policies and audit logs. |
- You need to manage encryption keys for securing data at rest or in transit. - You want to encrypt secrets stored in Secrets Manager or Parameter Store. - Not a secret store, but essential for encryption. |
Here’s a breakdown to help you choose the right secret storage solution:
-
Use AWS Secrets Manager:
- When you need automatic rotation of secrets (e.g., database credentials, API keys).
- For secrets that are critical to your applications and require frequent updates or rotations.
- Higher cost but includes automated secret management and integrations with other AWS services.
-
Use AWS Systems Manager Parameter Store (SecureString):
- When you need a cheaper solution for storing encrypted parameters (secrets) without requiring automatic rotation.
- Great for storing less critical secrets or when rotation can be handled manually or through custom automation.
- SecureString in Parameter Store is encrypted using AWS KMS keys.
-
Use AWS Key Management Service (KMS):
- When your focus is on managing encryption keys rather than directly managing secrets.
- KMS is essential for encrypting secrets stored in Parameter Store and Secrets Manager.
- Ideal for scenarios where you need fine-grained control over encryption keys and their usage policies.
| Service | Purpose | Key Features | When to Use |
|---|---|---|---|
| AWS CloudFormation | Declarative IaC tool using YAML/JSON templates. | - Automated provisioning, updates, and management of AWS resources. - Use StackSets for cross-account and cross-region deployments. - Supports nested stacks for modular design. - Tight integration with other AWS services. - Supports drift detection and rollback features. |
- Need a native AWS IaC tool. - Require infrastructure management across multiple accounts and regions. - Prefer declarative infrastructure definition. - Want modular, reusable templates. |
| AWS CDK (Cloud Development Kit) | Define AWS infrastructure using high-level programming languages. | - Supports TypeScript, Python, Java, etc. - Generates CloudFormation templates under the hood. - Object-oriented design patterns enable reusable and maintainable infrastructure code. - Allows using programming constructs (loops, conditionals, functions). |
- Teams that prefer using programming languages for IaC. - Need complex infrastructure management. - Want to reuse code constructs (e.g., inheritance, encapsulation). |
| AWS SAM (Serverless Application Model) | Simplifies IaC for serverless architectures. | - Shortens syntax for serverless resources (e.g., Lambda, API Gateway, DynamoDB). - Provides SAM CLI for local testing of serverless apps. - Automatically generates CloudFormation templates. - Integrates with AWS services like Step Functions and S3. |
- Need a simplified IaC tool for serverless applications. - Want fewer lines of code than CloudFormation for Lambda and API Gateway. - Require local testing capabilities with SAM CLI. |
| Terraform (Non-AWS) | Multi-cloud IaC tool from HashiCorp using HCL (HashiCorp Configuration Language). | - Supports AWS, Azure, GCP, and other cloud providers. - Manages resources using state files for tracking. - Provider agnostic with modules for each cloud provider. - Large community support with Terraform Registry for reusable modules. - Better suited for hybrid cloud environments. |
- Multi-cloud or hybrid cloud environments. - Need flexibility for managing resources across multiple cloud providers. - Teams already familiar with HCL and HashiCorp tools. |
- Amazon Athena: A serverless, interactive query service for analyzing data in Amazon S3 using standard SQL.
- Amazon EMR: A managed cluster platform to process and analyze big data using Hadoop, Spark, and other frameworks.
- Amazon Kinesis Data Firehose: A fully managed service for real-time streaming data delivery to destinations like S3 and Redshift.
- Amazon Kinesis Data Streams: A service for building custom real-time applications that process or analyze streaming data.
- Amazon OpenSearch Service: A managed service to deploy, operate, and scale OpenSearch clusters for search, monitoring, and visualization.
- Amazon AppFlow: A service to securely integrate AWS services with SaaS applications for data transfer.
- AWS App Runner: A fully managed service for deploying containerized web applications and APIs without managing infrastructure.
- EC2 Image Builder: A service to automate the creation, management, and deployment of custom machine images.
- AWS Serverless Application Repository: A repository of pre-built serverless applications that can be deployed in AWS.
- AWS App2Container: A tool to automatically containerize Java and .NET applications for deployment in ECS or EKS.
- AWS Copilot: A CLI tool to simplify the development and deployment of containerized applications on AWS.
- Amazon Elastic Container Registry (ECR): A fully managed Docker container registry for storing, managing, and deploying container images.
- Amazon Elastic Container Service (ECS): A fully managed container orchestration service for running and managing Docker containers.
- Amazon Elastic Kubernetes Service (EKS): A managed service to run Kubernetes without managing control planes or nodes.
- Amazon EKS Distro: A Kubernetes distribution that enables running Kubernetes on-premises or in any cloud.
- AWS Fargate: A serverless compute engine for containers that runs ECS or EKS workloads without managing infrastructure.
- Red Hat OpenShift Service on AWS (ROSA): A managed Red Hat OpenShift service for running Kubernetes clusters on AWS.
- Amazon Aurora: A high-performance, fully managed relational database compatible with MySQL and PostgreSQL.
- Amazon Aurora Serverless v2: A version of Aurora that automatically scales database capacity based on application needs.
- AWS Database Migration Service (AWS DMS): A service to migrate databases to AWS with minimal downtime.
- Amazon DocumentDB (with MongoDB compatibility): A fully managed document database service compatible with MongoDB workloads.
- Amazon DynamoDB: A fully managed NoSQL database service for fast and flexible performance at scale.
- Amazon ElastiCache: A fully managed in-memory caching service supporting Redis and Memcached.
- Amazon MemoryDB for Redis: A Redis-compatible, in-memory database for ultra-fast, low-latency data processing.
- Amazon RDS: A fully managed relational database service supporting multiple engines like MySQL, PostgreSQL, and Oracle.
- Amazon Redshift: A fully managed data warehouse for fast querying and analysis of large datasets.
- AWS CLI: A unified tool to manage AWS services from the command line.
- AWS CloudShell: A browser-based shell to interact with AWS resources from the console.
- Amazon CodeGuru: A machine learning-powered tool for automated code reviews and performance recommendations.
- AWS CodeStar: A platform for setting up CI/CD pipelines to automate application deployments.
- AWS Fault Injection Simulator (AWS FIS): A fully managed service for running chaos engineering experiments to test application resilience.
- AWS Compute Optimizer: A service that recommends optimal EC2 instance types based on resource usage.
- AWS Control Tower: A service to set up and govern multi-account AWS environments using best practices.
- AWS License Manager: A service to manage, track, and control software licenses across AWS and on-premises environments.
- Amazon Managed Grafana: A fully managed service for running Grafana dashboards to visualize metrics from multiple data sources.
- Amazon Managed Service for Prometheus: A managed monitoring service compatible with Prometheus for collecting and querying metrics.
- AWS Proton: A fully managed service for automating infrastructure provisioning and code deployment for container and serverless applications.
- AWS Resilience Hub: A service for assessing and improving the resilience of applications on AWS.
- AWS Service Catalog: A tool to centrally manage and distribute approved products (e.g., CloudFormation templates) for organizational use.
- AWS Trusted Advisor: A service that provides real-time best practice recommendations for cost optimization, security, and performance.
- Amazon API Gateway: A fully managed service to create, publish, and manage secure APIs at scale.
- AWS Client VPN: A fully managed, secure VPN service for connecting to AWS and on-premises networks.
- AWS PrivateLink: A service for private connectivity between VPCs and AWS services without exposing traffic to the public internet.
- Amazon Route 53: A scalable DNS and domain name registration service with health-checking capabilities.
- AWS Site-to-Site VPN: A service for securely connecting on-premises networks to AWS over VPN.
- AWS Transit Gateway: A network hub that connects VPCs and on-premises networks in a scalable manner.
- Amazon VPC: A service to provision logically isolated cloud resources in a virtual network.
- AWS Certificate Manager (ACM): A service for provisioning and managing SSL/TLS certificates for securing websites and applications.
- AWS CloudHSM: A managed hardware security module (HSM) service for cryptographic operations.
- Amazon Cognito: A service to add user sign-up, sign-in, and authentication to web and mobile apps.
- Amazon Detective: A service for analyzing and investigating security issues using machine learning.
- AWS Directory Service: A service that enables AWS resources to work with Microsoft Active Directory.
- Amazon GuardDuty: A threat detection service that continuously monitors for malicious activity.
- AWS IAM Identity Center (AWS SSO): A service for centrally managing SSO access to AWS accounts and business applications.
- AWS Identity and Access Management (IAM): A service for managing access to AWS resources with fine-grained permission control.
- Amazon Inspector: A service that automatically assesses the security of applications deployed on AWS.
- AWS Key Management Service (KMS): A managed service for creating and controlling cryptographic keys.
- Amazon Macie: A security service that uses machine learning to identify and protect sensitive data in S3.
- AWS Network Firewall: A managed firewall service for controlling traffic to and from your VPCs.
- AWS Resource Access Manager (AWS RAM): A service for sharing AWS resources across multiple accounts.
- AWS Security Hub: A service that provides a comprehensive view of security alerts across AWS accounts.
- AWS Security Token Service (STS): A service for providing temporary, limited-privilege credentials for AWS resources.
- AWS Shield: A managed DDoS protection service for AWS applications.
- AWS WAF: A web application firewall that helps protect against common web exploits.
- Amazon Simple Notification Service (SNS): A fully managed pub/sub messaging service for sending notifications.
- Amazon Simple Queue Service (SQS): A fully managed message queue service for decoupling application components.
- AWS Step Functions: A service for orchestrating distributed applications using visual workflows.
- AWS Backup: A fully managed service to automate backups of AWS resources.
- Amazon Elastic Block Store (EBS): Persistent block storage for EC2 instances.
- AWS Elastic Disaster Recovery: A service to recover applications from outages using automated, scalable disaster recovery.
- Amazon Elastic File System (EFS): A fully managed, scalable file system for use with AWS services and on-premises resources.
- Amazon FSx for Lustre: A fully managed file system optimized for compute-intensive workloads like machine learning.
- Amazon FSx for NetApp ONTAP: A fully managed NetApp ONTAP file system on AWS.
- Amazon FSx for OpenZFS: A fully managed ZFS file system for performance-intensive workloads.
- Amazon FSx for Windows File Server: A fully managed Windows file system with support for SMB protocol.
- Amazon S3: An object storage service that offers industry-leading scalability, data availability, and security.
- Amazon S3 Glacier: A low-cost storage service for long-term data archiving and backup.
- AWS Storage Gateway: A hybrid cloud storage service that provides on-premises access to virtually unlimited cloud storage.
- For orchestration of multiple AWS services or complex workflows, the answer is likely AWS Step Functions.
- When choosing a solution for storing and rotating secrets (like API keys, passwords), the answer is most likely AWS Secrets Manager. If automatic rotation is not needed and cost is a concern, consider AWS Systems Manager Parameter Store.
- For questions related to centralizing and automating management across multiple AWS accounts (such as governance or account creation), the answer is likely AWS Control Tower or AWS Organizations.
- If the question involves automating infrastructure deployment or managing infrastructure as code, the answer will likely be AWS CloudFormation or AWS CDK (Cloud Development Kit) if programming languages are involved.
- If the scenario is about improving application performance or cost optimization in compute resources, the answer is likely AWS Compute Optimizer or Auto Scaling.
- For logging, monitoring, or tracking performance of applications and infrastructure, the answer will often be Amazon CloudWatch. For log aggregation, consider Amazon CloudWatch Logs or AWS CloudTrail (if tracking API calls or governance is involved).
- If the question involves auditing, maintaining, or tracking compliance with security standards, it's most likely AWS Config or AWS Audit Manager (for audit-related tasks).
- For multi-region or disaster recovery (DR) architectures, the answer is often Amazon Route 53 for DNS failover or AWS Elastic Disaster Recovery for DR-specific scenarios.
- For hybrid cloud architectures or extending on-premise environments to the cloud, the answer will often involve AWS Direct Connect or AWS Storage Gateway.
- For encryption and key management (i.e., securing sensitive data, especially encryption of data at rest or in transit), look for AWS Key Management Service (KMS) or AWS CloudHSM (for hardware-level encryption needs).
- When deploying and managing Docker containers without worrying about infrastructure (serverless for containers), the answer will likely be AWS Fargate.
- For managing a containerized application with orchestration (e.g., microservices), the answer will likely be Amazon Elastic Kubernetes Service (EKS) for Kubernetes or Amazon Elastic Container Service (ECS) for AWS-native container orchestration.
- If the question is about managing Docker images (storing, retrieving, and managing container images), the answer will likely be Amazon Elastic Container Registry (ECR).
- If the question involves controlling access to AWS resources, it's almost always AWS Identity and Access Management (IAM) or IAM Identity Center (formerly AWS SSO) for single sign-on needs.
- For DDoS protection, the answer is typically AWS Shield (Standard or Advanced).
- For protecting web applications from exploits and common web vulnerabilities, the answer is likely AWS WAF (Web Application Firewall).
- For securing traffic between VPCs or between on-premises and AWS, the answer is likely AWS PrivateLink or AWS Transit Gateway.
- For real-time threat detection and anomaly detection across AWS services, the answer is likely Amazon GuardDuty.
- For centralized security findings aggregation and compliance tracking, the answer is likely AWS Security Hub.
- For vulnerability scanning of EC2 instances and container images, the answer is likely Amazon Inspector.