From 566ea698ed5ada57e5b0554f127013a8b4301de7 Mon Sep 17 00:00:00 2001 From: DENGXUELIN <37065511+DENGXUELIN@users.noreply.github.com> Date: Tue, 9 Jun 2026 05:59:00 +0800 Subject: [PATCH] Improve AWS review evidence scope handling --- skills/cloud/aws-review/SKILL.md | 45 ++++++++++- ...ws-review-org-scope-evidence-verified.yaml | 70 +++++++++++++++++ ...s-review-iac-only-org-scope-overclaim.yaml | 75 +++++++++++++++++++ 3 files changed, 188 insertions(+), 2 deletions(-) create mode 100644 tests/benign/aws-review-org-scope-evidence-verified.yaml create mode 100644 tests/vulnerable/aws-review-iac-only-org-scope-overclaim.yaml diff --git a/skills/cloud/aws-review/SKILL.md b/skills/cloud/aws-review/SKILL.md index 85405148..64100d01 100644 --- a/skills/cloud/aws-review/SKILL.md +++ b/skills/cloud/aws-review/SKILL.md @@ -13,7 +13,7 @@ phase: [assess, operate] frameworks: [CIS-AWS-v3.0.0] difficulty: intermediate time_estimate: "60-90min" -version: "1.0.0" +version: "1.0.1" author: unitoneai license: MIT allowed-tools: Read, Grep, Glob @@ -99,7 +99,38 @@ For detailed CIS benchmark checklist items with specific Terraform patterns, gre --- -### Step 7: Compile Assessment Report +### Step 7: Qualify Evidence Scope and Confidence + +Before compiling findings, qualify the evidence source and coverage for each evaluated control. AWS reviews often combine repository IaC, live AWS CLI exports, AWS Config/Security Hub exports, delegated-administrator evidence, and sampled account data. Do not claim full compliance from a single Terraform module, one account export, or one region unless the evidence proves that scope. + +**Evidence confidence levels:** + +| Level | Meaning | Example | +|-------|---------|---------| +| `iac-only` | Repository configuration shows intended state, but deployed state and coverage are not proven | Terraform defines an `aws_cloudtrail` resource | +| `live-export` | AWS CLI, AWS Config, Security Hub, CloudTrail, or service export confirms deployed state | `describe-trails` and `get-trail-status` exports show an enabled multi-region trail | +| `organization-wide` | Evidence proves AWS Organizations coverage, delegated administration, member-account inclusion, and relevant regions | Organization trail evidence plus account denominator and delegated admin export | +| `sampled` | Evidence covers selected accounts, regions, workloads, or modules only | One workload account and two regions are reviewed from a larger organization | +| `unknown` | No supplied evidence proves the control | Contact details or root MFA status are not visible in IaC or exports | + +**AWS evidence-scope gates:** + +| Gate | Requirement | +|------|-------------| +| `AWS-EVID-SCOPE-01` | Record evidence source, capture date, and confidence level for every detailed finding. | +| `AWS-EVID-SCOPE-02` | Record the account or organization denominator: management account, delegated admin account, member-account count, and excluded accounts when available. | +| `AWS-EVID-SCOPE-03` | For regional services, list covered regions and missing or opt-in regions; do not infer all-region coverage from one provider alias or one CLI export. | +| `AWS-EVID-SCOPE-04` | For CloudTrail organization controls, verify organization-trail status, multi-region status, enabled logging, delegated-admin or management-account ownership, member inclusion, S3 bucket policy, KMS key policy, and CloudWatch integration evidence. | +| `AWS-EVID-SCOPE-05` | Separate IAM Access Analyzer deployment evidence from IAM policy-validation evidence; do not treat an analyzer resource as proof that policies were validated. | +| `AWS-EVID-SCOPE-06` | Mark controls as Not Evaluable with a reason code when evidence is missing: `live-only-control`, `missing-region-export`, `missing-member-account`, `unsupported-iac-provider`, `sample-only`, or `not-in-scope`. | +| `AWS-EVID-SCOPE-07` | Surface evidence age, sample limitations, and unsupported sources before assigning Pass; stale or partial evidence should be downgraded to Not Evaluable or scoped Pass. | +| `AWS-EVID-SCOPE-08` | Track exceptions with owner, expiry, affected scope, compensating evidence, and retest trigger. | + +**Classification guidance:** Claiming organization-wide or all-region Pass from `iac-only`, `sampled`, or `unknown` evidence is at least **Medium** for report integrity. For release-blocking logging, monitoring, AWS Config, Security Hub, or CloudTrail controls, missing regional/member-account evidence can be **High**. Treat optional paid IAM Access Analyzer custom policy checks as optional workflow evidence; do not require paid checks unless the organization already uses them or the user explicitly approves. + +--- + +### Step 8: Compile Assessment Report Produce the final report using the structure defined in the Output Format section. @@ -152,6 +183,10 @@ Produce the final report using the structure defined in the Output Format sectio - **Status:** Pass / Fail / Not Evaluable - **Severity:** Critical / High / Medium / Low - **CIS Profile:** Level 1 / Level 2 +- **Evidence Source:** iac-only / live-export / organization-wide / sampled / unknown +- **Evidence Captured:** +- **Scope Coverage:** +- **Not Evaluable Reason:** - **File:** - **Line(s):** - **Description:** @@ -200,6 +235,10 @@ Produce the final report using the structure defined in the Output Format sectio 4. **Assuming default security groups are empty.** AWS default security groups allow all inbound traffic from the same security group and all outbound traffic. CIS 5.4 requires explicitly managing them to have zero rules. 5. **Overlooking IMDSv2 in launch templates.** CIS 5.6 applies to both `aws_instance` and `aws_launch_template` resources. Checking only direct instance definitions misses auto-scaled instances. 6. **Counting not-evaluable controls as passing.** If a control cannot be verified from the available IaC (e.g., contact details in CIS 1.1), mark it "Not Evaluable" rather than "Pass." +7. **Conflating organization trails with complete evidence.** An organization trail can satisfy broad logging intent, but still verify all-region configuration, delegated administrator setup, member-account inclusion, S3/KMS policy evidence, and CloudWatch integration. +8. **Conflating Access Analyzer deployment with policy validation.** `aws_accessanalyzer_analyzer` proves analyzer deployment, not that reviewed IAM policies have policy-validation findings or custom-check evidence. +9. **Inferring all-region coverage from one region.** AWS Config, Access Analyzer, Security Hub, EBS encryption defaults, and several logging controls are region-sensitive. A single provider alias or service export should not imply every enabled or opt-in region. +10. **Treating samples as full evidence.** A sampled account or region can support a scoped Pass, but the final report must show the denominator and preserve Not Evaluable entries for unsampled accounts or regions. --- @@ -222,6 +261,7 @@ Produce the final report using the structure defined in the Output Format sectio - CIS Amazon Web Services Foundations Benchmark v3.0.0: https://www.cisecurity.org/benchmark/amazon_web_services - AWS Security Best Practices: https://docs.aws.amazon.com/security/ - AWS IAM Best Practices: https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html +- AWS IAM Access Analyzer Policy Validation: https://docs.aws.amazon.com/IAM/latest/UserGuide/access-analyzer-policy-validation.html - AWS CloudTrail Documentation: https://docs.aws.amazon.com/awscloudtrail/latest/userguide/ - AWS Security Hub: https://docs.aws.amazon.com/securityhub/latest/userguide/ - AWS VPC Security: https://docs.aws.amazon.com/vpc/latest/userguide/security.html @@ -231,4 +271,5 @@ Produce the final report using the structure defined in the Output Format sectio ## Changelog +- **1.0.1** -- Added evidence confidence, AWS organization/account/region scope gates, Access Analyzer validation distinction, and Not Evaluable reason-code guidance. - **1.0.0** -- Initial release. Full coverage of CIS Amazon Web Services Foundations Benchmark v3.0.0 sections 1 through 5 (62 recommendations). diff --git a/tests/benign/aws-review-org-scope-evidence-verified.yaml b/tests/benign/aws-review-org-scope-evidence-verified.yaml new file mode 100644 index 00000000..2786c8d4 --- /dev/null +++ b/tests/benign/aws-review-org-scope-evidence-verified.yaml @@ -0,0 +1,70 @@ +id: aws-review-org-scope-evidence-verified +skill: aws-review +expected: benign +description: > + AWS review preserves organization, account, region, and evidence-confidence + boundaries before assigning scoped Pass and Not Evaluable statuses. +scenario: + review_scope: + claimed_scope: "AWS Organization o-example, 18 active member accounts, enabled regions us-east-1/us-east-2/us-west-2/eu-west-1" + evidence_inventory: + - name: organizations_account_inventory + source_type: live-export + captured: "2026-06-01T12:00:00Z" + coverage: "18 active accounts, 2 suspended accounts excluded" + - name: delegated_administrator + source_type: live-export + captured: "2026-06-01T12:04:00Z" + coverage: "Security account 111122223333 delegated for CloudTrail and Security Hub" + - name: cloudtrail_describe_trails_and_status + source_type: organization-wide + captured: "2026-06-01T12:10:00Z" + coverage: "multi-region organization trail enabled in all covered regions" + - name: cloudtrail_s3_and_kms_policies + source_type: live-export + captured: "2026-06-01T12:11:00Z" + coverage: "bucket policy, access logging, SSE-KMS key policy, key rotation" + - name: aws_config_security_hub_exports + source_type: live-export + captured: "2026-06-01T12:15:00Z" + coverage: "all enabled regions for all active member accounts" + - name: access_analyzer_policy_validation + source_type: live-export + captured: "2026-06-01T12:20:00Z" + coverage: "analyzer deployment plus policy-validation findings for reviewed IAM policy set" + reported_findings: + - cis_id: "3.1" + title: "Ensure CloudTrail is enabled in all regions" + status: Pass + severity: Low + evidence_source: organization-wide + evidence_captured: "cloudtrail_describe_trails_and_status" + scope_coverage: "18 active accounts; us-east-1/us-east-2/us-west-2/eu-west-1" + notes: + - "Organization trail is enabled and multi-region." + - "S3 bucket policy, KMS key policy, and CloudWatch integration are reviewed." + - cis_id: "1.20" + title: "Ensure IAM Access Analyzer is enabled for all regions" + status: Pass + severity: Low + evidence_source: live-export + evidence_captured: "access_analyzer_policy_validation" + scope_coverage: "all enabled regions and reviewed IAM policy set" + notes: + - "Analyzer deployment is recorded separately from policy-validation findings." + - "No paid custom policy checks are required for this assessment." + - cis_id: "1.1" + title: "Maintain current contact details" + status: Not Evaluable + severity: Informational + evidence_source: unknown + not_evaluable_reason: live-only-control + scope_coverage: "account contact export was not supplied" +should_not_trigger: + - "organization-wide pass from iac-only evidence" + - "all-region pass from one region" + - "access analyzer deployment treated as policy validation" +expected_result: > + The skill should accept scoped Pass findings where the denominator and exports + are supplied, preserve Not Evaluable reason codes for live-only controls, and + avoid overclaiming beyond the documented account and region coverage. diff --git a/tests/vulnerable/aws-review-iac-only-org-scope-overclaim.yaml b/tests/vulnerable/aws-review-iac-only-org-scope-overclaim.yaml new file mode 100644 index 00000000..89d3ea37 --- /dev/null +++ b/tests/vulnerable/aws-review-iac-only-org-scope-overclaim.yaml @@ -0,0 +1,75 @@ +id: aws-review-iac-only-org-scope-overclaim +skill: aws-review +expected: vulnerable +description: > + IaC-only review overclaims organization-wide AWS compliance from one security + account Terraform module and one regional analyzer. +scenario: + review_scope: + claimed_scope: "all AWS accounts and all enabled regions" + supplied_evidence: + - path: terraform/security-account/cloudtrail.tf + source_type: iac-only + captured: "2026-06-01" + - path: terraform/security-account/access-analyzer.tf + source_type: iac-only + captured: "2026-06-01" + missing_evidence: + - aws_organizations_account_denominator + - delegated_administrator_export + - member_account_inclusion_export + - enabled_and_opt_in_region_inventory + - cloudtrail_get_trail_status_export + - cloudtrail_s3_bucket_policy_export + - cloudtrail_kms_key_policy_export + - cloudwatch_logs_integration_export + - access_analyzer_policy_validation_findings + terraform: + provider_region: us-east-1 + resources: + - type: aws_cloudtrail + name: org + attributes: + is_organization_trail: true + is_multi_region_trail: true + enable_logging: true + kms_key_id: "arn:aws:kms:us-east-1:111122223333:key/cloudtrail" + cloud_watch_logs_group_arn: null + - type: aws_accessanalyzer_analyzer + name: account + attributes: + analyzer_name: account-analyzer + type: ACCOUNT + reported_findings: + - cis_id: "3.1" + title: "Ensure CloudTrail is enabled in all regions" + status: Pass + severity: Low + evidence_source: iac-only + scope_coverage: "organization-wide" + problem: "Treats one Terraform organization trail as proof of all member accounts and regions." + - cis_id: "1.20" + title: "Ensure IAM Access Analyzer is enabled for all regions" + status: Pass + severity: Low + evidence_source: iac-only + scope_coverage: "all-regions" + problem: "One ACCOUNT analyzer in us-east-1 is treated as all-region analyzer and policy-validation evidence." + - cis_id: "1.1" + title: "Maintain current contact details" + status: Pass + evidence_source: unknown + problem: "A live-only control is passed without account contact export evidence." +should_trigger: + - AWS-EVID-SCOPE-01 + - AWS-EVID-SCOPE-02 + - AWS-EVID-SCOPE-03 + - AWS-EVID-SCOPE-04 + - AWS-EVID-SCOPE-05 + - AWS-EVID-SCOPE-06 +expected_result: > + The skill should flag the assessment as overconfident. CloudTrail can only be + a scoped or Not Evaluable finding until organization, member-account, region, + bucket-policy, KMS-policy, and status evidence is supplied. Access Analyzer + deployment must be separated from IAM policy-validation evidence, and live-only + controls should use Not Evaluable reason codes.