Improve AWS review evidence scope handling

skills/cloud/aws-review/SKILL.md

@@ -13,7 +13,7 @@ phase: [assess, operate]
 frameworks: [CIS-AWS-v3.0.0]
 difficulty: intermediate
 time_estimate: "60-90min"
-version: "1.0.0"
+version: "1.0.1"
 author: unitoneai
 license: MIT
 allowed-tools: Read, Grep, Glob
@@ -99,7 +99,38 @@ For detailed CIS benchmark checklist items with specific Terraform patterns, gre
 
 ---
 
-### Step 7: Compile Assessment Report
+### Step 7: Qualify Evidence Scope and Confidence
+
+Before compiling findings, qualify the evidence source and coverage for each evaluated control. AWS reviews often combine repository IaC, live AWS CLI exports, AWS Config/Security Hub exports, delegated-administrator evidence, and sampled account data. Do not claim full compliance from a single Terraform module, one account export, or one region unless the evidence proves that scope.
+
+**Evidence confidence levels:**
+
+| Level | Meaning | Example |
+|-------|---------|---------|
+| `iac-only` | Repository configuration shows intended state, but deployed state and coverage are not proven | Terraform defines an `aws_cloudtrail` resource |
+| `live-export` | AWS CLI, AWS Config, Security Hub, CloudTrail, or service export confirms deployed state | `describe-trails` and `get-trail-status` exports show an enabled multi-region trail |
+| `organization-wide` | Evidence proves AWS Organizations coverage, delegated administration, member-account inclusion, and relevant regions | Organization trail evidence plus account denominator and delegated admin export |
+| `sampled` | Evidence covers selected accounts, regions, workloads, or modules only | One workload account and two regions are reviewed from a larger organization |
+| `unknown` | No supplied evidence proves the control | Contact details or root MFA status are not visible in IaC or exports |
+
+**AWS evidence-scope gates:**
+
+| Gate | Requirement |
+|------|-------------|
+| `AWS-EVID-SCOPE-01` | Record evidence source, capture date, and confidence level for every detailed finding. |
+| `AWS-EVID-SCOPE-02` | Record the account or organization denominator: management account, delegated admin account, member-account count, and excluded accounts when available. |
+| `AWS-EVID-SCOPE-03` | For regional services, list covered regions and missing or opt-in regions; do not infer all-region coverage from one provider alias or one CLI export. |
+| `AWS-EVID-SCOPE-04` | For CloudTrail organization controls, verify organization-trail status, multi-region status, enabled logging, delegated-admin or management-account ownership, member inclusion, S3 bucket policy, KMS key policy, and CloudWatch integration evidence. |
+| `AWS-EVID-SCOPE-05` | Separate IAM Access Analyzer deployment evidence from IAM policy-validation evidence; do not treat an analyzer resource as proof that policies were validated. |
+| `AWS-EVID-SCOPE-06` | Mark controls as Not Evaluable with a reason code when evidence is missing: `live-only-control`, `missing-region-export`, `missing-member-account`, `unsupported-iac-provider`, `sample-only`, or `not-in-scope`. |
+| `AWS-EVID-SCOPE-07` | Surface evidence age, sample limitations, and unsupported sources before assigning Pass; stale or partial evidence should be downgraded to Not Evaluable or scoped Pass. |
+| `AWS-EVID-SCOPE-08` | Track exceptions with owner, expiry, affected scope, compensating evidence, and retest trigger. |
+
+**Classification guidance:** Claiming organization-wide or all-region Pass from `iac-only`, `sampled`, or `unknown` evidence is at least **Medium** for report integrity. For release-blocking logging, monitoring, AWS Config, Security Hub, or CloudTrail controls, missing regional/member-account evidence can be **High**. Treat optional paid IAM Access Analyzer custom policy checks as optional workflow evidence; do not require paid checks unless the organization already uses them or the user explicitly approves.
+
+---
+
+### Step 8: Compile Assessment Report
 
 Produce the final report using the structure defined in the Output Format section.
 
@@ -152,6 +183,10 @@ Produce the final report using the structure defined in the Output Format sectio
 - **Status:** Pass / Fail / Not Evaluable
 - **Severity:** Critical / High / Medium / Low
 - **CIS Profile:** Level 1 / Level 2
+- **Evidence Source:** iac-only / live-export / organization-wide / sampled / unknown
+- **Evidence Captured:** <date/time or export identifier>
+- **Scope Coverage:** <account IDs, AWS Organization, regions, module scope, or sample scope>
+- **Not Evaluable Reason:** <reason code if applicable>
 - **File:** <path to relevant config>
 - **Line(s):** <line numbers if applicable>
 - **Description:** <what was found>
@@ -200,6 +235,10 @@ Produce the final report using the structure defined in the Output Format sectio
 4. **Assuming default security groups are empty.** AWS default security groups allow all inbound traffic from the same security group and all outbound traffic. CIS 5.4 requires explicitly managing them to have zero rules.
 5. **Overlooking IMDSv2 in launch templates.** CIS 5.6 applies to both `aws_instance` and `aws_launch_template` resources. Checking only direct instance definitions misses auto-scaled instances.
 6. **Counting not-evaluable controls as passing.** If a control cannot be verified from the available IaC (e.g., contact details in CIS 1.1), mark it "Not Evaluable" rather than "Pass."
+7. **Conflating organization trails with complete evidence.** An organization trail can satisfy broad logging intent, but still verify all-region configuration, delegated administrator setup, member-account inclusion, S3/KMS policy evidence, and CloudWatch integration.
+8. **Conflating Access Analyzer deployment with policy validation.** `aws_accessanalyzer_analyzer` proves analyzer deployment, not that reviewed IAM policies have policy-validation findings or custom-check evidence.
+9. **Inferring all-region coverage from one region.** AWS Config, Access Analyzer, Security Hub, EBS encryption defaults, and several logging controls are region-sensitive. A single provider alias or service export should not imply every enabled or opt-in region.
+10. **Treating samples as full evidence.** A sampled account or region can support a scoped Pass, but the final report must show the denominator and preserve Not Evaluable entries for unsampled accounts or regions.
 
 ---
 
@@ -222,6 +261,7 @@ Produce the final report using the structure defined in the Output Format sectio
 - CIS Amazon Web Services Foundations Benchmark v3.0.0: https://www.cisecurity.org/benchmark/amazon_web_services
 - AWS Security Best Practices: https://docs.aws.amazon.com/security/
 - AWS IAM Best Practices: https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html
+- AWS IAM Access Analyzer Policy Validation: https://docs.aws.amazon.com/IAM/latest/UserGuide/access-analyzer-policy-validation.html
 - AWS CloudTrail Documentation: https://docs.aws.amazon.com/awscloudtrail/latest/userguide/
 - AWS Security Hub: https://docs.aws.amazon.com/securityhub/latest/userguide/
 - AWS VPC Security: https://docs.aws.amazon.com/vpc/latest/userguide/security.html
@@ -231,4 +271,5 @@ Produce the final report using the structure defined in the Output Format sectio
 
 ## Changelog
 
+- **1.0.1** -- Added evidence confidence, AWS organization/account/region scope gates, Access Analyzer validation distinction, and Not Evaluable reason-code guidance.
 - **1.0.0** -- Initial release. Full coverage of CIS Amazon Web Services Foundations Benchmark v3.0.0 sections 1 through 5 (62 recommendations).

tests/benign/aws-review-org-scope-evidence-verified.yaml

@@ -0,0 +1,70 @@
+id: aws-review-org-scope-evidence-verified
+skill: aws-review
+expected: benign
+description: >
+  AWS review preserves organization, account, region, and evidence-confidence
+  boundaries before assigning scoped Pass and Not Evaluable statuses.
+scenario:
+  review_scope:
+    claimed_scope: "AWS Organization o-example, 18 active member accounts, enabled regions us-east-1/us-east-2/us-west-2/eu-west-1"
+    evidence_inventory:
+      - name: organizations_account_inventory
+        source_type: live-export
+        captured: "2026-06-01T12:00:00Z"
+        coverage: "18 active accounts, 2 suspended accounts excluded"
+      - name: delegated_administrator
+        source_type: live-export
+        captured: "2026-06-01T12:04:00Z"
+        coverage: "Security account 111122223333 delegated for CloudTrail and Security Hub"
+      - name: cloudtrail_describe_trails_and_status
+        source_type: organization-wide
+        captured: "2026-06-01T12:10:00Z"
+        coverage: "multi-region organization trail enabled in all covered regions"
+      - name: cloudtrail_s3_and_kms_policies
+        source_type: live-export
+        captured: "2026-06-01T12:11:00Z"
+        coverage: "bucket policy, access logging, SSE-KMS key policy, key rotation"
+      - name: aws_config_security_hub_exports
+        source_type: live-export
+        captured: "2026-06-01T12:15:00Z"
+        coverage: "all enabled regions for all active member accounts"
+      - name: access_analyzer_policy_validation
+        source_type: live-export
+        captured: "2026-06-01T12:20:00Z"
+        coverage: "analyzer deployment plus policy-validation findings for reviewed IAM policy set"
+  reported_findings:
+    - cis_id: "3.1"
+      title: "Ensure CloudTrail is enabled in all regions"
+      status: Pass
+      severity: Low
+      evidence_source: organization-wide
+      evidence_captured: "cloudtrail_describe_trails_and_status"
+      scope_coverage: "18 active accounts; us-east-1/us-east-2/us-west-2/eu-west-1"
+      notes:
+        - "Organization trail is enabled and multi-region."
+        - "S3 bucket policy, KMS key policy, and CloudWatch integration are reviewed."
+    - cis_id: "1.20"
+      title: "Ensure IAM Access Analyzer is enabled for all regions"
+      status: Pass
+      severity: Low
+      evidence_source: live-export
+      evidence_captured: "access_analyzer_policy_validation"
+      scope_coverage: "all enabled regions and reviewed IAM policy set"
+      notes:
+        - "Analyzer deployment is recorded separately from policy-validation findings."
+        - "No paid custom policy checks are required for this assessment."
+    - cis_id: "1.1"
+      title: "Maintain current contact details"
+      status: Not Evaluable
+      severity: Informational
+      evidence_source: unknown
+      not_evaluable_reason: live-only-control
+      scope_coverage: "account contact export was not supplied"
+should_not_trigger:
+  - "organization-wide pass from iac-only evidence"
+  - "all-region pass from one region"
+  - "access analyzer deployment treated as policy validation"
+expected_result: >
+  The skill should accept scoped Pass findings where the denominator and exports
+  are supplied, preserve Not Evaluable reason codes for live-only controls, and
+  avoid overclaiming beyond the documented account and region coverage.

tests/vulnerable/aws-review-iac-only-org-scope-overclaim.yaml

@@ -0,0 +1,75 @@
+id: aws-review-iac-only-org-scope-overclaim
+skill: aws-review
+expected: vulnerable
+description: >
+  IaC-only review overclaims organization-wide AWS compliance from one security
+  account Terraform module and one regional analyzer.
+scenario:
+  review_scope:
+    claimed_scope: "all AWS accounts and all enabled regions"
+    supplied_evidence:
+      - path: terraform/security-account/cloudtrail.tf
+        source_type: iac-only
+        captured: "2026-06-01"
+      - path: terraform/security-account/access-analyzer.tf
+        source_type: iac-only
+        captured: "2026-06-01"
+    missing_evidence:
+      - aws_organizations_account_denominator
+      - delegated_administrator_export
+      - member_account_inclusion_export
+      - enabled_and_opt_in_region_inventory
+      - cloudtrail_get_trail_status_export
+      - cloudtrail_s3_bucket_policy_export
+      - cloudtrail_kms_key_policy_export
+      - cloudwatch_logs_integration_export
+      - access_analyzer_policy_validation_findings
+  terraform:
+    provider_region: us-east-1
+    resources:
+      - type: aws_cloudtrail
+        name: org
+        attributes:
+          is_organization_trail: true
+          is_multi_region_trail: true
+          enable_logging: true
+          kms_key_id: "arn:aws:kms:us-east-1:111122223333:key/cloudtrail"
+          cloud_watch_logs_group_arn: null
+      - type: aws_accessanalyzer_analyzer
+        name: account
+        attributes:
+          analyzer_name: account-analyzer
+          type: ACCOUNT
+  reported_findings:
+    - cis_id: "3.1"
+      title: "Ensure CloudTrail is enabled in all regions"
+      status: Pass
+      severity: Low
+      evidence_source: iac-only
+      scope_coverage: "organization-wide"
+      problem: "Treats one Terraform organization trail as proof of all member accounts and regions."
+    - cis_id: "1.20"
+      title: "Ensure IAM Access Analyzer is enabled for all regions"
+      status: Pass
+      severity: Low
+      evidence_source: iac-only
+      scope_coverage: "all-regions"
+      problem: "One ACCOUNT analyzer in us-east-1 is treated as all-region analyzer and policy-validation evidence."
+    - cis_id: "1.1"
+      title: "Maintain current contact details"
+      status: Pass
+      evidence_source: unknown
+      problem: "A live-only control is passed without account contact export evidence."
+should_trigger:
+  - AWS-EVID-SCOPE-01
+  - AWS-EVID-SCOPE-02
+  - AWS-EVID-SCOPE-03
+  - AWS-EVID-SCOPE-04
+  - AWS-EVID-SCOPE-05
+  - AWS-EVID-SCOPE-06
+expected_result: >
+  The skill should flag the assessment as overconfident. CloudTrail can only be
+  a scoped or Not Evaluable finding until organization, member-account, region,
+  bucket-policy, KMS-policy, and status evidence is supplied. Access Analyzer
+  deployment must be separated from IAM policy-validation evidence, and live-only
+  controls should use Not Evaluable reason codes.