Fix s3_lifecycle to work with new and old syntax of lifecycle | fix idempotency | fix duplicate lifecycle rule per execution

[rfakit]

Fix: Prevent S3 Lifecycle Policy Duplication When rule_id is Specified

Problem Description

The s3_lifecycle Ansible module was creating duplicate lifecycle rules on every playbook run when a rule_id was specified in the configuration. This occurred because the module only matched existing rules by their ID, and when existing rules had auto-generated IDs (which S3 creates when no ID is provided), they were not recognized as matches, causing new rules to be created on each execution.

Symptoms

• Multiple duplicate lifecycle rules with identical configuration but different IDs
• Rules accumulating in S3 buckets over time (e.g., 4+ identical rules)
• Playbook showing changed: true on every run even when the desired state already exists
• Example output showing multiple rules with same expiration, prefix, and status but different IDs:

  {
    "Rules": [
      {"ID": "5voiavj06zkjjrhwud205tkaxriivj2om0k82qa1c3l52zwo", "Expiration": {"Days": 1}, "Prefix": "", "Status": "Enabled"},
      {"ID": "ID-1", "Expiration": {"Days": 1}, "Prefix": "", "Status": "Enabled"},
      {"ID": "us944tgwgbxl52sjwt6mudd0ibpzf7wx6q6zeug8dx5ekjdh", "Expiration": {"Days": 1}, "Prefix": "", "Status": "Enabled"},
      {"ID": "w09je6ajineekl5yy5adz9b7d1sw68nq1g5rtaw3n2k8qla8", "Expiration": {"Days": 1}, "Prefix": "", "Status": "Enabled"}
    ]
  }

Root Cause Analysis

1. ID-Only Matching: The module's compare_and_update_configuration() function only attempted to match existing rules by their ID. When a rule_id was provided (e.g., "ID-1"), it would look for an existing rule with that exact ID. If no match was found (because existing rules had auto-generated IDs), it would create a new rule.

2. Format Incompatibility: S3 lifecycle rules can exist in two formats:

   • Old format: {"Prefix": "", "Status": "Enabled", ...} (Prefix directly on the rule)
   • New format: {"Filter": {"Prefix": ""}, "Status": "Enabled", ...} (Filter object)

   The module's comparison logic didn't handle both formats consistently, causing mismatches even when rules were functionally identical.

3. No Duplicate Cleanup: When duplicate rules were created, the module didn't remove existing duplicates, only added new ones.

Solution

The fix implements a multi-layered approach:

1. Rule Format Normalization: Added normalize_rule_format() to convert all rules to a consistent format (new Filter format) when fetched from S3, ensuring consistent comparison.

2. Filter Extraction Helper: Added get_rule_filter() to extract filter information from rules regardless of whether they use the old or new format.

3. Enhanced Matching Logic: Updated compare_and_update_configuration() to:

   • First attempt to match by ID (existing behavior)
   • If no ID match is found and rule_id is provided, match by filter (prefix) and rule content (expiration, status, transitions, etc.)
   • When a match is found, update the existing rule with the specified rule_id instead of creating a new one
   • Remove all duplicate rules that match the same filter and content

4. Consistent Filter Handling: Updated all comparison functions to use the new get_rule_filter() helper for consistent filter extraction.

Changes Made

New Functions

1. normalize_rule_format(rule) (lines 290-306)

   • Converts old S3 lifecycle rule format (Prefix directly) to new format (Filter object)
   • Ensures all rules have a consistent structure for comparison
   • Handles edge cases where Filter might be missing or malformed

2. get_rule_filter(rule) (lines 310-318)

   • Extracts filter information from rules, handling both old and new formats
   • Returns a consistent Filter object structure for comparison
   • Used throughout the module for consistent filter extraction

Modified Functions

1. fetch_rules() (lines 272-286)

   • Added normalization step to convert all fetched rules to Filter format
   • Ensures consistent rule structure before comparison

2. compare_and_update_configuration() (lines 429-512)

   • Updated to use get_rule_filter() for consistent filter extraction
   • Added fallback matching logic: when rule_id is provided but no ID match is found, match by filter and content
   • Added duplicate removal: when a matching rule is found, all other duplicate rules (matching filter and content) are removed
   • Prevents creation of new rules when functionally identical rules already exist

3. compare_and_remove_rule() (lines 554-567)

   • Updated to use get_rule_filter() for consistent filter extraction
   • Handles both old and new rule formats when removing rules

Benefits

1. Idempotency: Playbooks now run idempotently - no changes when the desired state already exists
2. Duplicate Prevention: Prevents accumulation of duplicate rules in S3 buckets
3. Backward Compatibility: Works with both old and new S3 lifecycle rule formats
4. Automatic Cleanup: Removes existing duplicate rules when a match is found
5. Consistent Behavior: Handles rules consistently regardless of their original format

Testing

Before Fix

• Running playbook multiple times created 4+ duplicate rules
• Each run showed changed: true even when rules already existed
• Bucket had multiple rules with identical configuration

After Fix

• Running playbook multiple times maintains exactly one rule with the specified rule_id
• Subsequent runs show changed: false when the rule already exists correctly
• All duplicate rules are removed and consolidated into a single rule

Test Case

# Configuration
lifecycle_policy:
  rule_id: "ID-1"
  expiration_days: 1
  prefix: ""
  status: "enabled"
  state: "present"

Expected Behavior:

• First run: Creates rule with ID "ID-1"
• Subsequent runs: Updates existing rule (if needed) or reports no changes
• No duplicate rules are created

Files Changed

• plugins/library/s3_lifecycle.py
  • Added normalize_rule_format() function
  • Added get_rule_filter() function
  • Modified fetch_rules() to normalize rules
  • Enhanced compare_and_update_configuration() with fallback matching and duplicate removal
  • Updated compare_and_remove_rule() to use consistent filter extraction

Impact

• Low Risk: Changes are backward compatible and only enhance existing functionality
• No Breaking Changes: All existing configurations continue to work
• Improves Reliability: Prevents rule accumulation and ensures idempotent playbook runs

Related Issues

This fix addresses the issue where S3 lifecycle policies with specified rule_id values were being recreated on every playbook execution, leading to rule duplication in S3 buckets.

[softwarefactory-project-zuul]

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/290099919ae34a90a5b2ef963242faad

✔️ ansible-galaxy-importer SUCCESS in 4m 58s (non-voting)
✔️ build-ansible-collection SUCCESS in 11m 17s
✔️ ansible-test-splitter SUCCESS in 4m 21s
✔️ integration-community.aws-1 SUCCESS in 23m 02s
Skipped 21 jobs