test: reduce storage and transmission costs by compressing large noti…

[Osuochasam]

…fication payloads before processing/storing them, and decompressing them upon retrieval

🚀 Overview: Payload Compression for Large Notifications

This work addresses high storage and transmission costs by introducing an automated compression mechanism for large notification payloads before they are processed and stored.

By compressing data at the ingestion layer and decompressing it on retrieval, we significantly optimize our database footprint and minimize network overhead while ensuring total data integrity.

---

🛠️ Proposed Changes

1. Core Compression Utility

• Threshold-Based Execution: Implemented a utility that evaluates payload size and applies compression only if the payload exceeds a specific threshold (e.g., 10KB). This avoids unnecessary CPU overhead on small, standard notifications.
• Algorithm Selection: Used a standard, high-efficiency algorithm optimized for text/JSON data structures to maximize data reduction.

2. Pipeline Integration & Retrieval

• Before Processing: Intercepted the notification ingestion flow to compress qualified payloads right before they hit the data layer.
• On Retrieval: Integrated automatic decompression during data fetching, seamlessly restoring payloads to their original format before passing them to downstream services.

3. Graceful Fallbacks & Backward Compatibility

• Wrapped decompression logic in strict safety blocks to gracefully handle legacy data.
• If a payload is already uncompressed (old data) or unsupported, the system automatically falls back to raw data processing instead of throwing fatal errors or crashing the pipeline.

4. Efficiency Metrics & Monitoring

• Added inline logging to capture performance telemetry, tracking Original Payload Size, Compressed Size, and the overall Space Reduction Percentage for future auditing.

---

✅ Acceptance Criteria Checklist

• Payload Reduction: Large payloads are verified to occupy significantly less storage space.
• Data Integrity: Compression followed by decompression perfectly restores the original payload with zero data loss.
• Backward Compatibility: Legacy (uncompressed) notifications already in the system are read and processed safely.
• Error Resilience: Malformed or corrupted inputs fail safely without breaking active application threads.

---

🧪 Testing Plan

• Unit Tests: Added tests to verify successful compression/decompression, safe pass-through of sub-threshold payloads, and correct handling of corrupted or legacy strings.
• Integration Tests: Validated the complete end-to-end pipeline hook from the initial notification trigger down to the storage and retrieval layers.

closes #308

[drips-wave]

@Osuochasam Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits