Customer ETL pipeline failing with OutOfMemoryError during nightly batch job

[archdata-ai]

Description

The customer ETL pipeline is consistently failing with an OutOfMemoryError during the nightly batch processing job.

Steps to Reproduce

1. Wait for the scheduled nightly batch job to run at 2:00 AM
2. Monitor the ETL pipeline execution
3. Observe the failure with OOM error

Expected Behavior

The ETL pipeline should complete successfully without memory-related failures during the nightly batch job.

Actual Behavior

The pipeline fails with the following error:

java.lang.OutOfMemoryError

Additional Context

• Frequency: Occurs daily at 2:00 AM
• Job Type: Nightly batch processing
• Error Type: java.lang.OutOfMemoryError
• Impact: Customer ETL pipeline is unable to complete processing

Potential Investigation Areas

• Memory allocation settings for the ETL job
• Data volume changes that might be causing increased memory usage
• Memory leaks in the pipeline code
• JVM heap size configuration