Summary
Currently, the AWS Glue job entities are set to expire daily. It creates a problem if a glue job execution is not scheduled to be invoked weekly or monthly.
Desired Behavior
AWS Glue job entities should not expire and deleted daily if the glue job was not invoked for the last 24 hours. Ideally, they should expire only after a quarter.
Possible Solution
NOTE: # ( Increase the entityExpirationTime to Quarterly in AWS Glue job similar to the one configured for APM entities)
Additional context
This block us putting up a monitoring dashboard that involves monitoring AWS Glue job run details as most of the jobs are not triggerd daily.
Summary
Currently, the AWS Glue job entities are set to expire daily. It creates a problem if a glue job execution is not scheduled to be invoked weekly or monthly.
Desired Behavior
AWS Glue job entities should not expire and deleted daily if the glue job was not invoked for the last 24 hours. Ideally, they should expire only after a quarter.
Possible Solution
NOTE: # ( Increase the entityExpirationTime to Quarterly in AWS Glue job similar to the one configured for APM entities)
Additional context
This block us putting up a monitoring dashboard that involves monitoring AWS Glue job run details as most of the jobs are not triggerd daily.