Motivation
Each stage automatically emits the recordsProcessed metric which may lead to zero values for stages that do not do any Spark reads. Avoid the zero values to not skew the dashboard which does a mean on the values returned for a given day.
Input
The sparkRecordsRead counter.
Output
No recordsProcessed custom metric is emitted for stages that does no reads.
Motivation
Each stage automatically emits the
recordsProcessedmetric which may lead to zero values for stages that do not do any Spark reads. Avoid the zero values to not skew the dashboard which does ameanon the values returned for a given day.Input
The
sparkRecordsReadcounter.Output
No
recordsProcessedcustom metric is emitted for stages that does no reads.