Skip to content

Do not emit recordsProcessed for records where no Spark reads where done #16

Description

@jonas
Motivation

Each stage automatically emits the recordsProcessed metric which may lead to zero values for stages that do not do any Spark reads. Avoid the zero values to not skew the dashboard which does a mean on the values returned for a given day.

Input

The sparkRecordsRead counter.

Output

No recordsProcessed custom metric is emitted for stages that does no reads.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions