Hey Christos,
I came across your Otel commits in Apache Airflow while doing hypothesis research for my own project and thought I’d reach out.
For background, I’m building a Rust monitoring library for scientific data pipelines and talking to experienced engineers across different industries to understand whether the monitoring challenges we see in scientific workloads also show up in general data engineering.
Two quick questions:
-
Is it true that as a data engineer you end up using observability solutions like Grafana, DataDog alongside DAG Pipeline Monitoring UIs (Astronomer, Dagster), since they handle control flow but don’t reveal infra behavior or runtime hotspots?
-
In your experience, do observability tools fall short because they aren’t pipeline-aware, making it hard to reconstruct workflows from loose server instances, and leaving correlation/debugging mostly manual?Even a short reply would help. Thanks!
Thanks,
Vincent
For more information, our repository is: https://github.com/Tracer-Cloud/tracer-client
Hey Christos,
I came across your Otel commits in Apache Airflow while doing hypothesis research for my own project and thought I’d reach out.
For background, I’m building a Rust monitoring library for scientific data pipelines and talking to experienced engineers across different industries to understand whether the monitoring challenges we see in scientific workloads also show up in general data engineering.
Two quick questions:
Is it true that as a data engineer you end up using observability solutions like Grafana, DataDog alongside DAG Pipeline Monitoring UIs (Astronomer, Dagster), since they handle control flow but don’t reveal infra behavior or runtime hotspots?
In your experience, do observability tools fall short because they aren’t pipeline-aware, making it hard to reconstruct workflows from loose server instances, and leaving correlation/debugging mostly manual?Even a short reply would help. Thanks!
Thanks,
Vincent
For more information, our repository is: https://github.com/Tracer-Cloud/tracer-client