Skip to content

Noora health long job timeout failure #216

Description

@Ishankoradia

One job of noora health configured has run multiple airbyte syncs (~ 20), which run sequentially one after the other.

Some observations from the recent failures

  • Failure can occure at any random step. Usually after a few steps are over.
  • The logs in prefect show Task run failed with exception: ReadTimeout('') - Retries are exhausted.
  • The step at which the job fails, seems to have executed successfully in airbyte. Saw this by comparing prefect and airbyte timestamps. Which means, prefect successfully triggered the connection but might have lost the connection to poll on. Found this log message in one of the airbyte workers 2025-08-23 19:47:16,821 [scheduler-executor-thread-6] INFO i.a.c.s.h.SchedulerHandler(createJob):644 - New job created, with id: 44003 . This is the id of the step that failed according to prefect.
  • From the error stack, i can see that this line 388 in prefect_airybte is the call where it failed. Maybe that api is timing out.
Image
  • Works when retrying.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions