Skip to content

feat(observability): add structured logging to JobQueryService and enhance orphan recovery logs #151

Description

@kvithayathil

Context

The worker orphan recovery feature (#129) added retry_job() to JobQueryService and _recover_orphaned_jobs() to the worker loop. While worker.py already uses structlog extensively (20+ log calls), job_query_service.py has zero logging — no import, no logger, no log calls.

Gap

JobQueryService (no logging at all)

  • retry_job() — no log when a job is retried (who triggered it, previous state, job ID)
  • cancel_job() — no log when a job is cancelled
  • start_job() — no log when a job is started
  • create_job() — no log when a job is created

Worker orphan recovery (partial logging)

  • _recover_orphaned_jobs_with_session() logs only the count ("Periodic orphan recovery complete", count=recovered) — does not log individual job recoveries (previous status, job ID, stale duration)
  • _terminate_orphans_with_session() has no logging for individual terminations

Proposed scope

  1. Add structlog to job_query_service.py — log all state-changing operations (create, start, cancel, retry) with job ID, previous status, new status
  2. Enhance _recover_orphaned_jobs_with_session() in worker.py — log each recovered job with job ID, previous status, stale duration
  3. Enhance _terminate_orphans_with_session() — log each terminated orphan with job ID, previous status, termination reason

Why separate from #129

  • fix(worker): runtime orphan recovery for stuck OCR jobs #129 is already substantial (7 files, +722 lines, 3 commits)
  • Logging is a cross-cutting observability concern that applies to the entire service, not just retry
  • Deserves its own design discussion (log levels, fields, PII considerations)
  • Original plan was scoped to recovery mechanism only

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:backendBackend (Python/FastAPI) changesenhancementNew feature or requestobservabilityLogging, monitoring, tracing, alertingpriority:mediumNormal priority

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions