Skip to content

Improve validate_responses output: structured, machine-readable, more specific #660

Description

@Joe-Heffer-Shef

Problem

survey/management/commands/validate_responses.py reports validation failures too vaguely to be useful for triage or automation:

self.stderr.write(f"Survey {survey.pk} / Response {response.pk}: {message}")
...
self.stdout.write(f"Validated {total} responses - {errors} error(s)")

Issues with the current output:

  • No org/project context — only raw Survey {pk} / Response {pk} numbers are printed, so tracking down which organisation/project owns a bad response means a manual DB lookup every time.
  • No machine-readable format — output is free text on stdout/stderr, so it can't be piped into a dashboard, ticketing system, or CI job without fragile string parsing.
  • Only a flat totalerrors + total gives no breakdown by survey, by field, or by error type, so it's hard to tell if one survey config is systematically broken vs. scattered one-off bad records.
  • No response metadatacreated_at and other identifying info aren't included, making it hard to correlate with support tickets or know if a bad response is old/stale vs. newly submitted.
  • Unhandled non-ValidationError exceptions — if response.validate() raises anything else (e.g. a malformed survey.survey_config causing a schema-building error), the command crashes instead of reporting it and continuing.
  • No scoping — can't limit a run to one organisation/project/survey, which matters as the dataset grows.

Proposal

  • Add a --format {text,json} option. In JSON mode, emit one JSON object per invalid response (or a single JSON array) with fields such as: survey_id, survey_name, project_id, organisation_id, response_id, created_at, and an errors list of {path, message}. Machine-readable mode should go to stdout so it can be piped/redirected cleanly.
  • Include organisation and project name/id alongside the survey/response ids in text output, not just raw pks.
  • Add a summary breakdown: counts of errors grouped by survey (and ideally by failing field), not just a single total.
  • Catch unexpected exceptions per-response (e.g. schema build failures), report them distinctly from schema-validation failures, and continue processing rather than aborting the whole run.
  • Add filtering options (--survey, --project, --organisation) to scope validation runs.
  • Keep the non-zero exit code on any error for CI use, but make it distinguishable (e.g. different codes, or a --json summary with counts) so callers can branch on validation-errors vs unexpected-errors.

Context

Related to #657, which added JSON-path/field-label detail to the underlying ValidationError messages raised by SurveyResponse.validate(). This issue is about the management command's reporting layer built on top of those messages, not the validation logic itself.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions