Problem
survey/management/commands/validate_responses.py reports validation failures too vaguely to be useful for triage or automation:
self.stderr.write(f"Survey {survey.pk} / Response {response.pk}: {message}")
...
self.stdout.write(f"Validated {total} responses - {errors} error(s)")
Issues with the current output:
- No org/project context — only raw
Survey {pk} / Response {pk} numbers are printed, so tracking down which organisation/project owns a bad response means a manual DB lookup every time.
- No machine-readable format — output is free text on stdout/stderr, so it can't be piped into a dashboard, ticketing system, or CI job without fragile string parsing.
- Only a flat total —
errors + total gives no breakdown by survey, by field, or by error type, so it's hard to tell if one survey config is systematically broken vs. scattered one-off bad records.
- No response metadata —
created_at and other identifying info aren't included, making it hard to correlate with support tickets or know if a bad response is old/stale vs. newly submitted.
- Unhandled non-
ValidationError exceptions — if response.validate() raises anything else (e.g. a malformed survey.survey_config causing a schema-building error), the command crashes instead of reporting it and continuing.
- No scoping — can't limit a run to one organisation/project/survey, which matters as the dataset grows.
Proposal
- Add a
--format {text,json} option. In JSON mode, emit one JSON object per invalid response (or a single JSON array) with fields such as: survey_id, survey_name, project_id, organisation_id, response_id, created_at, and an errors list of {path, message}. Machine-readable mode should go to stdout so it can be piped/redirected cleanly.
- Include organisation and project name/id alongside the survey/response ids in text output, not just raw pks.
- Add a summary breakdown: counts of errors grouped by survey (and ideally by failing field), not just a single total.
- Catch unexpected exceptions per-response (e.g. schema build failures), report them distinctly from schema-validation failures, and continue processing rather than aborting the whole run.
- Add filtering options (
--survey, --project, --organisation) to scope validation runs.
- Keep the non-zero exit code on any error for CI use, but make it distinguishable (e.g. different codes, or a
--json summary with counts) so callers can branch on validation-errors vs unexpected-errors.
Context
Related to #657, which added JSON-path/field-label detail to the underlying ValidationError messages raised by SurveyResponse.validate(). This issue is about the management command's reporting layer built on top of those messages, not the validation logic itself.
Problem
survey/management/commands/validate_responses.pyreports validation failures too vaguely to be useful for triage or automation:Issues with the current output:
Survey {pk}/Response {pk}numbers are printed, so tracking down which organisation/project owns a bad response means a manual DB lookup every time.errors + totalgives no breakdown by survey, by field, or by error type, so it's hard to tell if one survey config is systematically broken vs. scattered one-off bad records.created_atand other identifying info aren't included, making it hard to correlate with support tickets or know if a bad response is old/stale vs. newly submitted.ValidationErrorexceptions — ifresponse.validate()raises anything else (e.g. a malformedsurvey.survey_configcausing a schema-building error), the command crashes instead of reporting it and continuing.Proposal
--format {text,json}option. In JSON mode, emit one JSON object per invalid response (or a single JSON array) with fields such as:survey_id,survey_name,project_id,organisation_id,response_id,created_at, and anerrorslist of{path, message}. Machine-readable mode should go to stdout so it can be piped/redirected cleanly.--survey,--project,--organisation) to scope validation runs.--jsonsummary with counts) so callers can branch on validation-errors vs unexpected-errors.Context
Related to #657, which added JSON-path/field-label detail to the underlying
ValidationErrormessages raised bySurveyResponse.validate(). This issue is about the management command's reporting layer built on top of those messages, not the validation logic itself.