Survey Grouping and Bulk Creation

[sjmf]

Motivation

For our upcoming survey, it would be useful to create surveys in bulk using per-institution / per-discipline participant lists. This would enable fast creation of large numbers of surveys from a single source of truth, eliminating the need to repetitively upload participants, create a survey for them, select their institution from the list, and hit create, again and again. Further, managing large numbers of surveys is likely difficult, given the paging system loads 10 at a time.

Proposal

Three features are proposed to address this scenario:

1. Allow the bulk upload of participant data from per-institution Excel spreadsheets.
2. Allow bulk survey creation: creating a large number of surveys from a selected list of institutions
3. Group bulk-uploaded surveys under a single row in the dashboard, with a dropdown and sub-pager

This is a non-trivial feature to implement, touching several parts of the code.

Implementation Plan

User interface

react-app/src/pages/dashboard
Add "bulk upload" and "bulk create" menu entries under the "Survey Tools" section. Keep the existing three-across button setup seen currently, but addfourth and fifth half-width buttons underneath for these entries. Slightly reduce the height and font-size of the existing three buttons in CSS to accommodate.

react-app/src/components
Add a new "BulkUpload.jsx" component that renders the modal dialog. We can use the same style as CreateForm.jsx, and actually some of its logic, too.

• Optionally hive off duplicated logic into shared components.

1. Bulk Upload

New component: BulkUpload.jsx

Page 1:

• The BulkUpload interface presents a multi-file upload dialog, which allows the user to either use the system file-picker dialog, or drag+drop multiple files at once for upload and processing into the database.
• We also need to assign institutions to each spreadsheet in this file-upload view: Each excel sheet, as per the AddParticipants component, needs a single institution to be assigned to it. Show each file in its own row by filename, with an institution picker next to it for assignment.
• Upload happens on-demand (i.e. as files are uploaded, after institution is assigned and confirmed via a simple button (checkmark))
• Processing is separate in the Django backend: clicking the button "Next" at the bottom of the dialog kicks off a deferred processing task, reading the files into the database using the UploadParticipantsView viewset logic, in a loop.

Page 2:

• On the second page, a spinner is shown until the deferred processing task completes. We can use a polling loop to check the process of this task using a uuid as a task identifier.
• Errors in this process should be reported on aggregate, i.e. with a return from the path with JSON representing the filenames of the files which encountered issues, prompting the user to fix them and re-upload.
• All files should be attempted before returning with a "completed" state. On error, we can show a warning on those specific files.

Page 3:

• Participants are now uploaded and assigned to institutions, parsed into the database. Show a confirmation. Return to the survey dashboard.

2. Bulk Create

New component: BulkCreate.jsx

The bulk survey creation dialog owns the logic for guiding the user through bulk survey creation.

• The Bulk Create button opens up a modal with the title "Create Many Surveys"
• Optionally: Extract any shared components from CreateForm.jsx
• This dialog is notionally similar to CreateForm.jsx, except it allows the user to select a list of institutions from a picker component
• The picker component loads the list of institutions from the database, and presents them in rows. It supports both clicking to pick and shift+click to select multiple institutions. It uses SurveyInstitutionViewSet.list on the backend.
• Submitting requires a new view on the backend to support the logic of survey creation in-bulk.

3. Survey Grouping

Backend:

• Surveys created in bulk need a new identifying marker on their data model.
• We can modify Survey to support this, adding a new integer column, group, which is a Foreign Key to...
• a new SurveyGroups table to handle grouped surveys.

The survey pager will also have to be modified to return a sub-object for grouped surveys. The behaviour I am planning is that we return 10 surveys for each grouped survey, and a route to request another page given a SurveyGroup id. The UI will handle sub-paging.

Frontend:

• SurveyDefinitionsProvider will need to support the new grouping format.
• We will need to consider how to render grouped surveys in the Dashboard component: visually, grouped surveys should be differentiated somehow, with an option to drop down their table rows showing a sub-table with those surveys.
• All sub-surveys will share the same title, so this only needs to be shown on the top-level survey.
• Paging buttons will need to be shown within the dropped-down table, allowing paging to the next 10 of that specific survey.

[sjmf]

Disregard the linked PR. Reproducing the output I wanted from Co-Pilot here. Poor quality Microsoft AI strikes again: this comment is all I wanted from it.

Great plan overall — one important gap is operational safety: please define idempotency/duplicate handling (re-uploaded files, retried bulk-create), transaction boundaries and rollback semantics for partial failures, and audit/permission controls for batch actions. Also please spell out migration + backward compatibility for introducing SurveyGroup (existing surveys, API/pagination response shape, and exports) so current dashboard behavior doesn’t break.

To define:

• Operational safety:
  • idempotency and duplicate handling for re-uploads/retries
  • transaction boundaries and rollback behavior for partial failures
  • auditability and permission controls for bulk operations
• Migration/backward-compatibility:
  • handling existing surveys when introducing SurveyGroup
  • API/pagination response-shape compatibility
  • export/dashboard behavior stability