Merge db voters and votes collections

[dsernst]

Work to merge the voters and votes collections, which was lingering technical debt. See #152 (comment).

This will unblock us to properly finish link-via-auth.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
siv	❌ Failed (Inspect)		Jun 18, 2025 5:45am

[dsernst]

INITIAL NOTES:

We have this new db architecture design...
- merge voters and votes into approved-voters.

• We need the application code for the new design.
  • Update POST /api/add-voters
  • GET /api/check-auth-token
  • Update POST /api/submit-vote
    • anything that reads the current list of votes or voters:
      • GET /api/accepted-votes
      • POST /api/unlock-votes
      • POST /api/invalidates-votes
        • POST /unlock filter out invalidated votes
        • GET /invalidated-votes
        • GET /was-invalidated
    • grep collection('votes')
      • 19 hits (7 are scripts)
    • grep collection('voters')
      • 20 hits (4 are scripts)
    • grep for collection('invalidated_votes')

Migration Script

• We need a migration script to convert the current database data into the new format.
  • We need to write the migration script
  • We need to test the migration script
    • Maybe want backup in case the migration script corrupts data?
  • We need to run the migration script for production data
  • Do we want to provide instructions / migration script completion checks for other people running SIV...?
    • We decided for now this is Nice-to-have, but not Must-have, so not blocking

Is this a SemVer breaking change?

• siv-web (codebase). Current version: 0.1.0 [according to package.json]
  • This would a SemVer breaking change, but because we are in 0.x pre-release, we're technically not really promising that we're documenting these breaking changes right now, anyway.
• SIV Protocol https://docs.siv.org/technical-specifications
  • evaluated this is not a breaking change for this SIV Protocol document.

GAME PLAN:

• backup all the production data - Initialized: Sep 13, 2024, 10:32:06 PM

• update the production data for the new application code shape
  - using DB atomic transactions
  - [x] sketched out high-level overview of this script 2024-09-13-merge-db-voters-and-votes-collections.ts
  - [ ] db system to track running new migration scripts just once
  - [x] overview of what that would look like

• update all the application code

STRATEGY:

1. Migration script to update just a single election for the new shape.
2. Get all the application code working locally for that one election.
3. Update the migration script to be able to work for all elections in the DB.
4. Deploy the new application code.
5. Run the migration script over all the old elections.

Migration script strategy:

1. db.migrations - Track which migration scripts have already been run
   • unique per deployment (e.g. production vs different people's personal dev environments)

title (string)      | run_on (timestamp)
----------------------------------------
2024-09-13-merge... | 2024-11-10

2. run-migrations script (which could run whenever SIV starts up, e.g. post-install hook, or manually triggered by admin)
   grab list from db.migrations table
   grab files in /migrations folder

   Are there any new migration script files that the DB doesn't have a record of running already?
   if so, run them.
   ...if succeeded, store their success time in db.migrations

[dsernst]

• 592382c probably introduced a merge-conflict, will need to adapt to this new db schema

[dsernst]

WHAT WE NEED NEXT:

• An import script to take a backup, and load it into a fresh clean Firebase DB.

Where are we at with the DB migration?

1. We have a pretty good seeming migration script
2. We now have some scaffolding around the migration script
3. Clarity in software versioning that this is a breaking change. (Made main 1.0, and this branch 2.0— semver major)
4. We now have a much easier to use backup tool. For safety before running migrations.
5. Reading data from new backup tool is now confirmed to work:

// db-data/test-reading-data.ts
import { admins } from './local-backup/2025-06-18T06:05'

console.log(Object.keys(admins).length)

Still missing?

• Restoring from backup not written
• Possible the backup is missing data? Or formatting got messed up?

I think there will be some elections with edge cases that will break the migration—

• Unclear what happens to the migration if it fails partway through.

Possible ways to apply the migration:

• Run a single migration script that updates all data on postinstall
  • a) and deletes the old data
  • b) doesn't delete the old data, later script can do so once it's confirmed the new format works
• Keep application code to support both formats (backwards compatibility), detect format to apply correct logic. create new elections in new format.
• Run migration script for each election whenever it is first loaded.
• Take down server for "scheduled maintenance".
• Make sure migration script has Rollback logic to undo it in case something goes wrong.
• Manually apply migration, halting whenever it encounters an error. Fix errors and resume migration from where it left off, incrementally improving the script to handle more edge cases.

Considerations:

• User facing downtime
• Ability to rollback from errors
• Ease for other developers to apply migration
• Ease for other production instances to apply migration
• Amount of work involved
• "Responsible" & "Reliable" choices
• Sets us up for easier future migrations

See spreadsheet for considerations analyzed https://docs.google.com/spreadsheets/d/17GsGYF4OSh3vcsvp4x3iz27dlXR1_K4xHkQ9497BQLA/edit?gid=0#gid=0

Favored strategy to pursue

• 1. Backup Production data locally
• 2. PUSH local backup into a new Firestore DB instance
  • Requires new restore tool
• 3. Try applying our migration script to the new, isolated instance. (the production site still points at the original firebase db, untouched)
• 4. That lets us work out the kinks in the migration script, make sure it's working.
• 5. Locally test new application code is working well with the new post-migration DB.
• 6. Finally, once everything seems good, we can point production siv.org app to this new, migrated DB.
• 7. Celebrate 🎉

[arianabuilds]

@dsernst how urgent is this from 0-10 (most)?

[dsernst]

It's not blocking anything, it's just tech debt. I would say like 4 out of 10? It will be nice when we have a better system in place for this specific change and these sorts of deeper migrations more generally