[Enhancement]: Add state-first HPC upload ingestion flow with DB-backed dedupe parity

[tomvothecoder]

Is your feature request related to a problem?

For non-NERSC sites that must upload archives over HTTPS, SimBoard cannot currently follow the same state-first dedupe flow used by the NERSC path ingestor.

Current gaps:

• /api/v1/ingestions/state only reconstructs state from HPC_PATH ingestions.
• /api/v1/ingestions/from-upload does not accept caller-provided processed_execution_ids or stable case identity metadata.
• Upload ingestions are currently recorded as BROWSER_UPLOAD, not a dedicated HPC automation source.
• There is no upload-side automation runner that fetches DB-backed state before deciding which cases to submit.

Result: upload-based HPC automation cannot skip unchanged cases before ingestion, and cannot contribute back to the same DB-backed dedupe state model.

Describe the solution you'd like

Implement a dedicated HPC upload ingestion flow that mirrors NERSC path-ingestion behavior, using one case per upload request.

Proposed scope:

• Add a dedicated HPC upload ingestion contract or endpoint.
• Require one case archive per request plus stable case identity metadata.
• Accept processed_execution_ids from the upload-side automation job.
• Persist those ingestions as HPC_UPLOAD.
• Extend /api/v1/ingestions/state so it includes both HPC_PATH and HPC_UPLOAD rows.
• Add an upload-side automation runner that:
  • fetches /api/v1/ingestions/state
  • scans local archive contents
  • skips unchanged cases
  • uploads only changed cases
• Document the distinction between browser/manual upload and automated HPC upload.

Acceptance criteria:

• A service-account-driven upload workflow can fetch known execution state before upload.
• HPC upload requests can persist per-case processed_execution_ids needed for future dedupe decisions.
• /api/v1/ingestions/state reconstructs state from both HPC_PATH and HPC_UPLOAD records.
• Unchanged cases are skipped by the upload automation runner.
• Browser/manual upload behavior remains supported and clearly separated from HPC automation behavior.

Describe alternatives you've considered

1. Reuse current browser upload flow.
   This is not enough because the current upload contract does not carry state metadata and persists uploads as BROWSER_UPLOAD.

2. Support multi-case HPC upload requests.
   This would require a more complex persistence model because current ingestion audit rows only store one source_reference and one flat processed_execution_ids list, which is not sufficient for reconstructing per-case state from a multi-case upload.

3. Keep upload flow as-is and rely on duplicate detection during ingest.
   This preserves correctness at the execution_id level, but loses the main benefit of state-first dedupe: skipping unchanged cases before upload and ingestion.

Additional context

Recommendation: implement this in a separate PR from docs-only clarification work.

Reason for preferring one case per HPC upload request:

• Matches current state model keyed by case path.
• Minimizes schema and audit-model complexity.
• Keeps browser/manual multi-case uploads separate from HPC automation semantics.
• Lowest-risk path to parity with the existing NERSC runner.

[tomvothecoder]

This issue is the required follow-up to #189 for Chrysalis and other non-NERSC sites, which need upload-mode ingestion with DB-backed state lookup and pre-upload dedupe.

[TonyB9000]

@tomvothecoder Considering the materials found in the various USER directories, and (for the moment) ignoring "state", how does one identify and count "cases", as opposed to their (subordinate) execution_ids? I was taking "exec_ids" to correspond to "runs" - but can their be multiple COMPLETE runs (exec_ids) to a "case"? I need an ontology.

Also you wrote:

Add an upload-side automation runner that:

fetches /api/v1/ingestions/state
scans local archive contents
skips unchanged cases
uploads only changed cases

I assume "changed" includes "new" cases as well.

And in the event that a "case" contains multiple (new, complete) executions (runs?). you want ONE TGZ package for that "case".

My conception of PACE behavior was not "case-based", so this is a bit new - or else I misunderstood.

[tomvothecoder]

My conception of PACE behavior was not "case-based", so this is a bit new - or else I misunderstood.

Yes, PACE is only execution based while SimBoard is case-based. My understanding is that PACE expects unique execution ids. If you're finding repeating execution ids for COMPLETE executions (has all of the required files with required metadata), then that might be a problem for PACE. If those directories don't have all the required files, are empty, etc. then it's not an issue.

@tomvothecoder Considering the materials found in the various USER directories, and (for the moment) ignoring "state", how does one identify and count "cases", as opposed to their (subordinate) execution_ids? I was taking "exec_ids" to correspond to "runs" - but can their be multiple COMPLETE runs (exec_ids) to a "case"? I need an ontology.

Case is the top-level identifier that nests executions (aka jobs or runs).

Refer to the filesystem tree diagram below.

filesystem/
└── performance_archive/
    └── ac.golaz  								  <--- User (`hpc_username`)
	    └── v3.LR.historical_0121/ 		   <--- case (`case_name`)
	        └── 1081156.251218-200923/ <--- execution 1 (`execution_id`)
	            ├── e3sm_timing.001.001
	            ├── CaseStatus.001.gz
	            ├── GIT_DESCRIBE.001.gz
	            ├── GIT_CONFIG.001.gz
	            ├── GIT_STATUS.001.gz
	            └── CaseDocs/
	                ├── README.case.001.gz
	                ├── env_case.xml.001.gz
	                ├── env_build.xml.001.gz
	                └── env_run.xml.001.gz
 	        └── 1081156.251220-2001202/  <--- execution 2 (`execution_id`)
				....

It might if you go to on SimBoard to view a case page. Example: https://simboard-dev.e3sm.org/cases/a9191227-7b29-46e8-91d6-4a2fb074e66d.

I assume "changed" includes "new" cases as well.

Yes.

And in the event that a "case" contains multiple (new, complete) executions (runs?). you want ONE TGZ package for that "case".

One tar attached to one upload REST API request per execution (aka run), not the entire case. This ensures that if a case has new executions, the entire case doesn't need to be zipped again. It also ensures that each tar is < 50mb.