Skip to content

Fix migrations for 10.5.1 -> 11.0.0 -> 11.0.1.#2237

Open
aaronweeden wants to merge 4 commits into
ubccr:xdmod11.0from
aaronweeden:fix-11.0-migrations
Open

Fix migrations for 10.5.1 -> 11.0.0 -> 11.0.1.#2237
aaronweeden wants to merge 4 commits into
ubccr:xdmod11.0from
aaronweeden:fix-11.0-migrations

Conversation

@aaronweeden

@aaronweeden aaronweeden commented May 28, 2026

Copy link
Copy Markdown
Contributor

Description

This PR does two main things:

  1. 4382225: Add CircleCI jobs for testing fresh install and upgrade with only the Jobs and Resource Specifications realms enabled using the new images created in https://tools-int-01.ccr.xdmod.org/git/UBCCR/ccr-private-xdmod/pulls/411. A bit of test refactoring is done in the process.
  2. 8662fc2: Fix a bug (exposed by the new CircleCI jobs) in which the upgrades from 10.5.1 to 11.0.0 and from 11.0.0 to 11.0.1 fail if the Cloud realm is not enabled.

Motivation and Context

Previously in #1986 I had mistakenly renamed some ETL migration pipelines such that they automatically ran during the upgrade rather than only conditionally running if the modw_cloud.event table existed; in that PR I had mistakenly assumed the names had typos and were intended to run automatically. As a consequence of this, if someone tries to upgrade along the path 10.5.1 -> 11.0.0 -> 11.0.1 without ever having ingested cloud data, it fails with the error message Schema 'modw_cloud' does not exist. This PR fixes this by renaming the pipelines back to names that do not match the filter used by Etlv2Migration (and which are not similar to the filter to reduce confusion).

Tests performed

I updated the CircleCI tests and confirmed that they succeeded and the correct number of tests ran.

Checklist:

  • The pull request description is suitable for a Changelog entry
  • The milestone is set correctly on the pull request
  • The appropriate labels have been added to the pull request

@aaronweeden aaronweeden changed the base branch from main to xdmod11.0 May 28, 2026 14:22
@aaronweeden aaronweeden force-pushed the fix-11.0-migrations branch from 3bfdc55 to ae669c1 Compare June 2, 2026 23:07
@aaronweeden aaronweeden changed the title Add separate CI pipeline for jobs realm only. Add separate CI pipeline for Jobs realm only. Jun 2, 2026
@aaronweeden aaronweeden added the bug Bugfixes label Jun 12, 2026
@aaronweeden aaronweeden force-pushed the fix-11.0-migrations branch from bb6b9df to 4382225 Compare June 16, 2026 04:24
@aaronweeden aaronweeden changed the title Add separate CI pipeline for Jobs realm only. Fix migrations for 10.5.1 -> 11.0.0 -> 11.0.1. Jun 16, 2026
@aaronweeden aaronweeden force-pushed the fix-11.0-migrations branch from 99fdb71 to 8662fc2 Compare June 17, 2026 18:52
@aaronweeden aaronweeden added this to the 11.0.4 milestone Jun 17, 2026
@aaronweeden aaronweeden marked this pull request as ready for review June 18, 2026 16:30
Comment thread tests/ci/bootstrap.sh Outdated
jq . /etc/xdmod/resource_specs2.json > /etc/xdmod/resource_specs.json
rm -f /etc/xdmod/resource_specs2.json

xdmod-ingestor

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Elsewhere in this file we run xdmod-ingestor as the xdmod user via sudo -u xdmod xdmod-ingestor. This will run as root - is this intended if so why?

Also is it really necessary to run xdmod-ingestor so many times? Why this extra run of it now?

@aaronweeden aaronweeden Jun 18, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Running as root was not intentional; I have fixed this.

The changes to resources.json and resource_specs.json above this line are testing these instructions which specify to run xdmod-ingestor after making changes to the files. Not running xdmod-ingestor here makes the regression tests fail because the resource allocation types aren't ingested yet. It isn't necessary to run xdmod-ingestor again if the Storage realm is being tested because the --datatype storage option makes it run the staging-ingest-common pipeline that includes ingesting the resources.json file. Since the call I'm adding to xdmod-ingestor only needs to happen if Storage isn't being tested, I've moved it to the else below.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Bugfixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants