Fix migrations for 10.5.1 -> 11.0.0 -> 11.0.1.#2237
Conversation
3bfdc55 to
ae669c1
Compare
bb6b9df to
4382225
Compare
99fdb71 to
8662fc2
Compare
| jq . /etc/xdmod/resource_specs2.json > /etc/xdmod/resource_specs.json | ||
| rm -f /etc/xdmod/resource_specs2.json | ||
|
|
||
| xdmod-ingestor |
There was a problem hiding this comment.
Elsewhere in this file we run xdmod-ingestor as the xdmod user via sudo -u xdmod xdmod-ingestor. This will run as root - is this intended if so why?
Also is it really necessary to run xdmod-ingestor so many times? Why this extra run of it now?
There was a problem hiding this comment.
Running as root was not intentional; I have fixed this.
The changes to resources.json and resource_specs.json above this line are testing these instructions which specify to run xdmod-ingestor after making changes to the files. Not running xdmod-ingestor here makes the regression tests fail because the resource allocation types aren't ingested yet. It isn't necessary to run xdmod-ingestor again if the Storage realm is being tested because the --datatype storage option makes it run the staging-ingest-common pipeline that includes ingesting the resources.json file. Since the call I'm adding to xdmod-ingestor only needs to happen if Storage isn't being tested, I've moved it to the else below.
Description
This PR does two main things:
Motivation and Context
Previously in #1986 I had mistakenly renamed some ETL migration pipelines such that they automatically ran during the upgrade rather than only conditionally running if the
modw_cloud.eventtable existed; in that PR I had mistakenly assumed the names had typos and were intended to run automatically. As a consequence of this, if someone tries to upgrade along the path 10.5.1 -> 11.0.0 -> 11.0.1 without ever having ingested cloud data, it fails with the error messageSchema 'modw_cloud' does not exist. This PR fixes this by renaming the pipelines back to names that do not match the filter used byEtlv2Migration(and which are not similar to the filter to reduce confusion).Tests performed
I updated the CircleCI tests and confirmed that they succeeded and the correct number of tests ran.
Checklist: