Fix #5989, #6053, #6059, #6060, #6097, #6200: Rebuild app & WorkManager initialization#6062
Fix #5989, #6053, #6059, #6060, #6097, #6200: Rebuild app & WorkManager initialization#6062BenHenning wants to merge 31 commits into
WorkManager initialization#6062Conversation
This includes substantial changes to how the app starts up in all cases, but especially for workers. The new approach is robust against worker changes (including class renames or input changes), centralizes intialization so that all workers always run in a sane environment, and ensures all app initialization is done in a completely robust and safe way. Also, re-write much of the work manager wiki page to include new design details and significant debugging recommendations. Testing, code, and wiki documentation are not yet complete. This also has only been tested on the developer build of the app, so more validation will be necessary.
|
Thanks for submitting this pull request! Some main reviewers have taken time off for the next few weeks, so it may take a little while before we can look at this PR. We appreciate your patience while some of our team members recharge. We'll be fully returning on 05 January 2026. |
|
Note that this is probably in a good enough place for me to work off of it to make progress on #3506 (which was my original goal heading into this), and I'll need to circle back to this PR. It still needs a lot of documentation and testing work, including figuring out hopefully a better testing strategy than using fakes (I'm hoping it's possible to actually properly orchestrate kicking off the jobs but I have a lot more digging to do in order to figure that out fully). |
|
Hi @BenHenning, I'm going to mark this PR as stale because it hasn't had any updates for 7 days. If no further activity occurs within 7 days, it will be automatically closed so that others can take up the issue. |
|
Hi @BenHenning, I'm going to mark this PR as stale because it hasn't had any updates for 7 days. If no further activity occurs within 7 days, it will be automatically closed so that others can take up the issue. |
Some previous code was missed when converting the jobs, and there were some API simplifications possible. Also, this ensures that platform parameter syncing intervals can't be tweaked to the point of essentially bricking app syncing.
This is needed for testing, though this will likely not be a permanent upgrade since it requires Kotlin 1.8. For proof-of-concept purposes this also includes ignoring the metadata check so that tests can run with 1.8 dependencies in the 1.6 environment.
These may not land in this PR but they were discovered as gaps when trying to make work manager synchronization work in tests.
This includes some visibility changes as well as alterations to TestPlatformParameterModule to ensure parameters can actually be initialized correctly with the work manager pathway. This commit essentially establishes the baseline for work manager job testing and will be used as the foundation for adding tests for all of the other work manager jobs.
This introduces new utilities to simplify testing with WorkManager and one of which (the mixin) actually provides tests with the ability to test multiple rounds of jobs with just time management by leveraging a custom scheduler (or, rather, a simple self-scheduler to simulate WorkManager).
Also refined some of the shared utilities including the fake performance logger.
Also a bunch more shared utility refinement, plus some integration tests for MetricLogSchedulingWorker in how it relates to LogUploadWorker.
Not much to add yet since downloading platform parameters isn't fully supported yet.
Still many more to go, but this is a good start.
Conflicts: domain/src/main/java/org/oppia/android/domain/oppialogger/analytics/ApplicationLifecycleObserver.kt domain/src/main/java/org/oppia/android/domain/oppialogger/analytics/LearnerAnalyticsLogger.kt
This makes the new test synching mechanism work correctly for UI tests (which use a different sync-first pathway).
This fixes the dispatcher logic to actually behave correctly. This will be moved outside of the PR so that it can be more iterated.
@Multibinds should also be exempted.
These are all managed through the central test utilities, now.
This isolates test WorkManager interactions to just a single utility now (instead of being across multiple tests and multiple utilities).
The new version is hopefully much more streamlined and resilient to user error.
|
Not stale. |
|
@coderabbitai review |
|
@coderabbitai full review |
|
Hi @BenHenning, I'm going to mark this PR as stale because it hasn't had any updates for 7 days. If no further activity occurs within 7 days, it will be automatically closed so that others can take up the issue. |
APK & AAB differences analysisNote that this is a summarized snapshot. See the CI artifacts for detailed differences. DevExpand to see flavor specificsUniversal APKAPK file size: 19 MiB (old), 19 MiB (new), 10 KiB (Added) APK download size (estimated): 18 MiB (old), 18 MiB (new), 4699 bytes (Added) Method count: 265520 (old), 265395 (new), 125 (Removed) Features: 1 (old), 1 (new), 0 (No change) Permissions: 6 (old), 6 (new), 0 (No change) Resources: 7119 (old), 7118 (new), 1 (Removed)
Lesson assets: 113 (old), 113 (new), 0 (No change) AAB differencesExpand to see AAB specificsSupported configurations:
Base APKAPK file size: 19 MiB (old), 19 MiB (new), 10 KiB (Added)
Configuration hdpiAPK file size: 50 KiB (old), 50 KiB (new), 0 bytes (No change) Configuration ldpiAPK file size: 49 KiB (old), 49 KiB (new), 0 bytes (No change) Configuration mdpiAPK file size: 46 KiB (old), 46 KiB (new), 0 bytes (No change) Configuration tvdpiAPK file size: 86 KiB (old), 86 KiB (new), 0 bytes (No change) Configuration xhdpiAPK file size: 57 KiB (old), 57 KiB (new), 0 bytes (No change) Configuration xxhdpiAPK file size: 63 KiB (old), 63 KiB (new), 0 bytes (No change) Configuration xxxhdpiAPK file size: 64 KiB (old), 64 KiB (new), 0 bytes (No change) AlphaExpand to see flavor specificsUniversal APKAPK file size: 11 MiB (old), 11 MiB (new), 4044 bytes (Added) APK download size (estimated): 10 MiB (old), 10 MiB (new), 1074 bytes (Added) Method count: 118637 (old), 118648 (new), 11 (Added) Features: 1 (old), 1 (new), 0 (No change) Permissions: 6 (old), 6 (new), 0 (No change) Resources: 6054 (old), 6053 (new), 1 (Removed)
Lesson assets: 114 (old), 114 (new), 0 (No change) AAB differencesExpand to see AAB specificsSupported configurations:
Base APKAPK file size: 11 MiB (old), 11 MiB (new), 4036 bytes (Added)
Configuration hdpiAPK file size: 43 KiB (old), 43 KiB (new), 0 bytes (No change) Configuration ldpiAPK file size: 45 KiB (old), 45 KiB (new), 0 bytes (No change) Configuration mdpiAPK file size: 38 KiB (old), 38 KiB (new), 0 bytes (No change) Configuration tvdpiAPK file size: 73 KiB (old), 73 KiB (new), 0 bytes (No change) Configuration xhdpiAPK file size: 50 KiB (old), 50 KiB (new), 0 bytes (No change) Configuration xxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) Configuration xxxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) BetaExpand to see flavor specificsUniversal APKAPK file size: 11 MiB (old), 11 MiB (new), 4364 bytes (Added) APK download size (estimated): 10 MiB (old), 10 MiB (new), 3606 bytes (Added) Method count: 118641 (old), 118662 (new), 21 (Added) Features: 1 (old), 1 (new), 0 (No change) Permissions: 6 (old), 6 (new), 0 (No change) Resources: 6054 (old), 6053 (new), 1 (Removed)
Lesson assets: 114 (old), 114 (new), 0 (No change) AAB differencesExpand to see AAB specificsSupported configurations:
Base APKAPK file size: 11 MiB (old), 11 MiB (new), 4364 bytes (Added)
Configuration hdpiAPK file size: 43 KiB (old), 43 KiB (new), 0 bytes (No change) Configuration ldpiAPK file size: 45 KiB (old), 45 KiB (new), 0 bytes (No change) Configuration mdpiAPK file size: 38 KiB (old), 38 KiB (new), 0 bytes (No change) Configuration tvdpiAPK file size: 73 KiB (old), 73 KiB (new), 0 bytes (No change) Configuration xhdpiAPK file size: 50 KiB (old), 50 KiB (new), 0 bytes (No change) Configuration xxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) Configuration xxxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) GaExpand to see flavor specificsUniversal APKAPK file size: 11 MiB (old), 11 MiB (new), 4180 bytes (Added) APK download size (estimated): 10 MiB (old), 10 MiB (new), 4350 bytes (Added) Method count: 118641 (old), 118662 (new), 21 (Added) Features: 1 (old), 1 (new), 0 (No change) Permissions: 6 (old), 6 (new), 0 (No change) Resources: 6054 (old), 6053 (new), 1 (Removed)
Lesson assets: 114 (old), 114 (new), 0 (No change) AAB differencesExpand to see AAB specificsSupported configurations:
Base APKAPK file size: 11 MiB (old), 11 MiB (new), 4168 bytes (Added)
Configuration hdpiAPK file size: 43 KiB (old), 43 KiB (new), 0 bytes (No change) Configuration ldpiAPK file size: 45 KiB (old), 45 KiB (new), 0 bytes (No change) Configuration mdpiAPK file size: 38 KiB (old), 38 KiB (new), 0 bytes (No change) Configuration tvdpiAPK file size: 73 KiB (old), 73 KiB (new), 0 bytes (No change) Configuration xhdpiAPK file size: 50 KiB (old), 50 KiB (new), 0 bytes (No change) Configuration xxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) Configuration xxxhdpiAPK file size: 55 KiB (old), 55 KiB (new), 0 bytes (No change) |
|
Don't close. |
|
Hi @BenHenning, I'm going to mark this PR as stale because it hasn't had any updates for 7 days. If no further activity occurs within 7 days, it will be automatically closed so that others can take up the issue. |
|
Still don’t close. |
|
Hi @BenHenning, I'm going to mark this PR as stale because it hasn't had any updates for 7 days. If no further activity occurs within 7 days, it will be automatically closed so that others can take up the issue. |
…epare for release automation (#6265) ## Explanation Fixes #5033 Fixes #6215 Fixes #6217 Fixes #6218 Fixes #6219 Fixes #6220 Fixes #6221 Fixes #6222 Fixes #6223 This PR does a whole bunch of things that all come down to preparing for supporting fully automated Android app releases (as part of the 4.2 2026 GSoC project). The main preparation here is making it possible to run a single CLI command (to address #6222 and all of its constituent pieces) in order to build a fully deployable, production-ready version of the app for direct upload to Google Play Console. This PR makes that possible and explains what was done in individual sections below. ### New version code strategy #5033 required introducing changes to `TransformAndroidManifest` to dynamically compute the version code of the current build. The issue explain this in much more detail but essentially the strategy is to: - Consider each commit to develop a potential release (since a release branch could be cut from it). - Consider each commit to a release branch a release candidate. - Provide each release with 1000 version codes to divide among its flavors and release candidates. - As the app currently has 4 build flavors (dev/alpha/beta/ga) this means upcoming releases will be allowed 250 release candidates. - It's planned that we'll never exceed 25 build flavors (which is a significant overestimate) which leads to a maximum of 40 release candidates (which, if exceeded, would warrant just dropping that release altogether). - Have an overall release limit of approximately 2 million which would allow for over 40 years of sustained development at 100x the team's maximum-ever output (and Android actually supports a much larger version code space that we would expand into long before then, anyway). - Ensure that the first release after this PR is merged skips all previous version codes by beginning at 1000 rather than 0 (since the team hasn't yet reached 1000 version codes). - **Important**: Update the app version name to include its release candidate number, as well. This is the first change to the version name in a few years so it will result in some discrepancy with previous version name references. There were some other changes that were made to facilitate this: - `GitClient` was updated: - `countCommits`: a new method for computing the version code as mentioned above. - `retrieveBranchMergeBase`: updated such that `baseCommit` was switched to `resolvedBaseCommit` because otherwise its warning was being incorrectly printed in certain cases (such as when the `baseCommit` was just `HEAD`). - `TestGitRepository` was updated: - The `develop` branch is now created by default (which resulted in a bunch of script tests being updated) since otherwise some commands won't work correctly if there isn't a default branch or it doesn't match `develop` (which is expected now in `TransformAndroidManifestTest` due to the commit counting behaviors). The changes in other scripts led to broad simplifications which is a nice bonus. - `initializeHistoricalCommits`: new method for very efficiently populating a large number of commits using Git's `fast-import` command. It can pre-populate thousands of commits in just seconds (which is needed to avoid building a configuration property into `TransformAndroidManifest` due to it expecting all previous Oppia Android commits in the default repositroy configuration). The alternative was directly creating each commit which, at this volume, took many minutes. - `createRemoteBranchRef`: new method to allow `TransformAndroidManifest` to directly use `origin/develop` as expected (since only that was ever used for production--the configurable base branch was needed for testing purposes because a remote wasn't being set up before). This led to a simplification in `TransformAndroidManifest`'s contract which is a nice win. - `CommandExecutor` was updated to support piping standard input lines directly to a command (which is the only way `fast-import` works). - Version codes were removed from `version.bzl` since they are now derived and no longer need to be defined. It's maybe worth also explaining that while the initialization for `TransformAndroidManifestTest` may seem a bit hacky (since it's pre-initializing a very specific number of commits), it will actually be stable once this PR is merged since that number will never change for the lifetime of this version code strategy. The alternative of making the script utility configurable seemed less ideal (and actually the way AntiGravity approached this to begin with) for something that's a true constant. ### Automatic production manifest configuration & analytics support #6215 tracked a few manual tasks that were needed to prepare the release, namely: - Enabling Firebase analytics & crashlytics collection. - Enabling and configuration automatic app expiration. Fortunately, since `TransformAndroidManifest` already existed it was the natural place to perform these configurations by exposing a few new API properties to the script: - `enable_firebase_analytics` - `enable_app_expiration` These were implemented as expected: the first property directly configures analytics and Crashlytics collection. The second enables app expiration and automatically sets the date to 12 months from the build date of the app (i.e. when the script runs). Some things to note: - App expiration automatically enables for alpha and beta flavors of the app. Developer and GA flavors do not expire, whereas alpha and beta flavors of the app should always have an expiration (even after the app launches to GA). - Analytics is only enabled when `//config:enable_firebase_analytics` (a new configuration property) is set to `true`. If this is enabled then `TransformAndroidManifest` will also print out a warning to make it clear to the developer that analytics will be collected in that version of the app (since it's off by default in all builds). This could potentially be simplified in the future if #4390 is addressed. Finally, analytics collection wouldn't be complete without addressing #6217 which simply requires enabling the `PERFORMANCE_METRICS_COLLECTION` feature flag by default for all builds (which is fine since metrics won't actually be uploaded by default without it being configured per the previous notes). This is largely considered a safe change since performance metrics have been enabled and deployed for multiple years now without significant issue (though it's worth noting that not all performance metrics have been correctly collected per the issues outlined in #6062). ### Automated Firebase configuration #6223 required introducing support for integrating the production `google-services.json` configuration (which is needed to authenticate the app for productiion Firebase services support) without requiring the manual steps of logging into the Firebase Console and downloading this file by hand (and then replacing the developer version with care to ensure it's never uploaded). This required: - Introducing a Starlark rule, `prepare_google_services_json`, for actually downloading the file based on provided Firebase configuration properties (see below). - Introducing new build-time configuration properties: - `//config:download_firebase_config`: set to `true` to enable support for downloading the configuration file. - `//config:firebase_project_id`: set to the string project ID for the production Firebase project. - `//config:firebase_app_id`: set to the string app ID for the production Firebase project. Caveats: - Originally, the implementation (per the requirements listed in #6223) actually derived the app ID from just the project ID. However, this required a very complex script since actual JSON needed parsing. Since this is a short-term solution that will be made better as part of the GSoC 4.2 project (i.e. by using a Kotlin script, instead), the configuration was instead set up to just take the app ID (which can be easily saved by a release coordinator). - This does require configuring the new rule to be `local` executed since it needs the system-installed `firebase` command (a new prerequisite for release coordinators). - The developer `google-services.json` is still always used by default unless the production variant is specifically configured. This can result in some Bazel caching quirkiness: - Changing the configuration properties will invalidate the analysis cache but the rebuild _should_ just be the services file (which seems to just be an asset and/or resource bundle within an `android_library` and thus shouldn't propagate too far in the build graph). - Since network is required, theoretically the local production JSON file could differ from what's available through Firebase. Realistically, this file never changes so this risk is basically zero. ### Removing `CachingModule` #6219 essentially required removing `CachingModule` which, until now, required manual changes for production builds. All this module did was configure whether to load assets from protos and whether images should be loaded from local assets. This was easily changed to two new feature flags, instead, with per-flavor defaults: - Developer builds do not load proto assets or images from assets by default (to maintain existing parity). - Alpha, beta, and GA build flavors now do load from proto assets (breaking thumbnail loading for now) and expect images in local assets (breaking all image loading for those flavors). Generally, the team doesn't test non-dev flavors with assets except for cursory inspections, but it's possible to disable either or both feature flags, as needed, for local investigations (so it's not expected that this will be disruptive to active development). Caveats: - This change did require a broad set of changes across the codebase since it required the removal of a module and requirements for configuring the new feature flags (for a few select situations), but fortunately the change is largely trivial. - One miscellaneous but specific change that was needed: `GlideImageLoader` now needs to depend on `//utility` due to now using platform parameters (per these new feature flags that were added). - `LoadImagesFromAssets` and `LoadLessonProtosFromAssets` no longer require testfile exemptions (since their files were removed). ### Third-party license texts #6220 required introducing support for automatically pulling in third-party license texts to avoid requiring a manual "download license texts" step. At a high level this requires introducing a new Starlark rule for downloading third-party license texts with some additional changes: - The existing `RetrieveLicenseTexts` script needed a change to make it work for Starlark integration by taking a path to directly output the manifest file rather than a directory. - Making `RetrieveLicenseTexts` a `java_binary` to ensure it could actually be properly executed in a Starlark context (similar to other scripts binaries). - Moving the developer-only `third_party_dependencies.xml` file to a new app layer `thirdparty` package (with its own resources file) since it can't be included directly in the `//app:resources` build by default (otherwise it wouldn't be swappable with the production version). No special configurations or permissions are needed to download these texts, but they are not enabled by default. Instead, `//config:download_license_texts` needs to be set to `true` on the command line in order to enable this downloading functionality (which does require internet connectivity). Separately, one simplification that was made was the removal of `LicenseTextsCheck` since it's no longer needed: it shouldn't be straightforwardly possible to override the developer-only resource file with a production variant (as in, a developer would have to try hard to do this since previous release instructions would also not actually make it obvious due to the new location of the resource XML file). ### First-class support for app signing and renaming #6221 covered several highly manual steps of the release process: - Renaming the app according to its version (which mostly often requires installing the app on a device or emulator and pulling the version using `adb`). - Signing the app using the team's production upload key (which requires unhiding it from the team's private key store). Both of these processes involve very particular steps and are rather tedious. To solve them, a new `SignAndRenameAab` workflow step was added to `oppia_android_application` that: - Uses `bundletool` (already available in the app's build graph) to extract the necessary details for renaming the AAB. - Leverages new configuration properties for configuring signing behaviors: - `//config:keystore_file`: the path to the actual upload keystore file. - `//config:keystore_password_file`: the path to the file containing the keystore password (in plaintext). This was validated locally to ensure that the signing process was actually using the keystore password file and not reading the password (since this approach helps minimize the risk of the password accidentally being exposed in command output). - `//config:key_alias`: the string key alias to use corresponding to the keystore. - Leverages `jarsigner` for the actual signing. - Note that this will attempt to use the `jarsigner` that's likely part of the remote JDK configured for the build graph, however newer JDKs do not come prebundled with `jarsigner` so the rule falls back to the system version (requiring Java utilities to be in the release coordinator's `PATH`). - The Bazel team seems to largely expect CI scripts and release coordinators to perform signing separately from the build process. The plan here was to have a single command be able to do everything (hence the need for signing to be done this way), but we can certainly explore other options in the future (so long as it can be reasonably automated). - Is used by default for non-developer builds (the signing configurations fall back to the Android debug key by default). Some caveats: - Some effort was put into making it obvious what the name of the new AAB was so that the release coordinator can find it. However, Bazel doesn't re-run certain actions when they're cached and it's somewhat impossible to guarantee output without really abusing the build graph. For this reason `no-cache` was set for the new rule (to try and encourage re-printing the path). - Bazel does not support dynamically computing the name of an output file (unless it's a repository rule), so the renamed AAB has to be put into a folder, instead (hence the need for reading out the name of the file and path to make it clear to the developer what's happening). ### Automatic lesson downloads & updated asset management #6218 is clearly the most complex part of this process since it requires integrating the not-yet-merged lesson download script (#4885) into the build pipeline. This involved a lot of changes: - Directly depending on the lesson download script branch. - Updating the `oppia_proto_api` version to match what's needed by the download script. - Introducing a new `jsvg` third-party dependency (which is needed by the lesson download script). Note that this will eventually come as part of merging the asset download script branch into `develop`, anyway. - Introducing new Starlark rules & macros for: - Updating new pinned lesson version textproto configuration files (which are used to ensure deterministic downloads--this was introduced a while back specifically for the intent of eventually integrating the download script within the build process). - Downloading the production lessons and configuring them for packaging within the app binary. - Representing the production lessons as assets in an `android_library` for easy swap-in replacement for developer assets. - Defining local assets also as an `android_library` for a simpler `domain/BUILD.bazel` setup and for parity between local and remote assets. - Updating the lesson download scripts to support configurable exemptions (this was done before this PR directly on the lesson download script branch, and the work in this PR leverages those existing configurations to keep things in one location rather than duplicating configurations). - Introducing new build configuration parameters: - `//config:assets_type`: specifying the type of assets that should be re-pinned or downloaded, one of: 'alpha' or 'prod' (as the two supported exemption configurations)--more on this below. - `//config:proto_api_key_file`: the path to a file containing the plaintext API key to authorize downloading production assets. Eventually we hope to replace this with a safer mechanism per #6057. - Reworking how domain assets are managed: - Existing developer assets are now split into categories: developer and test. - The actual asset files themselves have been moved to more correct locations. - The developer flavor of the app no longer includes the test assets. - Test suites may depend on either developer or test assets. - Thw new asset targets are: `//domain:domain_dev_assets` and `//domain:domain_test_assets`. - Introducing two new production lesson targets: `//domain:domain_alpha_assets` and `//domain:domain_prod_assets`. Caveats: - Since assets are configured through build-time parameters there are no defaults on a flavor basis. However, per the `CachingModule` changes it's expected that non-dev flavors will generally always have production assets enabled for local testing since the app otherwise won't work correctly with developer-only assets. - Changes like the Bazel 7 migration will likely introduce complex build discrepancies between `develop` and the lesson download script--this will require some careful coordination. - Depending on `//domain:domain` will no longer provide implicit assets. - This is by design since the top-level build flavors must select which assets that they want to use and we never want to duplicate assets or include developer-only/test-only assets as part of production builds. - Overall this is an improvement since now test suites must be explicit in needing to depend on assets. - This did necessitate a bunch of test build target changes throughout the codebase, but not too many. - Neither of the production lesson targets will build without the correct build-time configuration set. - The way the download script is setup in Starlark is to maximize caching potential (since redownloading the assets is very expensive in both time waiting for them and in the cost to the Oppia web platform), partly by wrapping the results in library targets that can then be trivially cached by Bazel. - The version pin files will need to be manually updated on occasion by the release coordinator (though this will eventually be automated as part of the GSoC 4.2 project), though it's expected that these files are largely stable since: - They will only contain versions known to be format compatible with the app. - Their order should be based on lesson ordering within Oppia web lesson files (which isn't expected to often change). - The new `local_assets_library` was set up specifically to make the JSON assets seem 'additional' since they are expected to eventually be removed from app builds per #5663. - The pipeline does partly support including questions in the app build but more work is needed yet before this can be formally supported (see #5571). Asset types: - `prod`: these are all lessons confirmed to be stable for broad production release. Currently, it's only meant that the beta (and eventually GA) flavor of the app include these. - `alpha`: these include all production lessons plus all other lessons that might eventually release on the app (which will soon include the science lessons). ### Miscellaneous changes & considerations There are a number of things to note about this PR that aren't quite covered in above sections, or span multiple parts of the implementation. Specifically: - There are a bunch of platform parameter resets that were added to tests. These were largely done by AntiGravity and some may not be necessary. However, rather than investigating and cleaning these up, it seems fine to include them since eventually all tests will automatically do this, anyway, as part of the feature flag infrastructure work. - `testonly` was removed on some script/script infrastructure libraries since it can't or shouldn't be included within parts of the build pipeline (since prod binaries are, by definition, not test-only, though there is some wiggle room here from a Bazel perspective). - Proguard file references needed to be updated due to the Proguard configuration files now being exported files within a Bazel package (since `config/` now has a `BUILD.bazel` file). - `TransformAndroidManifest` was also simplified to not need to depend on `.git` directly. AntiGravity revealed that using `no-sandbox` (implied by `local` as described below) allows the command to run within the context of the actual repository and thus have access to repository source files. I wasn't aware this existed. Besides it simplifying `.git` it also could simplify other scripts in the future, if we wanted it to. - Note that one alternative here was to try and use Bazel's built-in status files but this ended up needing a much more complex Bash script to be run very early in the build graph initialization for all users (and that seemed like a much worse option). - Note also that the 'fix' for other scripts would require introducing `bazel run`-able targets (like being done in this PR for lesson pinning) since that could then pre-bind access to the local repository and even default other parameters. - All of the new `.bzl` files were put under `//tools` since we needed a generally good place for this. Bazel libraries might leverage a `defs` folder but that didn't seem appropriate for an application (Gemini suggested either `//tools` or `//bazel`--we already had the former and the latter seemed odd). We could consider moving the other `.bzl` files here in the future, or further reorganizing. Nevertheless, it seemed preferable to keep them together over the first attempt (which had them locally positioned) since it keeps custom build infrastructure largely together. - The different `execution_requirements` properties may require some additional explanation, despite being partly reference above: - `local`: implies `no-sandbox`, `no-remote-exec`, and `no-remote-cache` at the same time. Generally `no-sandbox` is the main one we care about, and it means to run the command without using the sandbox environment (thus exposing repository access to the rule). We use `local` since there are remote build (RBE) implications that we probably also want to disable since these are cases where the command is highly system-sensitive (though we don't currently use any RBE for the Oppia Android project). - `no-cache`: implies to not cache the results and always rebuild (though there are limitations to this). We should rarely use this, and only for very fast or volatile (e.g. date-sensitive) commands. - `requires-network`: implies networking is needed. I'm not sure if this is actually necessary since the commands already clearly have internet connectivity access, but there may be other, more complex nuances to sandboxing we haven't yet run into with Bazel (particularly with RBE). - One important note: the presence of the keys for `execution_properties` matters, not their values, so I've made an attempt to normalize on using "1" to imply "on" (even though "0" does **not** imply "off"). - Some of the progress messages in `oppia_android_application.bzl` were updated to be shorter to try and encourage them being printed out by Bazel (since Bazel can be really picky about window size and mysteriously prefers defaulting to a confusing truncated build target rather than the running action's progress message if the message is longer than it likes). In some cases, messages or mnemomincs were updates just to be generally more helpful and useful than before. - The new rules have been set up to introduce execution phase failures rather than analysis time failures for missing configurations. This is done deliberately since the latter will trigger failures for queries even when using `"manual"` tag (and thus caused some issues with the Android lint tool). - Otherwise, `"manual"` is used for cases when a library definitely cannot build due to missing configurations (such as one of the production asset libraries) which makes it exempt from `//...` or `//:all` patterns. - Not noted elsewhere: `select()` is used heavily throughout this PR to ensure that the build graph is correctly using the developer vs. production variants of different targets. Note that this is an analysis phase control and permanently configures the build graph (hence why certain CLI configuration changes invalidate the entire graph and require a re-analysis). Confusingly, Bazel 6.x doesn't enforce visibilty requirements for the conditions of `select()` (though this is purportedly changing in later versions of Bazel). ### AI disclosure & metholodogy A substantial portion of this PR was generated by Google's AntiGravity tool (using Gemini models), though a substantial amount of manual work was needed both in prompting and direct follow-up work. In fact, much of my methodology followed a rough pattern when introducing new functionality: 1. Craft and send a prompt asking for a specific thing to be added. 2. Review the changes, comparing against the current git index if there is one. If I think they're fine, go to step (4) otherwise `git add` them and go to step (3). 3. Outline what I don't like about the changes so that AntiGravity can try again, then go to (1). 4. Manually tweak the files as needed, then commit them and repeat step (1) with new feature work. I actually started this PR with (1) being back-to-back prompts pointing directly at the issues above (with some exceptions noted below) and having AntiGravity sort of just go off on its own to fix them. With my local `GEMINI.md` file and very detailed bugs this actually worked pretty well. It caught cases I missed in the bugs and usually came up with reasonable defaults, though I would definitely tweak the solutions either directly or with subsequent prompting if there were aspects I didn't like. Specifically I found that: - AntiGravity is very bad at writing conformant code in a bunch of ways: line spacing, variable naming, file locations, required documentation, and more. - It would generally always err toward the simplest, not best, solution and definitely requires iteration. It was rare for it to produce a solution that was nearly perfect (i.e. just static check/lint issues to address) on its first go. - It's still very bad at writing PR templates (I tried different techniques to make it track its work in a way that could be transferred over to a template, and also write the PR description itself--both were quite bad and thus I needed to write this myself). - It needs a _lot_ of handholding and coercing to use the correct tools when Another methodology aspect that might be interesting to note: - I combined all of the work items into one prompt process except for the work for integrating production lessons and the version change work. - Both of the lesson download script integration and version code updates were done as separate branches entirely (so 3 total branches). - I manually finalized the version code work (which required a lot more testing and cleanup than the other automation work). - I did not validate the download script integration directly. - I directed AntiGravity to combine the other two branches into this one via patch files and it actually did that surprisingly well, even with conflicts to address. If it helps, I tried documenting in individual commits when I used AntiGravity and (roughly) to what extent, though my notes in the commit messages aren't super detailed. ### Setup & example commands In order to use the new automation, some initial setup is needed. Specifically: - Release coordinators may need to have `jarsigner` available in their local `PATH`. - Release coordinators will need to install the Firebase CLI (https://firebase.google.com/docs/cli) and authenticate with it locally (i.e. `firebase login`). There is a bit of a gap with the script not obviously failing due to authentication issues. The pinned version files can be regenerated using one of two commands: For alpha lessons: ```sh bazel run //config:update_pinned_lesson_versions \ --//config:assets_type=alpha \ --//config:web_api_key_file=/path/to/prod_server.key ``` For production lessons: ```sh bazel run //config:update_pinned_lesson_versions \ --//config:assets_type=prod \ --//config:web_api_key_file=/path/to/prod_server.key ``` (where `prod_server.key` is the file with the plaintext backend API access key). For actually building a fully production version of the beta app: ```sh bazel build //:oppia_beta \ --//config:assets_type=prod \ --//config:web_api_key_file=/path/to/prod_server.key \ --//config:download_firebase_config=true \ --//config:firebase_project_id=firebase-project-id \ --//config:firebase_app_id=firebase-app-id \ --//config:download_license_texts=true \ --//config:enable_firebase_analytics=true \ --//config:keystore_file=/path/to/upload.keystore \ --//config:keystore_password_file=/path/to/upload_keystore.password \ --//config:key_alias=upload_key_alias ``` (where values are correctly filled in by the release coordinator). Note that these configurations can be combined in different ways for testing aspects of this PR's work without corresponding access, for example: - Full license texts can be downloaded. - A different signing key can be used (e.g. one generated locally for testing purposes). - Production lessons can be downloaded for a regular build without the other properties enabled. And, finally, optimization should still be enabled for non-developer builds that are actually being uploaded to the Play Console for deployment (which is not demonstrated in the command above). ## Essential Checklist - [x] The PR title starts with "Fix #bugnum: " (If this PR fixes part of an issue, prefix the title with "Fix part of #bugnum: ...".) - [x] The explanation section above starts with "Fixes #bugnum: " (If this PR fixes part of an issue, use instead: "Fixes part of #bugnum: ...".) - [x] Any changes to [scripts/assets](https://github.com/oppia/oppia-android/tree/develop/scripts/assets) files have their rationale included in the PR explanation. - [x] The PR follows the [style guide](https://github.com/oppia/oppia-android/wiki/Coding-style-guide). - [x] The PR does not contain any unnecessary code changes from Android Studio ([reference](https://github.com/oppia/oppia-android/wiki/Guidance-on-submitting-a-PR#undo-unnecessary-changes)). - [x] The PR is made from a branch that's **not** called "develop" and is up-to-date with "develop". - [x] The PR is **assigned** to the appropriate reviewers ([reference](https://github.com/oppia/oppia-android/wiki/Guidance-on-submitting-a-PR#clarification-regarding-assignees-and-reviewers-section)). ## For UI-specific PRs only Assessing UI aspects of this PR is tricky. Technically this PR changes nothing about the core end user UI/UX experience of the app, thus no demonstrations are added here. However, it has significant potential impact: - The third-party license infrastructure was reworked (though the existing developer experience has been validated). - The remote lesson download support means that it's possible to include production lessons in the build (which obviously differ the developer experience). Ditto for downloading third-party license texts. - Defaulting the proto and image loading for non-dev builds means a regression in behavior when building alpha/prod/GA without production lessons being downloaded.
|
Hi @BenHenning, I'm going to mark this PR as stale because it hasn't had any updates for 7 days. If no further activity occurs within 7 days, it will be automatically closed so that others can take up the issue. |
|
Hi @BenHenning, I'm going to mark this PR as stale because it hasn't had any updates for 7 days. If no further activity occurs within 7 days, it will be automatically closed so that others can take up the issue. |
|
Hi @BenHenning, I'm going to mark this PR as stale because it hasn't had any updates for 7 days. If no further activity occurs within 7 days, it will be automatically closed so that others can take up the issue. |
Explanation
Fixes #5989
Fixes #6053
Fixes #6059
Fixes #6060
Fixes #6097
Fixes #6200
This PR fixes the app's
WorkManagerconfiguration by essentially rebuilding how it works on a fundamental level.Existing behaviors
Previously, the app attempted to provide a custom
WorkManagerConfiguration(viaAbstractOppiaApplication) which, perWorkManagerConfigurationModule, added a bunch of customWorkerFactorys per a providedDelegatingWorkerFactory. However, there are two problems with this setup (#6053 covers both of these):Configurationthe app must remove the custom initialization provider (https://developer.android.com/develop/background-work/background-tasks/persistent/configuration/custom-configuration). Technically this is being done in the app today, however it's removing all app initialization which can affect more than justWorkManager.exportedtofalsefor theprovider. This is technically redundant sinceWorkManageralready declares it as not exported, but the Android Linter is only reviewing the localAndroidManifest.xmland not the fully merged manifest so it's a reasonable way to silence the warning (and be a bit more explicit about safety).WorkerFactorys are not adhering toDelegatingWorkerFactory's contract. The delegating factory expects subsequent factories to returnnullif they do not match the request worker being created, but Oppia's workers always return non-null. This means that the first worker always wins which is in this case theMetricLogSchedulingWorker(hence why we didn't notice this easily before because the majority of analytics events are successfully being uploaded, and most performance metrics are captured via direct app callbacks). This also wasn't caught in testing because of tests directly validating the jobs and not their scheduled and periodic behaviors. [BUG]: PlatformParemeterSyncUpWorkerFactory and MetricLogSchedulingWorkerFactory are not being initialised correctly #5989 is tracking this issue.Another issue with the current setup is that platform parameters cannot be safely initialized when the app starts up via a worker which has led to a top crash: #6060. That issue thread plus the documentation in
AbstractOppiaApplicationgo into a lot more detail about the initialization order changes.New design
BootstrapOppiaWorkerThe new design involves introducing a single
WorkManagerworker:BootstrapOppiaWorker. This introduces a very elegant improvement over the previous solution since it allows us to solve several classes of common issues every worker will face:WorkManager's own internal workers (which run through the same factory).WorkManager; these now clear themselves). This behavior works retroactively for past Oppia workers, as well, once this code is deployed to user's devices.Much of these weren't actually handled in the existing design. Using a common worker allows all of that to be solved in one place, and additional benefits could also be realized:
Configurationimplementation since the worker itself never changes.WorkManager(which actually, combined with some testing requirements, led to a complete isolation ofWorkManagerthroughout the codebase).OppiaWorkerinterface with Dagger-managed bindings. In the case of subtasks, this is mainly establishing an architectural pattern around an emergent behavior that was already the case for basically all of the current workers. Each subtask operates as a distinct top-level worker from the eyes ofWorkManager.Note that
BootstrapOppiaWorkerandWorkManagerConfigurationModuleare both being very thoroughly tested to ensure everything built on them can be comfortably relied upon and to cover the specific core cases that led to needing this PR in the first place (to help guard against regressions, however unlikely with these two classes rarely needing to change).New initialization logic
Beyond that, the recreation of initialization led to some other nice improvements:
ApplicationStartupListenerwas updated to have two different callbacks depending on whether platform parameters are initialized. This actually provides significant more stability in listeners that rely on platform parameters.WorkManagertrying to run a worker that relies on platform parameters andAbstractOppiaApplicationattempting to asynchronously load platform parameters. This two-step initialization process fully eliminates that race condition specifically for the path of a worker immediately running upon scheduling it (see next point). The other part of [BUG]: java.lang.IllegalStateException - Attempting to access feature flag LOGGING_LEARNER_STUDY_IDS before initialization #6060 was fixed by forcingBootstrapOppiaWorkerto synchronously load platform parameters before proceeding with creating the actual underlyingOppiaWorker.AnalyticsStartupListenerwas replaced withStartupWorkerScheduleReadinessListenersince that's effectively what it was: an application-bound listener that wanted to run on startup to schedule periodic jobs to run. Now that's done completely in an initialization order-safe way and with a custom scheduler (described later in this description).New scheduler
A new
WorkManagerSchedulerhas been introduced to simplify interacting withWorkManagerwhen wanting to schedule jobs by reusing common and obvious constraints across jobs and also ensuring that jobs cannot be scheduled in a way where they'd effectively never run (to address the potential of #6097 being an issue). Jobs are enforced to run no more than every 15 minutes (which is also a hardWorkManagerrule) and no less often than every 14 days.This scheduler works in tandem with the new
StartupWorkerScheduleReadinessListenerto provide an easy-to-use method for defining custom schedulers for a job.StartupWorkerScheduleReadinessMonitoris a new utility (bound itself as anApplicationStartupListener) which injects all of theStartupWorkerScheduleReadinessListeners and calls each at an initialization order safe time to schedule jobs (shortly after platform parameters are fully initialized). A few things to note:StartupWorkerScheduleReadinessListenerimplementations to use platform parameters directly (which is actually necessary since they may use platform parameters for configurations like job periodicity intervals as is the case forMetricLogSchedulingWorkerScheduler.ApplicationStartupListenerhelps to keepAbstractOppiaApplicationas lean as possible. One thing found during the development of this PR was that reasoning aboutAbstractOppiaApplication's initialization order and nuances was really challenging. Moving responsibilities out of theonCreate()should hopefully help make it much easier to understand and perform the necessary isolation for correctness.One limitation to note: the current version of
WorkManagermeans thatWorkManagerSchedulermust useExistingPeriodicWorkPolicy.KEEPwhen scheduling jobs whereasExistingPeriodicWorkPolicy.UPDATEwould actually be better since the latter properly supports property changes being applied without needing to cancel and restart the job. Practically this isn't expected to cause any problems for a while but it could potentially in the future (such as if we tweak the periodicity of performance metric captures or platform parameter syncing). #6115 is capturing the upgrade work needed to address this and there's a TODO pointing to that issue in code to ensure the update happens once it's possible.Finally, actual scheduling logic relies on
WorkManager. Even though we attempt to schedule jobs upon startup they may not actually run unless it's time for them to run again (i.e. their constraints are met and they haven't run within their configured interval). In most cases, requesting to schedule a job is a no-op since it's will already be tracked byWorkManager.OppiaWorkerOppiaWorkeris described a bit above but it's meant to be an extremely straightforward way to represent scheduled workers without having to worry about the extra complexity ofWorkManagerand edge integration-specific quirks. The idea is that all a developer needs to do is:OppiaWorkerimplementation with whatever subtasks are desired.StartupWorkerScheduleReadinessListenerto schedule the worker's subtasks upon application startup.And that's basically it. An
OppiaWorkerimplementation and itsStartupWorkerScheduleReadinessListenerimplementation can both directly depend on platform parameters and feature flags as needed and perform whatever background logic that needs to be done for the worker. The hope is this significantly simplifies implementing such workers in the future.Developer reference improvements
To help illustrate how to set up a new
OppiaWorkerand test it, a developer-onlyDebugWorkerhas been introduced along with its scheduler and module classes. This is a fully functioning worker that actually runs in//:oppia_devbuilds of the app and can be used as both a coding and maintenance reference (see wiki bits below).A bunch of updates were made to the
WorkManagerdocumentation covering not only the new implementation and testing patterns but also how to actually forcibly run workers usingadb. Unfortunately these instructions basically only work whenadbcan be rooted which is limited for personal devices (but should be fine for emulators). Hopefully these instructions significantly help when debugging workers in the future.Testing infrastructure changes
Testing workers is surprisingly complex which is likely why existing tests weren't actually validating the periodicity part of workers, and were introducing a lot of "backdoors" to directly access configuration values rather than reading what was passed to
WorkManager. A lot of investigation and iteration went into the creation of a newOppiaWorkManagerTestDriverutility which solves all of these (and more) problems:WorkManager.WorkManager's own internal testing library for managing constraints (thoughWorkManageritself doesn't do a great job here so the utility is limited).WorkManager's own behavior in a way that's interoperable withTestCoroutineDispatchers. I suggest looking at the code to understand what this does more.USE_TIME_BASED_SCHEDULINGcan be used which allows direct binding to theFakeOppiaClockand seems to actually "just work" with Oppia's test coroutine management (with some other tweaks).FakeLogScheduler,FakeLogUploader, andFirebaseLogUploader(andFirebaseLogUploaderModule) since workers should actually just be tested directly, instead.It's worth noting that the configuration test setup for the driver is rather complex and won't be fully described here since the documentation and code do a better job (though one thing to note is that using the driver will change the time management mode of the test away from the standard Oppia Android default). This and the other overall complexity of properly interacting with
WorkManager, its driver, orTestCoroutineDispatchersis ultimately why this PR also mandates using this driver overWorkManageror its test utility directly. Trying to make jobs work correctly in a test environment is a complete minefield, so it makes sense to just eliminate whole classes of developer frustration and confusion by requiring using a utility that trivializes the majority of these interactions.Some other notes:
WorkManager's internal database) that might break in future versions, but it's the best solution for what it's being used for currently.CoroutineExecutorServiceis needed as part of helping to bridge the gap and ensureTestCoroutineDispatcherinteroperability. This will continue to be the case even after [Feature Request]: Upgrade work manager to 2.9.0 #6115 is addressed, and will likely continue until Introduce managed executors to back dispatchers instead of using managed dispatchers #3715 is addressed.Changes to worker implementations & simplifications
The event log upload worker, performance metrics capture worker, and platform parameter worker all obviously require substantial reworking (along with their corresponding peer classes) to fit the new architectural patterns. Note that some attempt was also made to ensure naming consistency when it was easy (which is why
LogUploadWorkerremains inconsistent for now--updating it to be consistent would be a rather large codebase-wide change on top of an already very large PR for minimal gain). Fortunately the new architectural pattern makes renaming these classes trivial in the future.Note that this change does migrate over the platform parameter work which means it actually should run now. However, it doesn't actually perform syncing logic anymore and that will be addressed in #6061 (which actually prompted the investigation that led to this PR).
Miscellaneous non-test changes
There are a bunch of other changes that were made in passing:
ApplicationLifecycleListenerhas a new KDoc update that comes from the fact thatProcessLifecycleOwnerhas built-in delays for foreground/background changes to avoid invalid "went to background"/"wen to foreground" subsequent events just for navigating activities normally (sinceonStop/onStartis called between navigation moments). This was found to cause some initialization order complexities in tests so the documentation is meant to help make this clear for other developers in the future.ApplicationLifecycleObserverhad a bunch of changes:ApplicationLifecycleLoggerwhich is responsible for the actual metric logging so that the observer can focus on activity-level state tracking.Provider.AbstractOppiaApplication.onCreatewas further simplified by abstracting awayAppCheckProviderFactoryto the class's constructor. This allows it to default to the default production-friendly factory but also allowDeveloperOppiaApplicationto useDebugAppCheckProviderFactoryas needed. This led to an overall cleanup of no longer needing the build flavor as part of the root application component anymore.AndroidManifest.xmlinstructions was added to clarify how to actually enable faster event logs. This isn't directly related to the PR but it was noticed in passing. The previous support article was changed over the last few years and it's much harder to find the actualadbcommand needed.CpuPerformanceSnapshotterwas made injectable to simplify the Dagger graph a bit and which modules are actually needed for certain test scenarios. It's not obvious why it wasn't done this way originally from looking at code history.OppiaWorkManagerTestDriver).ExceptionsControllerwas noticed in passing that it wasn't a singleton, and this has been fixed. It needs to be to avoid potential race conditions against itsPersistentCacheStoreif/when more than one is created. This almost certainly has caused unseen problems.UncaughtExceptionLoggerStartupListenerwas updated in a few ways:Miscellaneous test & test infrastructural changes
There are other miscellaneous test and test infrastructure changes outside the main work manager bits:
WorkManagerinitialization was removed in tests that didn't actually use workers (e.g.ProfileAndDeviceIdActivityTest).ExplorationActivityLocalTestdid have an event reordering update but I don't remember precisely why this was correct.ApplicationLifecycleObserverTestwas updated to use an actual real activity to validate lifecycle integration (since before it wasn't actually testing that the utility was wired correctly). There's some messy reflection to ensureProcessLifecycleOwnerproperly resets across tests because it uses static state internally and the current version being used doesn't allow it to be properly reset. Addressing this is being tracked in [Feature Request]: Migrate to a cleaner method for resettingProcessLifecycleOwnerin tests #6187.CpuPerformanceSnapshotterTesthas been updated so that the snapshotter itself is being tested rather than testing happening viaApplicationLifecycleObserverfor more correct separations of concern.TestPlatformParameterModulehas become even more complex due to a realization during debugging thatTestCoroutineDispatchers.runCurrent()cannot be safely called off the main thread (see [BUG]: Fix dispatcher synchronization issues #6116), yet it still being possible via certain worker initialization pathways for platform parameters to now be needed off the main thread. This has been addressed by specifically delegating back to the main thread and correctly blocking on it.Loopercheck done is actually an SDK 21-compatible variant toisCurrentThreadwhich is, unfortunately, SDK 23+. This technically doesn't matter for Robolectric tests but Android Lint is unhappy without this approach.FakeAnalyticsEventLogger,FakeExceptionLogger,FakeFirestoreEventLogger, andFakePerformanceMetricsEventLoggerhave been updaed to have similar APIs (particularly for event count and exceptions) since this was needed for a bunch of tests. The loggers' tests themselves have also been updated to reflect these changes.Script, pattern, an exemption updates
Other non-app changes:
KdocValidityCheckwas updated to also ignoreMultibinds(which it should've done before, this was just missed since it's a rarely used annotation). A new test was added to validate this behavior change.WorkManagerlibrary interaction as mentioned in previous sections.StartupWorkerScheduleReadinessMonitorfor simplicity since it's a large PR and a relatively simple utility. [Feature Request]: Add tests forStartupWorkerScheduleReadinessMonitor#6189 is tracking adding tests for this in a future PR.ApplicationLifecycleLoggerbecauseApplicationLifecycleObserver's tests need to be audited to determine which should move into a dedicated logger test suite, and what additional logger-specific tests should be added. This is being tracked in [Feature Request]: Add tests forApplicationLifecycleLogger#6190.OppiaWorkManagerTestDriverbecause it's highly complex to test and not worth it at this point because it will actually be simplified with the upgrade to 2.9.0 per [Feature Request]: Upgrade work manager to 2.9.0 #6115. Nevertheless, adding tests for this driver is being tracked by [Feature Request]: Add tests forOppiaWorkManagerTestDriver#6191.MetricLogSchedulingWorkerScheduler). This revealed that theTestFileCheckutility doesn't correctly fail when an exemption is added but a test actually exists. [BUG]: Test file checker doesn't catch existing tests #6188 is tracking fixing this.DebugWorkerDebugModulewas exempted for coverage compatibility since it actually caused a failure during the finalization of the PR. I didn't dig into the specifics as to why it has an incompatibility, but this seems to be a reasonable workaround for now.Disclosure of LLM usage
I did use Gemini pretty heavily for diagnosing the early issues with the work manager integration and how to effectively debug them. This actually all began with #6061 when I was trying to figure out why I couldn't get the jobs to run, and pulling on that thread led (with LLM help) uncovered all of the broader problems that ended up being addressed here. Gemini did help a bit with ideating on an approach but the vast majority of the bootstrapping pattern and all of its corresponding code came from me. I used Gemini CLI a bit for some of the testing for
UncaughtExceptionLoggerStartupListenerbut that was it.Essential Checklist
For UI-specific PRs only
Largely this PR should have zero impact on direct UIs in the app, though it will indirectly affect the overall UX since it will enable platform parameters to actually work correctly (though not entirely, #6061 still needs to be finished). No screens are being updated as part of these changes.