Refactor#71
Merged
Merged
Conversation
TaskGroup necessarily gives us fail-fast task group implementation. However, in CI and even for local development, it can be useful to run to completion and then look more carefully at the failures once the successful tasks are all handled. Unfortunately, this means that we cannot use TaskGroup -- there's no alternative. So let's implement our own (TaskTracker) which has a similar API, but exiting the context will cancel all outstanding tasks, and it has an `as_completed()` method that tells you about the success. That way it's easy to switch off the fail-fast logic, if desired. Unfortunately, this reveals a problem with state management in kconfigs: If a task fails, the fetcher state may still get updated, resulting in the subsequent call skipping a distro which failed on the previous run and still needs an update. This can be papered over, but the entire fetcher / extractor abstraction is a bit messy right now, so an upcoming refactor will clean things up substantially. Signed-off-by: Stephen Brennan <stephen.s.brennan@oracle.com>
The future kconfigs architecture will include a package Index, which can be sync'd to get the IndexState, and then the IndexState can used to resolve the final artifact location, which is then downloaded and extracted. To start, introduce the IndexState and Artifact data structures, which are entirely immutable. To support this, there's a bit of logic to "freeze" and "thaw" loaded JSON so that we end up with really immutable structures. Signed-off-by: Stephen Brennan <stephen.s.brennan@oracle.com>
To replace the Fetcher API, create a new one called Index, which can be implemented to provide the logic for *syncing* the latest index state, and *resolving* an artifact URL from that state. To support the migration, add parallel logic in main.py to download for the new approach. Signed-off-by: Stephen Brennan <stephen.s.brennan@oracle.com>
This change also tweaks the index abstraction a bit to be simpler. Signed-off-by: Stephen Brennan <stephen.s.brennan@oracle.com>
Signed-off-by: Stephen Brennan <stephen.s.brennan@oracle.com>
Signed-off-by: Stephen Brennan <stephen.s.brennan@oracle.com>
Signed-off-by: Stephen Brennan <stephen.s.brennan@oracle.com>
Signed-off-by: Stephen Brennan <stephen.s.brennan@oracle.com>
Prefixing the filename by checksum is good for ensuring that the cached file is always the one we want. But two sha-1 checksums pushes the filename beyond 255 bytes. Most RPM database filenames already include the checksum in the filename. Detect this and avoid prepending. Signed-off-by: Stephen Brennan <stephen.s.brennan@oracle.com>
The design allows us to re-run after failures. Ensure that when we re-run, we verify previously-downloaded file hashes, and delete temporary results so that it's less likely we encounter a cache validity issue. Signed-off-by: Stephen Brennan <stephen.s.brennan@oracle.com>
Signed-off-by: Stephen Brennan <stephen.s.brennan@oracle.com>
The whole purpose of the "save" directory is to make it easier to cache data for local development when doing testing and running. However, generally indices won't use the data they save after all distros successfully resolve using the latest index state. So, add logic to track this. If all distros successfully run, clear the data. Signed-off-by: Stephen Brennan <stephen.s.brennan@oracle.com>
Signed-off-by: Stephen Brennan <stephen.s.brennan@oracle.com>
I've made several copy/paste mistakes in editing the config.ini file which indicates that it should be easier to edit. Add templates for distros so that specific targets (versions, architectures, etc) can override just the settings that matter. Further, use template strings with a set of variables that allow strings in the config to be templated by things like architecture and version. With these two things, the config becomes much more brief. It makes mistakes like the one the prior commit corrected much more difficult to create. Signed-off-by: Stephen Brennan <stephen.s.brennan@oracle.com>
oracle-contributor-agreement
Bot
added
the
OCA Verified
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.