[202405][GCU] Backport sonic-yang-mgmt + libyang perf-dep patches (co-req PR #417 on sonic-utilities.msft)#2733
Open
rimunagala wants to merge 3 commits into
Open
Conversation
When a uses clause imports a grouping, it was only processing leaf entries and ignoring leaf-list and choice clauses. That means that for instance in bgp route maps, route_map_in and route_map_out validations would fail. Honoring the uses refine clause is now also honored which is depended upon in sonic-utilities. This now precompiles the uses clause and integrates it as if the uses clause was not part of the schema as multiple end users were having to do this additional processing. This fixes that behavior and adds test cases to ensure it doesn't regress in the future. This is the proper fix, replacing #21078 that just worked around it. Removal of uses logic in sonic-utilities here: sonic-net/sonic-utilities#3814 Fixes #22382 Work item tracking Microsoft ADO (number only): How I did it Added leaf-list lookup. How to verify it See test cases pass Description for the changelog sonic-yang-mgmt: uses clause with leaf-list, choice not honored
…22254) Why I did it Generic Config Updater (GCU) is notoriously slow. These patches add some helpers for the GCU overhaul (mostly in sonic-utilities) in order to facilitate the optimizations. These changes are in sonic-yang-mgmt plus a patch to libyang 1. Changes include: Libyang v1 was not exposing must data for leaf nodes (like it does for other node types). Patch to correct this oversight. Loading of sonic configuration data should not mutate the user-provided data, this forces callers to know they need to deepcopy the data, plus in most instances data won't be mutated. sonic-yang-mgmt test models should more closely mimic real sonic yang models as we can't implement other features that assume this otherwise (sonic mandates that the top-level container has the same name as the module). Import sonic configdb<>yang xpath conversion from sonic-utilities as this should be shared code. 90% of this code is copied from the original source but does contain some bugfixes and enhancements including caching. Make find_schema_dependencies() public, plus add the ability to find dependencies recursively. This implementation is caching. Add new find_schema_must_count(), with recursive capabilities to find if a node (and its children) have must clauses. This implementation is caching. sonic-utilities PR: sonic-net/sonic-utilities#3831 Stats below ... (stats need both this and the sonic-utilities PR to be relevant)... Original Performance: Dry Run: time sudo config replace -d ./config_db.json ... real 2m51.588s user 2m23.777s sys 0m25.300s Full: time sudo config replace ./config_db.json ... real 14m53.772s user 12m2.376s sys 2m8.908s With Patch: Dry Run: time sudo config replace -d ./config_db.json ... real 0m59.602s user 0m56.434s sys 0m2.110s Full: time sudo config replace ./config_db.json ... real 1m54.303s user 0m58.482s sys 0m2.545s So that's roughly 3x improvement for dry-run, and 7.5x improvement for full commit. There is room for improvement on the full commit due to a sleep(1) being used between each patch because of a race condition found in the prior code (that was hidden due to a costly sanity check that has been removed). Work item tracking How I did it Gathered profiling data using cProfile and evaluated where the largest gains could be had. How to verify it This patch is standalone as it will not cause any issues in other projects which use sonic-yang-mgmt or libyang, however the performance benefits are in sonic-utilities. Apply both this commit and the sonic-utilities PR to a local branch, build and run sonic-utilities tests. Then create a full image, load it onto a DUT (with default configuration), and use the attached config_db.json to attempt a config replace operation (tested on Dell S5248F). Which release branch to backport (provide reason below if selected) 202411 Tested branch (Please provide the tested image version) master as of 20250521 Description for the changelog sonic-yang-mgmt: Generic Config Updater - performance dependencies Fixes #22372
loadData currently logs LOG_ERR "Data Loading Failed" to syslog on every exception (log-and-throw antipattern) before raising. This is undesirable for callers that use loadData speculatively to test many candidate configs for validity, e.g. generic_config_updater patch sorter (sonic-utilities) FullConfigMoveValidator — every forward leafref during the search emits a LOG_ERR to syslog even though the caller handles the exception and uses it as a prune signal. Add a quiet=False kwarg. When quiet=True, suppress the informational "Try to load Data" line and the LOG_ERR "Data Loading Failed" line, but still raise SonicYangException so the caller sees the failure. Default quiet=False preserves existing behavior for all current callers (including config apply-patch final validation, db_migrator upgrades, sonic-cfggen, and unit tests). Paired with sonic-utilities change that makes patch_sorter.FullConfigMoveValidator pass quiet=True. <!-- Please make sure you've read and understood our contributing guidelines: https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md failure_prs.log skip_prs.log Make sure all your commits include a signature generated with `git commit -s` ** If this is a bug fix, make sure your description includes "fixes #xxxx", or "closes #xxxx" or "resolves #xxxx" Please provide the following information: --> #### Why I did it On T0 devices, a single `config apply-patch` invocation leaks 100+ `Data Loading Failed` ERR lines to /var/log/syslog. These come from the GCU patch sorter's speculative `FullConfigMoveValidator`, which deliberately tests candidate moves that may transiently violate leafref constraints during the search — the sorter already handles the failure via the returned `(False, error)` tuple. The ERR lines trip `loganalyzer` in sonic-mgmt baseline tests (e.g. `test_dhcp_relay_with_non_default_vrf`, `test_dhcp_relay_with_different_non_default_vrf`) and obscure real errors in support bundles. The root cause is `loadData`'s log-and-throw antipattern: it emits a LOG_ERR *and* raises, forcing log noise on callers that are expected to recover. ##### Work item tracking - Microsoft ADO **(number only)**: #### How I did it Added a `quiet=False` kwarg to `SonicYangExtMixin.loadData`. When `True`, guard the informational `"Try to load Data"` sysLog and the `LOG_ERR "Data Loading Failed"` sysLog behind `if not quiet:`. `SonicYangException` is still raised on failure, so nothing about the control flow changes — only the syslog side-effect is suppressed. Default `quiet=False` preserves behavior for every current caller. #### How to verify it 1. Build `python3-sonic-yang-mgmt` wheel from this branch and install on a T0 DUT. 2. Pair with the sonic-utilities PR that threads `quiet=True` from `FullConfigMoveValidator` → `validate_config_db_config` → `loadData`. 3. Run `sonic-mgmt` generic_config_updater tests and/or `dhcp_server/test_dhcp_server_port_based_customize_options` on a T0 and count `"Data Loading Failed"` lines in `/var/log/syslog` across the window. <!-- If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012. --> #### Which release branch to backport (provide reason below if selected) <!-- - Note we only backport fixes to a release branch, *not* features! - Please also provide a reason for the backporting below. - e.g. - [x] 202006 --> - [ ] 202305 - [ ] 202311 - [ ] 202405 - [ ] 202411 - [ ] 202505 - [x] 202511 #### Tested branch (Please provide the tested image version) <!-- - Please provide tested image version - e.g. - [x] 20201231.100 --> - [ ] <!-- image version 1 --> - [ ] <!-- image version 2 --> #### Description for the changelog <!-- Write a short (one line) summary that describes the changes in this pull request for inclusion in the changelog: --> <!-- Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU. --> #### Link to config_db schema for YANG module changes <!-- Provide a link to config_db schema for the table for which YANG model is defined Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md --> Signed-off-by: Sonic Build Admin <sonicbld@microsoft.com> #### A picture of a cute animal (not mandatory but encouraged)
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport GCU perf dependencies: sonic-yang-mgmt helpers + libyang
must_size()patch to 202405This is Layers 2 + 3 of a 2-PR coordinated backport of upstream sonic-net/sonic-utilities#3831 (GCU perf improvements). The Layer 1 code lives at Azure/sonic-utilities.msft#417.
What's in this PR
Layer 2 —
src/sonic-yang-mgmt/(in-tree, not a submodule): 3 cherry-picks backporting upstream #22254:sonic_yang_path.py— YANG path-manipulation helpersfind_schema_dependencies()— schema dep index with cachingfind_schema_must_count()— must-constraint counting (needs Layer 3)Layer 3 —
src/libyang/patch/libyang-leaf-must.patch: exposesSchema_Node_Leaf::must_size()on the libyang 1.0.73 swig CPP binding. Without this, Layer 2'sfind_schema_must_count()raisesAttributeErrorat runtime.Why this MUST merge before #417
Without Layer 2, PR #417 python code hits
AttributeError: 'SonicYang' object has no attribute 'find_schema_dependencies'on import. Without Layer 3, Layer 2'sfind_schema_must_count()hitsAttributeError: 'Schema_Node_Leaf' object has no attribute 'must_size'.Validation (all 3 layers combined)
Full test matrix (perf, correctness, multi-ASIC, real writes, error handling, code audit) is in Azure/sonic-utilities.msft#417. Headline numbers:
Real (non-dry-run) apply of 50 ACL rules: rc=0 in 7.9s, all 50 present in CONFIG_DB. Rollback of 49 rules: rc=0 in 6.8s. Multi-ASIC patch spanning
/asic0/+/asic1/: rc=0, both namespaces validated. Four malformed-patch scenarios all produce graceful rc=2 errors with no python tracebacks.Tested on str3-7800-lc3-1 (Arista 7800R3A-36DM2-D36 chassis LC, stock SONiC.20240532.59, Python 3.11.2, libyang 1.0.73 patched).
Files
19 files changed:
src/sonic-yang-mgmt/**(Layer 2 python + tests) andsrc/libyang/patch/libyang-leaf-must.patch(Layer 3).Merge order
src/sonic-utilitiesat [code sync] Merge code from sonic-net/sonic-buildimage:202205 to 202205 #417's merged SHA