Bug: GCU apply-patch error logs although patch applied - PORTCHANNEL,INTERFACE

[okaravasi]

Is it platform specific

generic

Importance or Severity

High

Description of the bug

Hello,

When trying to remove or add PORTCHANNEL information via gcu unexpected syslog error logs are printed although apply-patch operation succeeds and config is updated.
Error logs are not seen in console output when giving the command, except if called with the verbosity output.
The operation is successful and change is applied but the existence of error logs is unexpected.

Example PATCH:

[
	{
        "op": "remove",
        "path": "/asic0/PORTCHANNEL_INTERFACE/PortChannel102"
    },
	{
        "op": "remove",
        "path": "/asic0/PORTCHANNEL_INTERFACE/PortChannel102|10.0.0.0~131"
    },
    {
        "op": "remove",
        "path": "/asic0/PORTCHANNEL_INTERFACE/PortChannel102|FC00::1~1126"
    },
	{
        "op": "remove",
        "path": "/asic0/PORTCHANNEL_MEMBER/PortChannel102|Ethernet0"
    },
	
	{
        "op": "remove",
        "path": "/asic0/PORTCHANNEL_MEMBER/PortChannel102|Ethernet8"
    },
    {
        "op": "remove",
        "path": "/localhost/PORTCHANNEL_INTERFACE/PortChannel102"
    },
    {
        "op": "remove",
        "path": "/localhost/PORTCHANNEL_INTERFACE/PortChannel102|10.0.0.0~131"
    },
    {
        "op": "remove",
        "path": "/localhost/PORTCHANNEL_INTERFACE/PortChannel102|FC00::1~1126"
    },
         {
        "op": "remove",
        "path": "/localhost/PORTCHANNEL_MEMBER/PortChannel102|Ethernet2~11"
    },
    {
        "op": "remove",
        "path": "/localhost/PORTCHANNEL_MEMBER/PortChannel102|Ethernet1~11"
    },
	{
        "op": "remove",
        "path": "/asic0/PORTCHANNEL"
    },
    {
        "op": "remove",
        "path": "/localhost/PORTCHANNEL/PortChannel102"
    }
]

Console output of apply-patch:

admin@ixre-egl-board211:~$ sudo config apply-patch /tmp/remove_pc.json
Patch Applier: asic0: Patch application starting.
Patch Applier: asic0: Patch: [{"op": "remove", "path": "/PORTCHANNEL_INTERFACE/PortChannel102"}, {"op": "remove", "path": "/PORTCHANNEL_INTERFACE/PortChannel102|10.0.0.0~131"}, {"op": "remove", "path": "/PORTCHANNEL_INTERFACE/PortChannel102|FC00::1~1126"}, {"op": "remove", "path": "/PORTCHANNEL_MEMBER/PortChannel102|Ethernet0"}, {"op": "remove", "path": "/PORTCHANNEL_MEMBER/PortChannel102|Ethernet8"}, {"op": "remove", "path": "/PORTCHANNEL/PortChannel102"}]
Patch Applier: asic0 getting current config db.
Patch Applier: asic0: simulating the target full config after applying the patch.
Patch Applier: asic0: validating all JsonPatch operations are permitted on the specified fields
Patch Applier: asic0: validating target config does not have empty tables,
                            since they do not show up in ConfigDb.
Patch Applier: asic0: sorting patch updates.
Patch Applier: The asic0 patch was converted into 6 changes:
Patch Applier: asic0: applying 6 changes in order:
Patch Applier:   * [{"op": "remove", "path": "/PORTCHANNEL_INTERFACE/PortChannel102"}]
Patch Applier:   * [{"op": "remove", "path": "/PORTCHANNEL_INTERFACE/PortChannel102|10.0.0.0~131"}]
Patch Applier:   * [{"op": "remove", "path": "/PORTCHANNEL_INTERFACE/PortChannel102|FC00::1~1126"}]
Patch Applier:   * [{"op": "remove", "path": "/PORTCHANNEL_MEMBER/PortChannel102|Ethernet0"}]
Patch Applier:   * [{"op": "remove", "path": "/PORTCHANNEL_MEMBER/PortChannel102|Ethernet8"}]
Patch Applier:   * [{"op": "remove", "path": "/PORTCHANNEL/PortChannel102"}]
Patch Applier: asic0: verifying patch updates are reflected on ConfigDB.
Patch Applier: asic0 patch application completed.
Patch Applier: localhost: Patch application starting.
Patch Applier: localhost: Patch: [{"op": "remove", "path": "/PORTCHANNEL_INTERFACE/PortChannel102"}, {"op": "remove", "path": "/PORTCHANNEL_INTERFACE/PortChannel102|10.0.0.0~131"}, {"op": "remove", "path": "/PORTCHANNEL_INTERFACE/PortChannel102|FC00::1~1126"}, {"op": "remove", "path": "/PORTCHANNEL_MEMBER/PortChannel102|Ethernet2~11"}, {"op": "remove", "path": "/PORTCHANNEL_MEMBER/PortChannel102|Ethernet1~11"}, {"op": "remove", "path": "/PORTCHANNEL/PortChannel102"}]
Patch Applier: localhost getting current config db.
Patch Applier: localhost: simulating the target full config after applying the patch.
Patch Applier: localhost: validating all JsonPatch operations are permitted on the specified fields
Patch Applier: localhost: validating target config does not have empty tables,
                            since they do not show up in ConfigDb.
Patch Applier: localhost: sorting patch updates.
Patch Applier: The localhost patch was converted into 6 changes:
Patch Applier: localhost: applying 6 changes in order:
Patch Applier:   * [{"op": "remove", "path": "/PORTCHANNEL_INTERFACE/PortChannel102"}]
Patch Applier:   * [{"op": "remove", "path": "/PORTCHANNEL_INTERFACE/PortChannel102|10.0.0.0~131"}]
Patch Applier:   * [{"op": "remove", "path": "/PORTCHANNEL_INTERFACE/PortChannel102|FC00::1~1126"}]
Patch Applier:   * [{"op": "remove", "path": "/PORTCHANNEL_MEMBER/PortChannel102|Ethernet1~11"}]
Patch Applier:   * [{"op": "remove", "path": "/PORTCHANNEL_MEMBER/PortChannel102|Ethernet2~11"}]
Patch Applier:   * [{"op": "remove", "path": "/PORTCHANNEL/PortChannel102"}]
Patch Applier: localhost: verifying patch updates are reflected on ConfigDB.
Patch Applier: localhost patch application completed.
Patch applied successfully.

Error logs in syslog:

admin@ixre-egl-board211:~$ sudo grep -ri "ERR " /var/log/syslog
2025 Oct 24 15:44:34.106593 ixre-egl-board211 ERR sonic_yang: Data Loading Failed:Leafref "/sonic-portchannel:sonic-portchannel/sonic-portchannel:PORTCHANNEL/sonic-portchannel:PORTCHANNEL_LIST/sonic-portchannel:name" of value "PortChannel102" points to a non-existing leaf.
2025 Oct 24 15:44:34.460636 ixre-egl-board211 ERR sonic_yang: Data Loading Failed:Leafref "/sonic-portchannel:sonic-portchannel/sonic-portchannel:PORTCHANNEL/sonic-portchannel:PORTCHANNEL_LIST/sonic-portchannel:name" of value "PortChannel102" points to a non-existing leaf.
2025 Oct 24 15:44:34.815571 ixre-egl-board211 ERR sonic_yang: Data Loading Failed:Leafref "/sonic-portchannel:sonic-portchannel/sonic-portchannel:PORTCHANNEL/sonic-portchannel:PORTCHANNEL_LIST/sonic-portchannel:name" of value "PortChannel102" points to a non-existing leaf.
2025 Oct 24 15:44:35.190970 ixre-egl-board211 ERR sonic_yang: Data Loading Failed:Leafref "/sonic-portchannel:sonic-portchannel/sonic-portchannel:PORTCHANNEL/sonic-portchannel:PORTCHANNEL_LIST/sonic-portchannel:name" of value "PortChannel102" points to a non-existing leaf.
2025 Oct 24 15:44:35.544785 ixre-egl-board211 ERR sonic_yang: Data Loading Failed:Leafref "/sonic-portchannel:sonic-portchannel/sonic-portchannel:PORTCHANNEL/sonic-portchannel:PORTCHANNEL_LIST/sonic-portchannel:name" of value "PortChannel102" points to a non-existing leaf.
2025 Oct 24 15:44:40.327811 ixre-egl-board211 ERR teamd0#tlm_teamd: :- get_dump: Can't get dump for LAG 'PortChannel102'. Skipping
2025 Oct 24 15:44:40.331778 ixre-egl-board211 ERR swss0#orchagent: :- removeLag: Failed to remove ref count 1 LAG PortChannel102
2025 Oct 24 15:44:41.349975 ixre-egl-board211 ERR swss0#intfmgrd: :- setIntfVrf: Command '/sbin/ip link set "PortChannel102" nomaster' failed with rc 1
2025 Oct 24 15:44:41.355276 ixre-egl-board211 ERR swss0#orchagent: message repeated 73 times: [ :- removeLag: Failed to remove ref count 1 LAG PortChannel102]
2025 Oct 24 15:44:42.434215 ixre-egl-board211 ERR sonic_yang: Data Loading Failed:Leafref "/sonic-portchannel:sonic-portchannel/sonic-portchannel:PORTCHANNEL/sonic-portchannel:PORTCHANNEL_LIST/sonic-portchannel:name" of value "PortChannel102" points to a non-existing leaf.
2025 Oct 24 15:44:42.608768 ixre-egl-board211 ERR sonic_yang: Data Loading Failed:Leafref "/sonic-portchannel:sonic-portchannel/sonic-portchannel:PORTCHANNEL/sonic-portchannel:PORTCHANNEL_LIST/sonic-portchannel:name" of value "PortChannel102" points to a non-existing leaf.
2025 Oct 24 15:44:42.781818 ixre-egl-board211 ERR sonic_yang: Data Loading Failed:Leafref "/sonic-portchannel:sonic-portchannel/sonic-portchannel:PORTCHANNEL/sonic-portchannel:PORTCHANNEL_LIST/sonic-portchannel:name" of value "PortChannel102" points to a non-existing leaf.
2025 Oct 24 15:44:42.957330 ixre-egl-board211 ERR sonic_yang: Data Loading Failed:Leafref "/sonic-portchannel:sonic-portchannel/sonic-portchannel:PORTCHANNEL/sonic-portchannel:PORTCHANNEL_LIST/sonic-portchannel:name" of value "PortChannel102" points to a non-existing leaf.
2025 Oct 24 15:44:43.130428 ixre-egl-board211 ERR sonic_yang: Data Loading Failed:Leafref "/sonic-portchannel:sonic-portchannel/sonic-portchannel:PORTCHANNEL/sonic-portchannel:PORTCHANNEL_LIST/sonic-portchannel:name" of value "PortChannel102" points to a non-existing leaf.

Steps to Reproduce

Please check above.

Actual Behavior and Expected Behavior

Syslog log should not be printed since the operation is successful.
OR
If there is indeed an error the patch should fail with appropriate reasoning.

Relevant log output

Output of show version, show techsupport

admin@ixre-egl-board211:~$ show ver

SONiC Software Version: SONiC.HEAD.1168725-nokia-master-005987d4e
SONiC OS Version: 12
Distribution: Debian 12.12
Kernel: 6.1.0-29-2-amd64
Build commit: 005987d4e
Build date: Mon Sep 22 22:13:49 UTC 2025
Built by: gitlab-runner@sonic-bld2

Attach files (if any)

No response

[prabhataravind]

@okaravasi we think Brad's PR sonic-net/sonic-utilities#3831 should help with this issue. Could you please give it a try?

[okaravasi]

@okaravasi we think Brad's PR sonic-net/sonic-utilities#3831 should help with this issue. Could you please give it a try?

thank you I will do and update soon

[okaravasi]

For reference similar issue happend for table INTERFACE.
Failure happens when patch includes the removal of the INTERFACE ip-prefix as well as the interface name.

Error logs printed during sorting of patch:

ERR sonic_yang: Data Loading Failed:Must condition not satisfied, Try adding PORT: {}, Example: 'Ethernet0': {}
ERR sonic_yang: Data Loading Failed:Must condition not satisfied, Try adding PORT: {}, Example: 'Ethernet0': {}
ERR sonic_yang: Data Loading Failed:Must condition not satisfied, Try adding PORT: {}, Example: 'Ethernet0': {}
ERR sonic_yang: Data Loading Failed:Must condition not satisfied, Try adding PORT: {}, Example: 'Ethernet0': {}

Example patch that produces the errors:

[
    {
        "op": "remove",
        "path": "/asic1/INTERFACE/Ethernet240|10.0.0.22~131"
    },
    {
        "op": "remove",
        "path": "/asic1/INTERFACE/Ethernet240|FC00::2D~1126"
    },
    {
        "op": "remove",
        "path": "/asic1/INTERFACE/Ethernet240"
    }
]

The W/A/ is to perform the operation in 2 steps:
-1st patch operation removes the INTERFACE ip-prefixes entries

• 2nd patch operation removes the INTERFACE name entry.

Root cause seems similar with the PORTCHANNEL issue that this ticket was originally created for, so I will test the suggested PR with the fix for this case too and I will update.

Thanks,
Olympia