zebra: show ip route displays stale nexthop after ECMP path withdrawal during interface bounce

[rejithomas-arista]

Describe the bug

After an interface bounce on an ECMP route, show ip route displays a nexthop as active (*) even though BGP has already withdrawn the corresponding path. The data plane (kernel FIB) is correct — only the zebra RIB display is wrong. The stale nexthop persists until the BGP session re-establishes and fresh routes replace the NHE.

[x] Did you check if this is a duplicate issue?
[ ] Did you test it on the latest FRRouting/frr master branch?

Not tested on latest master, but the affected code in zebra_nhg_nexthop_compare() and zebra_interface_nhg_reinstall() is unchanged in current upstream. A direct fix was attempted in FRRouting/frr#22709 (rejected), and the comprehensive fix FRRouting/frr#22383 is stalled.

To Reproduce

1. Establish an ECMP route with N nexthops (e.g., BGP multipath)
2. Shut down an interface that carries one of the ECMP nexthops
3. Wait for BGP to withdraw the path
4. Bring the interface back up while BGP is still reconnecting (peer in Active state)
5. Compare routing state across layers:

show bgp ipv4 unicast <prefix>   # Correct: N-1 paths
show ip route <prefix>           # BUG: N nexthops, withdrawn one shown as active *
ip route show <prefix>           # Correct: N-1 nexthops

Expected behavior

show ip route should display N-1 nexthops, consistent with the BGP table and kernel FIB. The withdrawn nexthop should not appear as active.

Versions

• OS Kernel: Linux 5.10 (SONiC)
• FRR Version: 10.0.1 (commit 88f5c06). Also confirmed present in current upstream master (July 2026).

Additional context

Root Cause

Two functions in zebra/zebra_nhg.c interact to produce this bug:

1. zebra_nhg_nexthop_compare() — When BGP sends an updated route with N-1 nexthops, zebra compares the new NHE against the old (N nexthops, one inactive). The comparison skips inactive old nexthops:

if (!CHECK_FLAG(old_nhop->flags, NEXTHOP_FLAG_ACTIVE)) {
    ...
    old_nhop = old_nhop->next;
    continue;
}

The N-1 new nexthops align with the N-1 active old nexthops. The inactive old nexthop is ignored. zebra_nhg_rib_compare_old_nhe() returns the old N-nexthop NHE instead of the new (N-1)-nexthop NHE.

This skip logic is an optimization for the case where the same route is re-announced with one nexthop temporarily unreachable. It is incorrect when a nexthop has been explicitly removed by the protocol.

2. zebra_interface_nhg_reinstall() — When the interface comes back up, this function unconditionally re-activates all singleton NHEs on zif->nhg_dependents, including the stale one that was never cleaned up because the old group NHE still holds a reference. The re-activation propagates to the group NHE, causing show ip route to display the withdrawn nexthop as active.

If the comparison had correctly returned the new (N-1) NHE, the old group NHE would have been released, the stale singleton's refcnt would drop to 0, and it would be removed from zif->nhg_dependents — preventing re-activation.

Suggested Fix

zebra_nhg_nexthop_compare() should not treat two NHEs as equivalent when the new NHE has fewer nexthops than the old NHE's active count. The inactive-skip optimization is valid when the nexthop sets are the same (one temporarily down), but not when the protocol has explicitly removed a nexthop.

• Data plane is not affected — kernel FIB is correct. This is a control plane display issue only.
• The stale window duration depends on the BGP ConnectRetry timer (up to 120s with default timers).
• zebra: do not reuse NHG from a removed route entry FRRouting/frr#22709 attempted a direct fix (guard against NHG reuse from REMOVED route entry). Rejected by maintainer as it would break NHG reuse during link flaps, pointed to #22383 as the preferred approach.
• Zebra: NHG event tracker — maximize NHG ID reuse during convergence FRRouting/frr#22383 (comprehensive NHG event tracker) would fix this as part of a broader convergence overhaul but is stalled (needs rebase, ~8400 lines, no reviewers).