Skip to content

vpp: fix race condition causing syncd crash#1874

Merged
yue-fred-gao merged 1 commit into
sonic-net:masterfrom
yue-fred-gao:event_queue_race
Apr 29, 2026
Merged

vpp: fix race condition causing syncd crash#1874
yue-fred-gao merged 1 commit into
sonic-net:masterfrom
yue-fred-gao:event_queue_race

Conversation

@yue-fred-gao

@yue-fred-gao yue-fred-gao commented Apr 28, 2026

Copy link
Copy Markdown
Contributor

Description of PR

Summary:
Fixes a race condition during init_switch causing syncd crash

Type of change

  • Bug fix
  • New feature
  • Refactor / cleanup
  • Documentation update
  • Test improvement

Approach

What is the motivation for this PR?

We saw below core dump during init_switch.

#0  0x0000ffffb5c369d0 in vpp_ev_enqueue (ev=0xaaaabe182c50) at vpp/vppxlate/SaiVppXlate.c:397
#1  vl_api_sw_interface_event_t_handler (mp=<optimized out>) at vpp/vppxlate/SaiVppXlate.c:687
#2  0x0000ffffb5264470 in vl_msg_api_handler_no_free ()
   from /nobackup/yuega2/sonic-buildimage-cisco.202603.arctos-periodic-40135/syncd_dbg/rootfs/lib/aarch64-linux-gnu/libvlibapi.so.25.02.0
#3  0x0000ffffb5205648 in ?? ()
   from /nobackup/yuega2/sonic-buildimage-cisco.202603.arctos-periodic-40135/syncd_dbg/rootfs/lib/aarch64-linux-gnu/libvlibmemoryclient.so.25.02.0
#4  0x0000ffffb5c3cd54 in vpp_intf_events_enable_disable (enable=true) at vpp/vppxlate/SaiVppXlate.c:1839
#5  init_vpp_client () at vpp/vppxlate/SaiVppXlate.c:1704
#6  0x0000ffffb5c208bc in saivs::SwitchVpp::vpp_dp_initialize (this=this@entry=0xaaaabe17a3d0) at vpp/SwitchVppRif.cpp:468
#7  0x0000ffffb5c0508c in saivs::SwitchVpp::SwitchVpp (this=this@entry=0xaaaabe17a3d0, switch_id=switch_id@entry=141733920768, manager=..., config=..., 
    warmBootState=...) at vpp/SwitchVpp.cpp:44

This happened during init_switch right after vpp_intf_events_enable_disable, an event came in but event_queue had not been properly initialized.

Work item tracking
  • Microsoft ADO (number only):

How did you do it?

Move event queue initialization before enabling events from vpp.

How did you verify/test it?

Run sonic-mgmt via sonic-net/sonic-buildimage#27039

Any platform specific information?

vpp specific

Documentation

Signed-off-by: Yue Gao <yuega2@cisco.com>
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@mssonicbld

Copy link
Copy Markdown
Collaborator

Cherry-pick PR to msft-202603: Azure/sonic-sairedis.msft#127

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants