Skip to content

[tokenize] Avoid istringstream in delimiter-based tokenize#1203

Open
Mohxen wants to merge 1 commit into
sonic-net:masterfrom
Mohxen:optimize-tokenize-no-istringstream
Open

[tokenize] Avoid istringstream in delimiter-based tokenize#1203
Mohxen wants to merge 1 commit into
sonic-net:masterfrom
Mohxen:optimize-tokenize-no-istringstream

Conversation

@Mohxen

@Mohxen Mohxen commented Jun 8, 2026

Copy link
Copy Markdown

Problem

tokenize(str, token) currently constructs a std::istringstream for every call.
This is more expensive than needed for splitting a string on a single delimiter,
because it brings in iostream and locale machinery.

This helper is used in SWSS hot paths, including route processing in orchagent.
During route burst profiling, perf showed time in:

  • swss::tokenize
  • std::getline
  • std::basic_ios::_M_cache_locale
  • std::locale
  • std::vector<std::string> growth
  • operator new

Fix

Replace the istringstream/getline implementation with direct
std::string::find() / substr() delimiter scanning.

This matches the implementation style already used by the limited-token tokenize
overload added in #96, while preserving the observed behavior of the base overload.

Testing

Configured locally with YANG modules disabled:

./autogen.sh
./configure --enable-debug --disable-yangmodules 'CXXFLAGS=-O0 -g'

@linux-foundation-easycla

Copy link
Copy Markdown

CLA Not Signed

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants