Eliminate movprfx stalls in SVE BBQ D1Q4 bulk scorer (~14% throughput improvement)

[ChrisHegarty]

This was factored out of a trial run of a harness I've been working on to find performance optimisations for Elasticsearch. It was very early stages when it found this change, but the change is good and I wanted to extract as a standalone PR rather than hold up as I continue to iterate on the harness itself in collaboration with @tveasey and team.

This changes 16 SVE AND intrinsics in the dotd1q4_inner_bulk hot loop from svand_u8_z (zeroing form) to svand_u8_x (relaxed form), eliminating unnecessary movprfx instructions that clang-21 emits. On Graviton 4, this yields a 12–16% throughput improvement for bulk BBQ binary quantized vector scoring.

While profiling BBQ D1Q4 scoring on a Graviton 4 (SVE 128-bit) instance, we noticed the inner loop had an unexpectedly high instruction count. Disassembling the clang-21 output revealed a movprfx instruction before every and — 16 per loop iteration, 32 extra instructions total. The movprfx is required by the SVE architecture to set up the destination register for the zeroing predication form (_z), since SVE's destructive two-operand AND can't both zero inactive lanes and operate in-place without it.

Why the change is correct

The loop loads vectors using svld1_u8(pg, ...) where pg = svwhilelt_b8(r, length). The masked load already zeros inactive lanes in the loaded data. Since inactive lanes of both operands (v0..v3 and q0..q3) are already zero, the result of ANDing them is zero regardless of whether the AND itself zeros inactive lanes. We can therefore safely use svand_u8_x(svptrue_b8(), ...) - the all-true predicate means the AND computes all lanes unconditionally, and the compiler can emit a single unpredicated AND instruction with no movprfx.

Performance results (Graviton 4, clang-21, JMH)

┌───────────────────────┬─────────────────┬────────────────┬─────────────┐
│ Benchmark             │ Before (ops/ms) │ After (ops/ms) │ Improvement │
├───────────────────────┼─────────────────┼────────────────┼─────────────┤
│ D1Q4 bulk (dims=1024) │ ~75             │ ~86            │ +14.6%      │
├───────────────────────┼─────────────────┼────────────────┼─────────────┤
│ D1Q4 bulk (dims=768)  │ ~100            │ ~113           │ +13.0%      │
├───────────────────────┼─────────────────┼────────────────┼─────────────┤
│ D1Q4 bulk (dims=384)  │ ~195            │ ~220           │ +12.8%      │
└───────────────────────┴─────────────────┴────────────────┴─────────────┘

Scalar, NEON-tier, and D2Q4 paths are unaffected (the change only touches the SVE tier-2 code in vec_bbq_2.cpp).

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[elasticsearchmachine]

Pinging @elastic/es-perf (Team:Performance)

[elasticsearchmachine]

Hi @ChrisHegarty, I've created a changelog YAML for you.

[github-actions]

🔍 Preview links for changed docs

⏳ Building and deploying preview... View progress

This comment will be updated with preview links when the build is complete.

[github-actions]

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

• Check out the cumulative docs guidelines
• Reach out in the #docs Slack channel