[PW_SID:1118643] futex: Use runtime constants for futex_hash computation#2166
[PW_SID:1118643] futex: Use runtime constants for futex_hash computation#2166linux-riscv-bot wants to merge 9 commits into
Conversation
Futex hash computation requires a mask operation with read-only after init data that will be converted to a runtime constant in the subsequent commit. Introduce runtime_const_mask_32 to further optimize the mask operation in the futex hash computation hot path. [ prateek: Broke off the x86 chunk, commit message. ] Link: https://patch.msgid.link/20260227161841.GH606826@noisy.programming.kicks-ass.net Not-yet-signed-off-by: Peter Zijlstra <peterz@infradead.org> Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
The current scheme to directly patch the kernel text for runtime
constants runs into the following issue with futex adapted to using
runtime constants on arm64:
Unable to handle kernel write to read-only memory at virtual address ...
The pc points to the *p assignment in the following call chain:
futex_init()
runtime_const_init(shift, __futex_shift)
__runtime_fixup_shift()
*p = cpu_to_le32(insn);
which suggests that core_initcall() is too late to patch the kernel text
directly unlike the "d_hash_shift" which is initialized during
vfs_caches_init_early() before the protections are in place.
Use aarch64_insn_patch_text_nosync() to patch the runtime constants
instead of doing it directly to allow runtime_const_init() slightly
later into the boot.
Since aarch64_insn_patch_text_nosync() calls caches_clean_inval_pou()
internally, __runtime_fixup_caches() ends up being redundant.
runtime_const_init() are rare and the overheads of multiple calls to
caches_clean_inval_pou() instead of batching them together should be
negligible in practice.
The cpu_to_le32() conversion of instruction isn't necessary since it is
handled later in the aarch64_insn_patch_text_nosync() call-chain:
aarch64_insn_patch_text_nosync(addr, insn)
aarch64_insn_write(addr, insn)
__aarch64_insn_write(addr, cpu_to_le32(insn))
Sashiko noted that aarch64_insn_patch_text_nosync() does not expect a
lm_alias() address and Catalin suggested it is safe to drop the
lm_alias() for runtime patching since the kernel text is readable. The
address passed to fixup function is interpreted as a __le32 and
dereferenced as is to read the opcode at the patch site.
No functional changes are intended.
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Futex hash computation requires a mask operation with read-only after init data that will be converted to a runtime constant in the subsequent commit. Introduce runtime_const_mask_32 to further optimize the mask operation in the futex hash computation hot path. Since all the current use-cases are of the form GENMASK(n, 0), with n > 0, a single: ubfx w0, w0, #0, #widthm1 // w0 = w0 [widthm1:0] instruction is used for amd64 to improve instruction dinsity and performance. "Arm A-profile A64 Instruction Set Architecture" manual, Sec. "A64 -- Base Instructions" [1] for UBFX instruction highlights the immediate "width" is encoded as width minus 1 in imms (Bits [15:10]) which is patched by __runtime_fixup_mask() once the mask is known. If a future use case arises that needs to tackle arbitrary mask, consider using: movz w1, #lo16, lsl #0 movk w1, #hi16, lsl #16 to patch the 32-bit mask in the asm block and return "__ret & (val)" from runtime_const_mask_32() which allows compiler to further optimize the logical and operation. __runtime_fixup_ptr() already patches a "movz, + movk lsl #16" sequence which can be reused when the need arises. A possible implementation for this alternate scheme can be found at [2]. Assisted-by: Claude:claude-sonnet-4-6 Suggested-by: Samuel Holland <samuel.holland@sifive.com> Suggested-by: Charlie Jenkins <thecharlesjenkins@gmail.com> Link: https://developer.arm.com/documentation/ddi0602/2026-03/Base-Instructions/ [1] Link: https://lore.kernel.org/lkml/20260430094730.31624-4-kprateek.nayak@amd.com/ [2] Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Define the placeholder used for lui + addi[w] patching sequence as RUNTIME_MAGIC and use that instead of open coding the constants in the inline assembly. No functional changes intended. Suggested-by: Guo Ren <guoren@kernel.org> Reviewed-by: Charlie Jenkins <thecharlesjenkins@gmail.com> Tested-by: Charlie Jenkins <thecharlesjenkins@gmail.com> Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Futex hash computation requires a mask operation with read-only after init data that will be converted to a runtime constant in the subsequent commit. Introduce runtime_const_mask_32 to further optimize the mask operation in the futex hash computation hot path. Since all the current use-cases are of the form GENMASK(n, 0), with n > 0, following sequence: srli a0, a1, imm slli a0, a0, imm is used for RISC-V where imm = (31 - width) to improve instruction density and performance. "The RISC-V Instruction Set Manual, Volume I - Unprivileged Architecture" [1] Sec. 2.4.1 "Integer Register-Immediate Instructions" notes the immediate shift for SRLI and SLLI are 5 bits wide starting at bit #10. __runtime_fixup_shift() is reused to patch the immediate shifts for the two instructions. If a future use case arises that needs to tackle arbitrary mask, consider using: lui a0, 0x12346 # upper; +0x800 then >>12 for correct rounding addi a0, a0, 0x678 # lower 12 bits to patch the 32-bit mask in the asm block and return "__ret & (val)" from runtime_const_mask_32() which allows compiler to further optimize the logical and operation. __runtime_fixup_ptr() already patches a lui + addi sequence which can be reused when the need arises. A possible implementation for this alternate scheme can be found at [2]. Assisted-by: Claude:claude-sonnet-4-5 Suggested-by: Samuel Holland <samuel.holland@sifive.com> Suggested-by: Charlie Jenkins <thecharlesjenkins@gmail.com> Link: https://docs.riscv.org/reference/isa/_attachments/riscv-unprivileged.pdf [1] Link: https://lore.kernel.org/lkml/20260430094730.31624-6-kprateek.nayak@amd.com/ [2] Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Futex hash computation requires a mask operation with read-only after init data that will be converted to a runtime constant in the subsequent commit. Introduce runtime_const_mask_32 to further optimize the mask operation in the futex hash computation hot path. GCC generates a: nilf %r1,<imm32> to tackle arbitrary 32-bit masks and the same is implemented here. Immediate patching pattern for __runtime_fixup_mask() has been adopted from __runtime_fixup_ptr(). Acked-by: Heiko Carstens <hca@linux.ibm.com> Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Add a dummy runtime_const_mask_32() for all the architectures that do not support runtime-const. Link: https://patch.msgid.link/20260227161841.GH606826@noisy.programming.kicks-ass.net Not-yet-signed-off-by: Peter Zijlstra <peterz@infradead.org> Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com> Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
Runtime constify the read-only after init data __futex_shift(shift_32),
__futex_mask(mask_32), and __futex_queues(ptr) used in __futex_hash()
hot path to avoid referencing global variable.
This also allows __futex_queues to be allocated dynamically to
"nr_node_ids" slots instead of reserving config dependent MAX_NUMNODES
(1 << CONFIG_NODES_SHIFT) worth of slots upfront.
Runtime constants are initialized before their first access and
runtime_const_init() provides necessary barrier to ensure subsequent
accesses are not reordered against their initialization.
No functional changes intended.
[ prateek: Dynamically allocate __futex_queues, mark the global data
__ro_after_init since they are constified after futex_init(). ]
Link: https://patch.msgid.link/20260227161841.GH606826@noisy.programming.kicks-ass.net
Reported-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> # MAX_NUMNODES bloat
Not-yet-signed-off-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: K Prateek Nayak <kprateek.nayak@amd.com>
Signed-off-by: Linux RISC-V bot <linux.riscv.bot@gmail.com>
|
Patch 1: "[v5,1/8] x86/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 1: "[v5,1/8] x86/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 1: "[v5,1/8] x86/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 1: "[v5,1/8] x86/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 1: "[v5,1/8] x86/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 1: "[v5,1/8] x86/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 1: "[v5,1/8] x86/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 1: "[v5,1/8] x86/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 1: "[v5,1/8] x86/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 1: "[v5,1/8] x86/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 1: "[v5,1/8] x86/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 1: "[v5,1/8] x86/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 2: "[v5,2/8] arm64/runtime-const: Use aarch64_insn_patch_text_nosync() for patching" |
|
Patch 2: "[v5,2/8] arm64/runtime-const: Use aarch64_insn_patch_text_nosync() for patching" |
|
Patch 2: "[v5,2/8] arm64/runtime-const: Use aarch64_insn_patch_text_nosync() for patching" |
|
Patch 2: "[v5,2/8] arm64/runtime-const: Use aarch64_insn_patch_text_nosync() for patching" |
|
Patch 2: "[v5,2/8] arm64/runtime-const: Use aarch64_insn_patch_text_nosync() for patching" |
|
Patch 2: "[v5,2/8] arm64/runtime-const: Use aarch64_insn_patch_text_nosync() for patching" |
|
Patch 2: "[v5,2/8] arm64/runtime-const: Use aarch64_insn_patch_text_nosync() for patching" |
|
Patch 2: "[v5,2/8] arm64/runtime-const: Use aarch64_insn_patch_text_nosync() for patching" |
|
Patch 2: "[v5,2/8] arm64/runtime-const: Use aarch64_insn_patch_text_nosync() for patching" |
|
Patch 6: "[v5,6/8] s390/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 6: "[v5,6/8] s390/runtime-const: Introduce runtime_const_mask_32()" |
|
Patch 7: "[v5,7/8] asm-generic/runtime-const: Add dummy runtime_const_mask_32()" |
|
Patch 7: "[v5,7/8] asm-generic/runtime-const: Add dummy runtime_const_mask_32()" |
|
Patch 7: "[v5,7/8] asm-generic/runtime-const: Add dummy runtime_const_mask_32()" |
|
Patch 7: "[v5,7/8] asm-generic/runtime-const: Add dummy runtime_const_mask_32()" |
|
Patch 7: "[v5,7/8] asm-generic/runtime-const: Add dummy runtime_const_mask_32()" |
|
Patch 7: "[v5,7/8] asm-generic/runtime-const: Add dummy runtime_const_mask_32()" |
|
Patch 7: "[v5,7/8] asm-generic/runtime-const: Add dummy runtime_const_mask_32()" |
|
Patch 7: "[v5,7/8] asm-generic/runtime-const: Add dummy runtime_const_mask_32()" |
|
Patch 7: "[v5,7/8] asm-generic/runtime-const: Add dummy runtime_const_mask_32()" |
|
Patch 7: "[v5,7/8] asm-generic/runtime-const: Add dummy runtime_const_mask_32()" |
|
Patch 7: "[v5,7/8] asm-generic/runtime-const: Add dummy runtime_const_mask_32()" |
|
Patch 7: "[v5,7/8] asm-generic/runtime-const: Add dummy runtime_const_mask_32()" |
|
Patch 8: "[v5,8/8] futex: Use runtime constants for __futex_hash() hot path" |
|
Patch 8: "[v5,8/8] futex: Use runtime constants for __futex_hash() hot path" |
|
Patch 8: "[v5,8/8] futex: Use runtime constants for __futex_hash() hot path" |
|
Patch 8: "[v5,8/8] futex: Use runtime constants for __futex_hash() hot path" |
|
Patch 8: "[v5,8/8] futex: Use runtime constants for __futex_hash() hot path" |
|
Patch 8: "[v5,8/8] futex: Use runtime constants for __futex_hash() hot path" |
|
Patch 8: "[v5,8/8] futex: Use runtime constants for __futex_hash() hot path" |
|
Patch 8: "[v5,8/8] futex: Use runtime constants for __futex_hash() hot path" |
|
Patch 8: "[v5,8/8] futex: Use runtime constants for __futex_hash() hot path" |
|
Patch 8: "[v5,8/8] futex: Use runtime constants for __futex_hash() hot path" |
|
Patch 8: "[v5,8/8] futex: Use runtime constants for __futex_hash() hot path" |
|
Patch 8: "[v5,8/8] futex: Use runtime constants for __futex_hash() hot path" |
89be0ce to
dc5ff20
Compare
PR for series 1118643 applied to workflow__riscv__fixes
Name: futex: Use runtime constants for futex_hash computation
URL: https://patchwork.kernel.org/project/linux-riscv/list/?series=1118643
Version: 5