diff --git a/Documentation/filesystems/bcachefs/swap.rst b/Documentation/filesystems/bcachefs/swap.rst new file mode 100644 index 0000000000000..8ab020994d79f --- /dev/null +++ b/Documentation/filesystems/bcachefs/swap.rst @@ -0,0 +1,262 @@ +.. SPDX-License-Identifier: GPL-2.0 + +========================== +bcachefs Swap File Support +========================== + +Overview +======== + +bcachefs supports swap files using the ``SWP_FS_OPS`` path, keeping the +filesystem in the I/O loop for swap operations. Unlike btrfs (which +disables COW, checksums, and compression for swap), bcachefs swap files +use the normal COW write path. Swap data gets checksumming, encryption, +compression, replication, and multi-device support. + +Usage:: + + fallocate -l 4G /mnt/bcachefs/swapfile + chmod 600 /mnt/bcachefs/swapfile + mkswap /mnt/bcachefs/swapfile + swapon /mnt/bcachefs/swapfile + +A small raw swap partition or zram as a safety net is recommended for +extreme memory pressure. + +The Reclaim Deadlock Problem +============================ + +Swap writes happen during memory reclaim — the kernel writes pages to +swap *because* it is out of memory. A COW filesystem needs memory for +btree updates, journal entries, and block allocation. If any allocation +in the swap write path tries to reclaim memory, reclaim tries to swap +more pages → filesystem re-entry → deadlock. + +How Other Filesystems Handle This +--------------------------------- + +**ext4, XFS** (non-COW): the kernel writes directly to pre-mapped +physical blocks via ``iomap_swapfile_activate``. The filesystem is not +involved after swapon. + +**btrfs**: disables COW, checksums, compression, RAID, and snapshots +for swap files. Effectively degrades to the ext4 model. + +**NFS**: uses ``SWP_FS_OPS`` — the only path where the filesystem stays +in the I/O loop. This is what we use. + +**dm-thin**: the existence proof that COW + swap can work. Uses +mempools for all critical allocations (1024-element pools that never +fail). + +Our Approach +============ + +Five mechanisms work together to make COW swap safe under memory +pressure. + +PF_MEMALLOC +----------- + +``swap_rw`` sets ``PF_MEMALLOC`` (via ``memalloc_noreclaim_save``) for +both reads and writes. This tells the page allocator to use emergency +reserves rather than entering direct reclaim. + +Why reads too: swap-in happens during page faults. If a read-path +allocation enters reclaim, reclaim may start swap writes that compete +for the same btree locks — deadlock. + +Critical detail: the write index update runs in a kworker thread +(``bch2_write_point_do_index_updates``), which does **not** inherit +task flags from the caller. The ``BCH_WRITE_swap`` flag on the write +op tells the worker to set ``PF_MEMALLOC`` for that op's duration. + +Btree Node Pinning +------------------ + +At swapon, leaf nodes for the extents, inodes, and alloc btrees +covering the swap file's key range are marked ``noevict``. This +prevents the btree cache shrinker from evicting them. + +Interior btree nodes are not pinned — they are few, hot, and covered +by the btree cache pre-reserve. + +Btree Cache Pre-reserve +----------------------- + +16 MB of btree node buffers are pre-allocated on ``bc->freeable`` at +swapon. ``bch2_btree_node_mem_alloc`` checks freeable first, stealing +a pre-allocated buffer instead of hitting the page allocator. + +The allocation uses ``GFP_NORETRY`` to avoid OOM on small VMs (the +pre-reserve is best-effort; partial allocation is handled gracefully). + +``bc->nr_reserve`` is bumped in parallel to reduce the shrinker's +``can_free`` budget, indirectly shielding the pre-allocated buffers +from being drained. + +Disk Reservation +---------------- + +``swap_pages × PAGE_SECTORS`` sectors are reserved at swapon. Each +COW write allocates a new physical block before freeing the old one. +The reservation reduces ``sectors_available``, causing normal writers +to ENOSPC sooner, preserving free buckets for swap. + +Swap writes themselves skip the reservation gate entirely: for a 1:1 +COW overwrite (same size, same replicas), ``disk_sectors_delta = 0`` +and the ``bch2_disk_reservation_add`` check in ``bch2_extent_update`` +is never reached. + +If the filesystem lacks space for the reservation, swapon fails +immediately with ENOSPC — failing loudly at mount time rather than +silently during reclaim. + +Bkey Buffer Pre-allocation +-------------------------- + +A 2048-byte buffer is allocated with ``GFP_NOWAIT`` before entering +``PF_MEMALLOC`` in the write index kworker. This avoids +``__GFP_NOFAIL`` WARN loops when ``bch2_bkey_buf_realloc`` spills +from its 96-byte on-stack buffer. + +``GFP_NOWAIT`` rather than ``GFP_KERNEL``: the kworker context can +amplify a pre-existing deadlock path where ``journal_write`` → +``kvmalloc`` → direct reclaim → btree cache shrinker → needs journal +→ deadlock. ``GFP_NOWAIT`` avoids entering reclaim entirely. If +the allocation fails, the existing ``__GFP_NOFAIL`` fallback under +``PF_MEMALLOC`` handles it (using emergency reserves). + +What Didn't Work +================ + +Pre-fragmentation of Extents +----------------------------- + +The idea: at swapon, split the swap file's extents to page granularity +so every COW swap write is a 1:1 extent replacement (same logical +range, different physical pointer) instead of a 1→3 split. This would +eliminate btree growth during swap I/O. + +**Why it fails**: ``bch2_trans_update_extent`` performs front/back +merging of adjacent extents with contiguous physical pointers. Since +the swap file is allocated contiguously (via ``fallocate``), each +split extent is immediately re-merged with its neighbor. The entire +65536-iteration prefrag loop is a no-op. + +All tests pass without pre-fragmentation. COW 1→3 splits at swap +write time work correctly via ``bch2_extent_trim_atomic``. + +Pre-fragmentation could be made to work by suppressing extent merging +during the split loop (e.g. a ``BTREE_UPDATE_no_merge`` flag) or by +inserting non-contiguous extents. Worth revisiting if profiling shows +the 1→3 splits are a bottleneck under sustained pressure. + +GFP_NOIO / GFP_NOFS for Individual Allocation Sites +---------------------------------------------------- + +Instead of ``PF_MEMALLOC``, we tried marking individual allocations +as ``GFP_NOIO`` or ``GFP_NOFS``. This is a game of whack-a-mole: +fixing one site (e.g. ``bch2_printbuf_make_room``) just moves the +deadlock to the next ``GFP_KERNEL`` allocation in the btree path. +``PF_MEMALLOC`` is the correct blanket fix — it prevents reclaim from +all allocations in the task. + +GFP_NOWAIT for Printbuf +------------------------ + +Making ``bch2_printbuf_make_room`` use ``GFP_NOWAIT`` causes the +allocation to fail, which makes the btree transaction retry, which +hits the next allocation. The retry loop burns CPU without progress. + +memalloc_noreclaim in swap_rw Only +----------------------------------- + +Setting ``PF_MEMALLOC`` only in the ``swap_rw`` caller doesn't help +because the write index update runs in a kworker thread that doesn't +inherit the caller's task flags. The ``BCH_WRITE_swap`` flag was +added specifically to propagate the noreclaim context to the kworker. + +GFP_KERNEL for Bkey Pre-allocation +------------------------------------ + +The initial bkey pre-alloc used ``GFP_KERNEL``. This triggered direct +reclaim in the write index kworker, amplifying the ``journal_write`` → +reclaim → btree shrinker → journal deadlock. Changing to +``GFP_NOWAIT`` eliminated the regression. + +Adversarial Analysis +==================== + +Dead-key Accumulation +--------------------- + +A btree node has at most 3 bsets. When a node is written to disk and +then dirtied, old keys become dead space in the written bset. Worst +case: a node starts at 50% live keys, all are overwritten via swap → +50% dead + 50% new live = 100% full. + +Natural throttle: journal pressure. Each swap write creates a journal +entry. When the journal fills, reclaim writes dirty btree nodes, +compacting dead keys and restoring headroom. Heavy swap → journal +fills → reclaim → compaction → headroom restored. + +Residual risk: if a burst of writes fills a node to 100% before +journal reclaim triggers, the insert fails. The 80% fill monitoring +(``btree/commit.c``) provides early warning. + +Shared Btree Nodes +------------------ + +The swap file's extent keys live in ``BTREE_ID_extents``, shared with +all other files. Other files' operations could insert keys into nodes +containing swap keys, eating into headroom. + +In practice, extent btree keys are ordered by ``(inode, offset, +snapshot)``. Different inodes share a leaf only when numerically +adjacent. The risk is confined to boundary nodes. + +Future options: reserved inode bit for swap files (no format change), +or a dedicated ``BTREE_ID_swap_extents`` (format change, structurally +eliminates sharing). + +ENOSPC During Swap Writes +-------------------------- + +Even with the disk reservation, free space may be in partially-used +buckets with zero completely-free buckets on a fragmented filesystem. +Copygc must consolidate, competing for I/O. The large reservation +helps copygc stay healthy, but ENOSPC during reclaim is extremely +unlikely rather than structurally impossible. + +Diagnostics +=========== + +- **80% fill monitoring**: rate-limited ``bch_info`` when extents btree + leaves exceed 80% fill (early warning before the 67% split + threshold). + +- **Swap I/O stall detection**: WARN at 2 seconds, BUG at 10 seconds + (debug builds only). Produces a crash dump with symbolized stacks + instead of a silent hang. + +- **Ablation toggles**: ``bcachefs.swap_nopin`` and + ``bcachefs.swap_noreclaim`` cmdline options disable individual + protections for A/B testing. + +Limitations and Future Work +============================ + +- ``PF_MEMALLOC`` uses emergency memory reserves. Under extreme + pressure these could be depleted. A small raw swap or zram as a + safety net is recommended. + +- Btree node pinning pins all alloc btree leaf nodes, which is + significant on large filesystems. Lazy pinning (pin on first use, + never unpin) is a future optimization. + +- No deferred btree splits under memory pressure. + +- No minimum free space check at swapon beyond the disk reservation + (a warning when free space is barely sufficient for copygc would + be useful). diff --git a/INVESTIGATION.md b/INVESTIGATION.md new file mode 100644 index 0000000000000..4c7f81508d3ea --- /dev/null +++ b/INVESTIGATION.md @@ -0,0 +1,301 @@ +# bcachefs SRCU lock held too long — crash investigation, audit, and patch series + +## The crash (2026-02-27 ~19:29-19:32 +08:00) + +Matthias' Arch Linux desktop machine (128 GB RAM, 32 cores, bcachefs root filesystem with +tiered SSD+HDD storage) became completely unresponsive and required a +hard reset. The previous boot ran kernel 6.18.9-arch1-2.1 with bcachefs +as a DKMS out-of-tree module. + +### Filesystem layout + +``` +/dev/sdd1:/dev/nvme0n1p3:/dev/sda:/dev/sdb:/dev/sdc on / type bcachefs + foreground_target: ssd + background_target: hdd + promote_target: ssd + background_compression: zstd:15 + data_replicas: 2 + metadata_replicas: 2 +``` + +Five devices: one NVMe (ssd label), four HDDs (hdd label). Data is +written to the SSD tier and reconcile moves it to the HDD tier with zstd +compression in the background. + +### Workload at the time + +Heavy development workload (~27 concurrent IDE sessions totalling ~68 GB +of anonymous memory), plus ~16 GB of kernel slab (bcachefs metadata). + +### Timeline (from `journalctl --boot=-1`) + +| Time | Event | +|------|-------| +| 19:29:28 | First `warn_alloc` (1471 callbacks suppressed) | +| 19:29:32 | kswapd0: order-0 page allocation failure. Free: ~76 MB. slab_unreclaimable: 16.6 GB. Anon: ~85 GB. Swap 750 GB free / 869 GB total. | +| 19:31:51 | Second failure. `warn_alloc: 17,143 callbacks suppressed`. Anon: ~90 GB. | +| 19:32:10 | bcachefs `do_reconcile_phys_thread` page allocation failure (`bch2_bio_alloc_pages` -> `bch2_data_update_init` -> `bch2_move_extent`). Free: ~45 MB. zswap shrinker caught in catch-22: decompression needs a page, but none available. | +| 19:32:26 | mypy page fault failure. Free: ~10 MB. | +| 19:32:32 | **`btree trans held srcu lock (delaying memory reclaim) for 13 seconds`** — two instances from `do_reconcile_phys_thread` on CPUs 10 and 13. | +| 19:32:52 | Last log entry. System unresponsive. Hard reset required. | + +### Why the OOM killer didn't fire + +No "Killed process" messages in the logs. bcachefs's SRCU read lock +blocked SRCU grace period completion, preventing reclaim of old btree +node memory. The system deadlocked in the reclaim path before OOM could +act. + +### Why swap didn't help + +Swap (869 GB on non-bcachefs partitions) was 85% free. The bottleneck +was not "where to put evicted pages" but "the kernel can't execute the +eviction": + +1. `slab_unreclaimable` (16.6 GB) is kernel memory, never eligible for + swap. +2. Reclaim needs temporary pages (zswap decompression buffers) but none + were available. +3. SRCU read lock prevented grace periods, blocking old btree node + reclaim. + +## Root cause analysis + +### The SRCU lock's role + +bcachefs uses an SRCU read lock to protect btree node memory from being +freed while transactions read it. Two unlock functions exist: + +- `bch2_trans_unlock(trans)` — releases btree node locks only +- `bch2_trans_unlock_long(trans)` — releases btree locks AND the SRCU + read lock + +When code calls `bch2_trans_unlock` before a blocking operation, the SRCU +lock is still held. If that blocking operation triggers page reclaim, +and reclaim needs to free bcachefs btree nodes, it must wait for the SRCU +grace period — which can't complete because we're holding the read lock. +Deadlock. + +### The specific crash path + +In `bch2_data_update_init()` (fs/bcachefs/data/update.c:1357), after +btree lookups are done, `bch2_trans_unlock(trans)` is called before +`bch2_data_update_bios_init()`, which does: + +- `kmalloc_array(nr_vecs, ..., GFP_KERNEL)` for bio vecs +- `bch2_bio_alloc_pages(..., GFP_KERNEL)` for the write bio + +Under memory pressure, these block in the page allocator for seconds +while the SRCU lock is held. + +## The fix + +Single commit on branch `fix/drop-srcu-pr`, based on `origin/master`. +Reproducer and investigation notes on the stacked branch +`fix/drop-srcu-before-data-update-alloc`. + +At each of the 25 sites below, `bch2_trans_unlock()` is changed to +`bch2_trans_unlock_long()`, or `drop_locks_do()` is changed to the new +`drop_locks_long_do()` macro (which also drops the SRCU read lock). + +Also introduces the `drop_locks_long_do()` macro in btree/iter.h and +cleans up `bch2_fsck_ask_yn()` in init/error.c (Kent's deferred +`unlock_long_at` workaround is no longer needed since we drop SRCU +immediately). + +### Summary: 25 sites fixed + +| Area | Sites | Operations | +|------|-------|------------| +| btree/cache.c | 4 | GFP_KERNEL allocs, sync I/O, wait_on_bit_io | +| btree/commit.c | 3 | GFP_KERNEL alloc, journal reclaim wait, journal res get | +| btree/interior.c | 5 | mutex, closure_sync, wait_event, down_read | +| btree/locking.c | 1 | mutex_lock | +| btree/read.c | 1 | sync disk I/O | +| alloc/foreground.c | 1 | mutex_lock | +| data/update.c | 3 | GFP_KERNEL bio alloc, closure_sync (x2) | +| data/ec/io.c | 1 | kvmalloc + multi-device I/O | +| data/migrate.c | 1 | closure_wait_event for in-flight writes | +| data/write.c | 1 | nocow lock + GFP_KERNEL | +| vfs/buffered.c | 1 | filemap_alloc_folio + GFP_KERNEL | +| vfs/fs.c | 1 | __wait_on_freeing_inode | +| init/error.c | 1 | user input wait | +| debug/tests.c | 1 | journal flush | + +## Related upstream issues + +Surveyed 14 issues on koverstreet/bcachefs. Our patch series directly +addresses the `bch2_data_update_init` bio allocation path and indirectly +helps many other paths. + +### Issues our patches help + +| Issue | Title | Status | Relevant path | +|-------|-------|--------|---------------| +| [#934](https://github.com/koverstreet/bcachefs/issues/934) | device evacuate: SRCU held 21s | Open | data_update_init + write buffer flush | +| [#636](https://github.com/koverstreet/bcachefs/issues/636) | SRCU held 48s during rebalance | Closed | data_update_init via do_rebalance | + +**Foreground I/O Stalls (Tiered Storage / Writeback)** +These patches were also found to completely resolve severe, multi-minute foreground I/O stalls (`ls`, `stat`, etc.) that occur when the system is performing heavy background writes to a slow tier (e.g. `dd` sequentially bypassing an SSD tier and writing directly to HDDs). Previously, the background writeback/promotion threads would hold the SRCU read lock while blocking on slow HDD I/O, freezing all foreground VFS operations that needed btree locks. With `bch2_trans_unlock_long()`, foreground I/O remains instantly responsive (e.g. 2ms) even under maximum background write pressure. + +### Issues with different root causes (not addressed by our patches) + +The most common SRCU-held-too-long path across issues is +`bch2_btree_write_buffer_flush_locked`, called from `bch2_trans_begin`. +This is an architectural issue — the write buffer flush happens inside +the transaction after SRCU is re-acquired, and can take a long time when +it needs to synchronize with journal reclaim or flush large numbers of +buffered keys. + +| Issue | Title | Status | Root cause | +|-------|-------|--------|------------| +| [#936](https://github.com/koverstreet/bcachefs/issues/936) | System freeze during snapshot remove | Open | write_buffer_flush mass-trigger | +| [#1021](https://github.com/koverstreet/bcachefs/issues/1021) | device remove: SRCU | Closed | reconcile write_buffer_flush | +| [#1028](https://github.com/koverstreet/bcachefs/issues/1028) | reclaim/reconcile both stuck | Closed | write_buffer_flush_seq mutex | +| [#1045](https://github.com/koverstreet/bcachefs/issues/1045) | Soft lockup during reconcile | Closed | sort in bp_scan | +| [#779](https://github.com/koverstreet/bcachefs/issues/779) | System lockup after NFS | Closed | rhashtable_insert_slow | +| [#605](https://github.com/koverstreet/bcachefs/issues/605) | SRCU held >10s (parent) | Closed | __bch2_create VFS path | +| [#811](https://github.com/koverstreet/bcachefs/issues/811) | Startup delays SRCU 19s | Closed | __bch2_create at boot | +| [#826](https://github.com/koverstreet/bcachefs/issues/826) | SRCU 13s interior update | Closed | btree_interior_update_work | +| [#807](https://github.com/koverstreet/bcachefs/issues/807) | OOM from fsck rhashtable | Closed | rhashtable growth in key cache | +| [#882](https://github.com/koverstreet/bcachefs/issues/882) | Excessive memory use | Open | btree_bounce_alloc | + +### Prior fixes by Kent Overstreet + +| Commit | Site | +|--------|------| +| 2ff6837f9be3 | btree/commit.c — journal reclaim wait timeout path | +| a727c2357464 | alloc/backpointers.c — bp scan sort | +| c4accde498dd | Early SRCU hold-time enforcement | + +## VM test reproducer + +### What the reproducer exercises + +The test creates a 128 MB RAM VM with pre-populated tiered bcachefs, +then eats 200 MB of memory to force every page allocation through +reclaim. The code paths exercised: + +**Reconcile thread** (`do_reconcile_phys_thread`): +- `bch2_data_update_init` → `bch2_data_update_bios_init` (commit 1) +- Btree lookups → `bch2_btree_node_fill` / `bch2_btree_node_mem_alloc` (commit 2) +- If btree nodes split → `bch2_btree_update_start` allocator wait (commit 4) + +**Btree commit path** (any transaction commit under pressure): +- Key cache insert → `btree_key_can_insert_cached_slowpath` (commit 3) +- Journal reclaim wait (commit 3) +- Journal reservation → `drop_locks_long_do` (commit 9) + +**Readahead** (if reconcile/user reads trigger it): +- `readpage_bio_extend` folio allocation (commit 7) + +**Not exercised** (would need different test setups): +- EC stripe reconstruction (no erasure coding configured) +- Device removal/migration +- `bch2_btree_write_buffer_flush_locked` (architectural, inside bch2_trans_begin) + +### Setup + +```bash +# Build kernel (tinyconfig + virtio + bcachefs + serial + swap) +make -j$(nproc) + +# Prepare disk images (one-time, needs root for loopback mount) +sudo ./prepare-vm-disks.sh + +# Build initramfs (static musl Rust binary, ~430 KB) +./build-initramfs.sh + +# Run test +./run-vm-test.sh fixed arch/x86/boot/bzImage +``` + +### How it works + +1. `prepare-vm-disks.sh` — formats bcachefs with tiered storage on the + host, writes 60 MB of data to the SSD tier, saves pristine images +2. `build-initramfs.sh` — builds the static Rust init binary (musl), + packs a minimal initramfs (~430 KB) +3. `run-vm-test.sh` — copies pristine images, boots 128 MB QEMU VM: + - bcachefs disks throttled to 512 KB/s write (slows reconcile) + - swap disk unthrottled (matches real crash: swap on non-bcachefs) + - 300s timeout +4. VM init (`vm-init-rs/`) — mounts swap, mounts pre-populated bcachefs + (reconcile starts immediately), forks a child that eats 200 MB, + prints heartbeats every 5s for 120s. If heartbeats stop, system hung. + +### Expected results + +- **Unfixed kernel**: SRCU warnings from multiple paths (btree cache + allocations, journal reclaim waits, reconcile bio allocation). + System may hang under sufficient pressure. +- **Fixed kernel**: no SRCU warnings, or warnings with much shorter + hold times. System remains responsive throughout. + +## Remaining work + +### Audit: sites NOT yet fixed + +Low-risk direct `bch2_trans_unlock` sites (no blocking follows, or +immediately followed by `bch2_trans_put`/`bch2_trans_begin`): + +- `btree/iter.c:3521` — `cond_resched()`, SRCU check follows immediately +- `btree/interior.c:878` — immediate `bch2_trans_begin` re-acquires SRCU +- `vfs/buffered.c:340` — `bch2_trans_put` releases SRCU right after +- `alloc/accounting.c:1145` — CPU-bound sort/fixup, no blocking + +Medium-risk `drop_locks_do` sites (bounded waits, could be converted): + +- `debug/debug.c:424` — `copy_to_user` (page fault possible) +- `btree/commit.c:960,1089` — `bch2_accounting_update_sb` (superblock I/O) +- `vfs/fs.c:489` — `__bch2_new_inode(GFP_NOFS)` (reclaim possible) +- `vfs/io.c:697,733` — pagecache operations +- `vfs/fiemap.c:144,265,276` — `copy_to_user` via fiemap + +### Architectural: `bch2_btree_write_buffer_flush_locked` + +The single most reported SRCU-held-too-long path (#934, #936, #1021, +#1028, #1045). Called from `bch2_trans_begin` which re-acquires SRCU +at the start of each transaction iteration. The flush can take a long +time when synchronizing with journal reclaim. This is not fixable with +simple `_long` conversions — it needs structural changes to how the +write buffer flush interacts with SRCU. + +### Cost analysis + +Each `bch2_trans_unlock_long` → `bch2_trans_relock` round-trip costs two +atomic operations (SRCU unlock + re-lock). Under normal conditions this +is negligible. Under memory pressure it's the difference between a +responsive system and a deadlock. + +## Files + +### Kernel patches + +All under `fs/bcachefs/`: + +- `btree/iter.h` — `drop_locks_long_do()` macro +- `btree/cache.c` — 4 sites +- `btree/commit.c` — 3 sites +- `btree/interior.c` — 5 sites +- `btree/locking.c` — 1 site +- `btree/read.c` — 1 site +- `alloc/foreground.c` — 1 site +- `data/update.c` — 3 sites +- `data/ec/io.c` — 1 site +- `data/migrate.c` — 1 site +- `data/write.c` — 1 site +- `vfs/buffered.c` — 1 site +- `vfs/fs.c` — 1 site +- `init/error.c` — 1 site (+ cleanup of Kent's deferred workaround) +- `debug/tests.c` — 1 site + +### Test infrastructure + +- `INVESTIGATION.md` — this file +- `vm-init-rs/` — static Rust init binary for the VM test +- `vm-disks/` — pristine disk images (fast, slow, swap) +- `prepare-vm-disks.sh` — format + pre-populate disk images on the host +- `build-initramfs.sh` — build static musl initramfs +- `run-vm-test.sh` — run test VM, analyze results diff --git a/build-initramfs.sh b/build-initramfs.sh new file mode 100755 index 0000000000000..23dcaa6352b5c --- /dev/null +++ b/build-initramfs.sh @@ -0,0 +1,38 @@ +#!/bin/bash +# Build a minimal initramfs for bcachefs SRCU lock testing. +# +# Uses the static Rust init binary (vm-init-rs) — no busybox, no shared libs. +# +# Usage: ./build-initramfs.sh [output-path] +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +INITRD="${1:-$SCRIPT_DIR/test-initramfs.img}" +WORKDIR=$(mktemp -d) + +echo "Building Rust init (static musl)..." +(cd "$SCRIPT_DIR/vm-init-rs" && cargo build --release --target x86_64-unknown-linux-musl --quiet) + +INIT_BIN="$SCRIPT_DIR/vm-init-rs/target/x86_64-unknown-linux-musl/release/vm-init" +if [ ! -f "$INIT_BIN" ]; then + echo "error: $INIT_BIN not found" >&2 + exit 1 +fi + +echo "Building initramfs in $WORKDIR" + +mkdir -p "$WORKDIR"/{dev,proc,sys,mnt/test,tmp} + +# The Rust binary is statically linked — it IS the init +cp "$INIT_BIN" "$WORKDIR/init" +chmod +x "$WORKDIR/init" + +# Minimal device nodes (devtmpfs is mounted by init, but console is needed early) +mknod "$WORKDIR/dev/console" c 5 1 +mknod "$WORKDIR/dev/null" c 1 3 + +# Pack it +(cd "$WORKDIR" && find . | cpio --quiet -o -H newc | gzip -9) > "$INITRD" + +echo "Initramfs written to $INITRD ($(du -h "$INITRD" | cut -f1))" +rm -rf "$WORKDIR" diff --git a/fs/bcachefs/Makefile b/fs/bcachefs/Makefile index aac94a8cdd4d7..d54e034b288ba 100644 --- a/fs/bcachefs/Makefile +++ b/fs/bcachefs/Makefile @@ -124,7 +124,8 @@ bcachefs-y := \ vfs/io.o \ vfs/buffered.o \ vfs/direct.o \ - vfs/pagecache.o + vfs/pagecache.o \ + vfs/swap.o ifdef CONFIG_DEBUG_FS bcachefs-y += debug/async_objs.o diff --git a/fs/bcachefs/alloc/foreground.c b/fs/bcachefs/alloc/foreground.c index 9e61fdef531ec..9ce57799fde7a 100644 --- a/fs/bcachefs/alloc/foreground.c +++ b/fs/bcachefs/alloc/foreground.c @@ -47,7 +47,7 @@ static void bch2_trans_mutex_lock_norelock(struct btree_trans *trans, struct mutex *lock) { if (!mutex_trylock(lock)) { - bch2_trans_unlock(trans); + bch2_trans_unlock_long(trans); mutex_lock(lock); } } diff --git a/fs/bcachefs/btree/bkey_buf.h b/fs/bcachefs/btree/bkey_buf.h index 7be03293d4530..9391260e475f4 100644 --- a/fs/bcachefs/btree/bkey_buf.h +++ b/fs/bcachefs/btree/bkey_buf.h @@ -56,6 +56,12 @@ static inline void bch2_bkey_buf_init(struct bkey_buf *s) bkey_init(&s->k->k); } +static inline void bch2_bkey_buf_init_prealloc(struct bkey_buf *s, void *buf) +{ + s->k = buf; + bkey_init(&s->k->k); +} + static inline void bch2_bkey_buf_exit(struct bkey_buf *s) { if (s->k != (void *) s->onstack) diff --git a/fs/bcachefs/btree/cache.c b/fs/bcachefs/btree/cache.c index 728205c87705b..0ce1cc3066810 100644 --- a/fs/bcachefs/btree/cache.c +++ b/fs/bcachefs/btree/cache.c @@ -54,6 +54,68 @@ void bch2_recalc_btree_reserve(struct bch_fs *c) c->btree.cache.nr_reserve = reserve; } +/* + * bch2_btree_cache_add_reserve - pre-allocate btree node buffers for swap I/O + * + * At swapon time, allocate @count empty btree node buffers and put them on + * bc->freeable. bch2_btree_node_mem_alloc() always checks freeable first and + * steals the data buffer from a freeable node rather than going to the page + * allocator — so under PF_MEMALLOC these buffers are used without hitting the + * emergency reserve at all. + * + * A single swap COW write touches at least three btrees (extents, inodes, and + * alloc/freespace), each up to BTREE_MAX_DEPTH levels deep. With mempool=8 + * the worst case is ~24 MB of concurrent node buffers; callers should reserve + * enough to cover their expected concurrency. + * + * We also increment nr_reserve by the same count. nr_reserve is the shrinker + * eviction watermark for bc->live[0]; raising it reduces the shrinker's + * can_free budget, which indirectly shields the freeable pool from being + * drained under memory pressure before swap I/O can use it. + * + * Returns the number of nodes actually allocated (may be less than @count on + * ENOMEM). + */ +static void __bch2_btree_node_to_freelist(struct bch_fs_btree_cache *, struct btree *); +static struct btree *__bch2_btree_node_mem_alloc_gfp(struct bch_fs *, gfp_t); + +int bch2_btree_cache_add_reserve(struct bch_fs *c, unsigned count) +{ + struct bch_fs_btree_cache *bc = &c->btree.cache; + unsigned allocated = 0; + + for (unsigned i = 0; i < count; i++) { + struct btree *b = __bch2_btree_node_mem_alloc_gfp(c, + GFP_KERNEL | __GFP_NORETRY); + + if (!b) + break; + + scoped_guard(mutex, &bc->lock) { + __bch2_btree_node_to_freelist(bc, b); + bc->nr_reserve++; + } + allocated++; + } + + return allocated; +} + +/* + * bch2_btree_cache_remove_reserve - release swap-time btree node reserve + * + * Called at swapoff. Decrements nr_reserve so the shrinker can reclaim the + * extra freeable buffers under normal memory pressure. The actual kvfree of + * the node data buffers happens lazily via the shrinker scan. + */ +void bch2_btree_cache_remove_reserve(struct bch_fs *c, unsigned count) +{ + struct bch_fs_btree_cache *bc = &c->btree.cache; + + scoped_guard(mutex, &bc->lock) + bc->nr_reserve -= min(bc->nr_reserve, (size_t)count); +} + static inline size_t btree_cache_can_free(struct btree_cache_list *list) { struct bch_fs_btree_cache *bc = @@ -224,22 +286,27 @@ static struct btree *__btree_node_mem_alloc(struct bch_fs *c, gfp_t gfp) return b; } -struct btree *__bch2_btree_node_mem_alloc(struct bch_fs *c) +static struct btree *__bch2_btree_node_mem_alloc_gfp(struct bch_fs *c, gfp_t gfp) { - struct btree *b = __btree_node_mem_alloc(c, GFP_KERNEL); + struct btree *b = __btree_node_mem_alloc(c, gfp); if (!b) return NULL; - if (btree_node_data_alloc(c, b, GFP_KERNEL, false)) { + if (btree_node_data_alloc(c, b, gfp, false)) { __btree_node_data_free(b); kfree(b); return NULL; } - bch2_btree_lock_init(&b->c, 0, GFP_KERNEL); + bch2_btree_lock_init(&b->c, 0, gfp); return b; } +struct btree *__bch2_btree_node_mem_alloc(struct bch_fs *c) +{ + return __bch2_btree_node_mem_alloc_gfp(c, GFP_KERNEL); +} + static inline bool __btree_node_pinned(struct bch_fs_btree_cache *bc, struct btree *b) { struct bbpos pos = BBPOS(b->c.btree_id, b->key.k.p); @@ -840,10 +907,12 @@ struct btree *bch2_btree_node_mem_alloc(struct btree_trans *trans, bool pcpu_rea } else { mutex_unlock(&bc->lock); bch2_trans_unlock(trans); + unsigned long _start = jiffies; b = __btree_node_mem_alloc(c, GFP_KERNEL); if (!b) goto err; bch2_btree_lock_init(&b->c, pcpu_read_locks ? SIX_LOCK_INIT_PCPU : 0, GFP_KERNEL); + bch2_trans_srcu_unlock_if_elapsed(trans, _start); mutex_lock(&bc->lock); } @@ -874,10 +943,12 @@ struct btree *bch2_btree_node_mem_alloc(struct btree_trans *trans, bool pcpu_rea if (btree_node_data_alloc(c, b, GFP_NOWAIT, true)) { bch2_trans_unlock(trans); + unsigned long _start = jiffies; if (btree_node_data_alloc(c, b, GFP_KERNEL|__GFP_NOWARN, true)) { __btree_node_data_free(b); goto err; } + bch2_trans_srcu_unlock_if_elapsed(trans, _start); } got_mem: @@ -1008,7 +1079,7 @@ static noinline struct btree *bch2_btree_node_fill(struct btree_trans *trans, /* Unlock before doing IO: */ six_unlock_intent(&b->c.lock); - bch2_trans_unlock(trans); + bch2_trans_unlock_long(trans); bch2_btree_node_read(trans, b, sync); @@ -1146,7 +1217,7 @@ static struct btree *__bch2_btree_node_get(struct btree_trans *trans, struct btr u32 seq = six_lock_seq(&b->c.lock); six_unlock_type(&b->c.lock, lock_type); - bch2_trans_unlock(trans); + bch2_trans_unlock_long(trans); bch2_btree_node_wait_on_read(b); diff --git a/fs/bcachefs/btree/cache.h b/fs/bcachefs/btree/cache.h index f9c3a65cb8f25..ae3f897a78420 100644 --- a/fs/bcachefs/btree/cache.h +++ b/fs/bcachefs/btree/cache.h @@ -11,6 +11,8 @@ extern const char * const bch2_btree_node_flags[]; struct btree_iter; void bch2_recalc_btree_reserve(struct bch_fs *); +int bch2_btree_cache_add_reserve(struct bch_fs *, unsigned count); +void bch2_btree_cache_remove_reserve(struct bch_fs *, unsigned count); void bch2_btree_node_to_freelist(struct bch_fs *, struct btree *); diff --git a/fs/bcachefs/btree/commit.c b/fs/bcachefs/btree/commit.c index acdab0cab4f01..d6432d998d71b 100644 --- a/fs/bcachefs/btree/commit.c +++ b/fs/bcachefs/btree/commit.c @@ -341,6 +341,13 @@ inline void bch2_btree_insert_key_leaf(struct btree_trans *trans, if (b->sib_u64s[1] != U16_MAX && live_u64s_added < 0) b->sib_u64s[1] = max(0, (int) b->sib_u64s[1] + live_u64s_added); + if (unlikely(b->nr.live_u64s * 5 > btree_max_u64s(c) * 4) && + b->c.btree_id == BTREE_ID_extents && b->c.level == 0) + bch_info_ratelimited(c, + "swap: extents leaf at %u%% fill (%u/%zu u64s)", + (unsigned)(b->nr.live_u64s * 100 / btree_max_u64s(c)), + b->nr.live_u64s, btree_max_u64s(c)); + if (u64s_added > live_u64s_added && bch2_maybe_compact_whiteouts(c, b)) bch2_trans_node_reinit_iter(trans, b); @@ -419,8 +426,9 @@ btree_key_can_insert_cached_slowpath(struct btree_trans *trans, unsigned flags, bch2_trans_unlock_updates_write(trans); bch2_trans_unlock(trans); + unsigned long _start = jiffies; - new_k = kmalloc(new_u64s * sizeof(u64), GFP_KERNEL); + new_k = kmalloc(new_u64s * sizeof(u64), GFP_NOFS); if (!new_k) { struct bch_fs *c = trans->c; bch_err(c, "error allocating memory for key cache key, btree %s u64s %u", @@ -934,7 +942,7 @@ static int __bch2_trans_commit_error(struct btree_trans *trans, unsigned flags, watermark < BCH_WATERMARK_reclaim) return bch_err_throw(c, journal_reclaim_would_deadlock); - return drop_locks_do(trans, + return drop_locks_escalating_do(trans, bch2_trans_journal_res_get(trans, (flags & BCH_WATERMARK_MASK)| JOURNAL_RES_GET_CHECK)); @@ -959,7 +967,7 @@ static int __bch2_trans_commit_error(struct btree_trans *trans, unsigned flags, case -BCH_ERR_btree_insert_need_mark_replicas: return drop_locks_do(trans, bch2_accounting_update_sb(trans)); case -BCH_ERR_btree_insert_need_journal_reclaim: - bch2_trans_unlock(trans); + bch2_trans_unlock_long(trans); event_inc_trace(c, trans_blocked_journal_reclaim, buf, ({ prt_printf(&buf, "%s\n", trans->fn); diff --git a/fs/bcachefs/btree/interior.c b/fs/bcachefs/btree/interior.c index 0584011a3f0bb..833349d18a442 100644 --- a/fs/bcachefs/btree/interior.c +++ b/fs/bcachefs/btree/interior.c @@ -834,7 +834,7 @@ static void btree_update_nodes_written(struct btree_update *as) * which may require allocations as well. */ - bch2_trans_unlock(trans); + bch2_trans_unlock_long(trans); /* * btree_interior_update_commit_lock is needed for synchronization with * btree_node_update_key(): having the lock be at the filesystem level @@ -1221,7 +1221,7 @@ bch2_btree_update_start(struct btree_trans *trans, struct btree_path *path, if (commit_flags & BCH_TRANS_COMMIT_journal_reclaim) return ERR_PTR(-BCH_ERR_journal_reclaim_would_deadlock); - ret = drop_locks_do(trans, + ret = drop_locks_long_do(trans, ({ wait_event(c->journal.wait, !journal_low_on_space(&c->journal)); 0; })); if (ret) return ERR_PTR(ret); @@ -1254,7 +1254,7 @@ bch2_btree_update_start(struct btree_trans *trans, struct btree_path *path, } if (!down_read_trylock(&c->gc.lock)) { - ret = drop_locks_do(trans, (down_read(&c->gc.lock), 0)); + ret = drop_locks_escalating_do(trans, (down_read(&c->gc.lock), 0)); if (ret) { up_read(&c->gc.lock); return ERR_PTR(ret); @@ -1349,7 +1349,9 @@ bch2_btree_update_start(struct btree_trans *trans, struct btree_path *path, if (!bch2_err_matches(ret, BCH_ERR_operation_blocked)) break; bch2_trans_unlock(trans); + unsigned long _start = jiffies; bch2_wait_on_allocator(c, req, ret, &cl); + bch2_trans_srcu_unlock_if_elapsed(trans, _start); } while (1); /* @@ -1358,8 +1360,14 @@ bch2_btree_update_start(struct btree_trans *trans, struct btree_path *path, * It would be nice if we could remove closures from waitlists * without waking up the waitlist: */ - if (closure_nr_remaining(&cl) > 1) + if (closure_nr_remaining(&cl) > 1) { bch2_trans_unlock(trans); + unsigned long _start = jiffies; + closure_sync(&cl); + bch2_trans_srcu_unlock_if_elapsed(trans, _start); + } else { + closure_sync(&cl); + } } if (ret) { diff --git a/fs/bcachefs/btree/iter.c b/fs/bcachefs/btree/iter.c index 056fadcf63f09..963c0667da5b5 100644 --- a/fs/bcachefs/btree/iter.c +++ b/fs/bcachefs/btree/iter.c @@ -30,6 +30,11 @@ #include #include +unsigned bch2_srcu_escalation_timeout_ms = 0; +module_param_named(srcu_escalation_timeout_ms, bch2_srcu_escalation_timeout_ms, uint, 0644); +MODULE_PARM_DESC(srcu_escalation_timeout_ms, + "0=always drop SRCU before blocking (original), N>0=two-phase with (N-1)ms timeout"); + static inline void btree_path_list_remove(struct btree_trans *, struct btree_path *); static inline void btree_path_list_add(struct btree_trans *, btree_path_idx_t, btree_path_idx_t); @@ -1738,7 +1743,7 @@ static noinline void btree_paths_realloc(struct btree_trans *trans) sizeof(struct btree_trans_paths) + nr * sizeof(struct btree_path) + nr * sizeof(btree_path_idx_t) + 8 + - nr * sizeof(struct btree_insert_entry), GFP_KERNEL|__GFP_NOFAIL); + nr * sizeof(struct btree_insert_entry), GFP_NOFS|__GFP_NOFAIL); unsigned long *paths_allocated = p; memcpy(paths_allocated, trans->paths_allocated, BITS_TO_LONGS(trans->nr_paths) * sizeof(unsigned long)); @@ -3510,6 +3515,9 @@ u32 bch2_trans_begin(struct btree_trans *trans) now = local_clock(); + if (!trans->last_yield_time) + trans->last_yield_time = now; + if (!IS_ENABLED(CONFIG_BCACHEFS_NO_LATENCY_ACCT) && time_after64(now, trans->last_begin_time + 10)) __bch2_time_stats_update(&btree_trans_stats(trans)->duration, @@ -3517,10 +3525,11 @@ u32 bch2_trans_begin(struct btree_trans *trans) if (!trans->restarted && (need_resched() || - time_after64(now, trans->last_begin_time + BTREE_TRANS_MAX_LOCK_HOLD_TIME_NS))) { + time_after64(now, trans->last_yield_time + BTREE_TRANS_MAX_LOCK_HOLD_TIME_NS))) { bch2_trans_unlock(trans); cond_resched(); now = local_clock(); + trans->last_yield_time = now; } trans->last_begin_time = now; @@ -3649,7 +3658,7 @@ struct btree_trans *__bch2_trans_get(struct bch_fs *c, unsigned fn_idx) if (s->max_mem) { unsigned expected_mem_bytes = roundup_pow_of_two(s->max_mem); - trans->mem = kmalloc(expected_mem_bytes, GFP_KERNEL|__GFP_NOWARN); + trans->mem = kmalloc(expected_mem_bytes, GFP_NOFS|__GFP_NOWARN); if (likely(trans->mem)) trans->mem_bytes = expected_mem_bytes; } @@ -3933,8 +3942,8 @@ int bch2_fs_btree_iter_init(struct bch_fs *c) if (!c->btree.trans.bufs) return -ENOMEM; - try(mempool_init_kmalloc_pool(&c->btree.trans.pool, 1, sizeof(struct btree_trans))); - try(mempool_init_kmalloc_pool(&c->btree.trans.malloc_pool, 1, BTREE_TRANS_MEM_MAX)); + try(mempool_init_kmalloc_pool(&c->btree.trans.pool, 8, sizeof(struct btree_trans))); + try(mempool_init_kmalloc_pool(&c->btree.trans.malloc_pool, 8, BTREE_TRANS_MEM_MAX)); try(init_srcu_struct(&c->btree.trans.barrier)); /* diff --git a/fs/bcachefs/btree/iter.h b/fs/bcachefs/btree/iter.h index 17445eee4b99e..d6df1efd3db30 100644 --- a/fs/bcachefs/btree/iter.h +++ b/fs/bcachefs/btree/iter.h @@ -346,6 +346,7 @@ int bch2_trans_relock(struct btree_trans *); int bch2_trans_relock_notrace(struct btree_trans *); void bch2_trans_unlock(struct btree_trans *); void bch2_trans_unlock_long(struct btree_trans *); +void bch2_trans_srcu_unlock_if_elapsed(struct btree_trans *, unsigned long); static inline int trans_was_restarted(struct btree_trans *trans, u32 restart_count) { @@ -1035,6 +1036,48 @@ struct bkey_s_c bch2_btree_iter_peek_root(struct btree_trans *, struct btree_ite (_do) ?: bch2_trans_relock(_trans); \ }) +/* + * Like drop_locks_do, but also drops the SRCU read lock so that SRCU grace + * periods can complete and the shrinker can free old btree nodes. Use only + * before operations known to block for an unbounded time (user input, disk + * I/O waits). + */ +#define drop_locks_long_do(_trans, _do) \ +({ \ + bch2_trans_unlock_long(_trans); \ + (_do) ?: bch2_trans_relock(_trans); \ +}) + +/* + * Two-phase unlock with runtime-tunable escalation timeout. + * + * srcu_escalation_timeout_ms controls the behavior: + * 0 = original behavior: always drop SRCU before the blocking op + * (equivalent to drop_locks_long_do) + * N > 0 = two-phase: keep SRCU during blocking op, escalate to + * unlock_long after (N-1) ms. So 1 = escalate after 0ms, + * 51 = escalate after 50ms, etc. + * + * Tunable at runtime via /sys/module/bcachefs/parameters/srcu_escalation_timeout_ms + */ +extern unsigned bch2_srcu_escalation_timeout_ms; + +#define drop_locks_escalating_do(_trans, _do) \ +({ \ + unsigned _timeout = bch2_srcu_escalation_timeout_ms; \ + if (_timeout == 0) { \ + bch2_trans_unlock_long(_trans); \ + } else { \ + bch2_trans_unlock(_trans); \ + } \ + unsigned long _start = jiffies; \ + int _ret = (_do); \ + if (_timeout && time_after(jiffies, \ + _start + msecs_to_jiffies(_timeout - 1))) \ + bch2_trans_srcu_unlock(_trans); \ + _ret ?: bch2_trans_relock(_trans); \ +}) + #define allocate_dropping_locks_errcode(_trans, _do) \ ({ \ gfp_t _gfp = GFP_NOWAIT; \ diff --git a/fs/bcachefs/btree/locking.c b/fs/bcachefs/btree/locking.c index ac0ab19e98a78..ff3cac32d5968 100644 --- a/fs/bcachefs/btree/locking.c +++ b/fs/bcachefs/btree/locking.c @@ -878,6 +878,19 @@ void bch2_trans_unlock_long(struct btree_trans *trans) bch2_trans_srcu_unlock(trans); } +/* + * Call after a blocking operation that was preceded by bch2_trans_unlock(). + * If the operation took longer than srcu_escalation_timeout_ms, also drop + * the SRCU read lock to allow grace periods to complete. + */ +void bch2_trans_srcu_unlock_if_elapsed(struct btree_trans *trans, + unsigned long start_time) +{ + if (time_after(jiffies, start_time + + msecs_to_jiffies(bch2_srcu_escalation_timeout_ms))) + bch2_trans_srcu_unlock(trans); +} + void bch2_trans_unlock_write(struct btree_trans *trans) { struct btree_path *path; @@ -892,7 +905,7 @@ void bch2_trans_unlock_write(struct btree_trans *trans) int __bch2_trans_mutex_lock(struct btree_trans *trans, struct mutex *lock) { - int ret = drop_locks_do(trans, (mutex_lock(lock), 0)); + int ret = drop_locks_escalating_do(trans, (mutex_lock(lock), 0)); if (ret) mutex_unlock(lock); diff --git a/fs/bcachefs/btree/locking.h b/fs/bcachefs/btree/locking.h index 9810a94a45ccb..c66d4fa56a00f 100644 --- a/fs/bcachefs/btree/locking.h +++ b/fs/bcachefs/btree/locking.h @@ -219,6 +219,7 @@ static inline void trans_set_unlocked(struct btree_trans *trans) lock_release(&trans->dep_map, _THIS_IP_); trans->locked = false; trans->last_unlock_ip = _RET_IP_; + trans->last_yield_time = 0; if (!trans->pf_memalloc_nofs) current->flags &= ~PF_MEMALLOC_NOFS; diff --git a/fs/bcachefs/btree/read.c b/fs/bcachefs/btree/read.c index 39c0e985a1d00..c3b639dcea312 100644 --- a/fs/bcachefs/btree/read.c +++ b/fs/bcachefs/btree/read.c @@ -1099,7 +1099,7 @@ static int __bch2_btree_root_read(struct btree_trans *trans, enum btree_id id, set_btree_node_read_in_flight(b); /* we can't pass the trans to read_done() for fsck errors, so it must be unlocked */ - bch2_trans_unlock(trans); + bch2_trans_unlock_long(trans); bch2_btree_node_read(trans, b, true); if (btree_node_read_error(b)) { diff --git a/fs/bcachefs/btree/types.h b/fs/bcachefs/btree/types.h index 9d83ed69e9de4..1eba1019b0002 100644 --- a/fs/bcachefs/btree/types.h +++ b/fs/bcachefs/btree/types.h @@ -582,6 +582,7 @@ struct btree_trans { #endif u64 last_begin_time; + u64 last_yield_time; unsigned long last_begin_ip; unsigned long last_restarted_ip; #ifdef CONFIG_BCACHEFS_DEBUG diff --git a/fs/bcachefs/btree/write.c b/fs/bcachefs/btree/write.c index 6930b89afc7cb..dba607024e647 100644 --- a/fs/bcachefs/btree/write.c +++ b/fs/bcachefs/btree/write.c @@ -490,10 +490,14 @@ void __bch2_btree_node_write(struct bch_fs *c, struct btree *b, unsigned flags) * blk-wbt.c throttles all writes except those that have both REQ_SYNC * and REQ_IDLE set... */ + blk_opf_t opf = REQ_OP_WRITE|REQ_META|REQ_SYNC|REQ_IDLE; + + if (type == BTREE_WRITE_journal_reclaim) + opf = REQ_OP_WRITE|REQ_META|REQ_SYNC|REQ_PRIO; wbio = container_of(bio_alloc_bioset(NULL, buf_pages(data, sectors_to_write << 9), - REQ_OP_WRITE|REQ_META|REQ_SYNC|REQ_IDLE, + opf, GFP_NOFS, &c->btree.bio), struct btree_write_bio, wbio.bio); diff --git a/fs/bcachefs/btree/write_buffer.c b/fs/bcachefs/btree/write_buffer.c index 06d54b1b4d537..2227de44e3f1d 100644 --- a/fs/bcachefs/btree/write_buffer.c +++ b/fs/bcachefs/btree/write_buffer.c @@ -649,8 +649,13 @@ static int btree_write_buffer_flush_seq(struct btree_trans *trans, u64 max_seq, /* * On memory allocation failure, bch2_btree_write_buffer_flush_locked() * is not guaranteed to empty wb->inc: + * + * PF_MEMALLOC prevents entering direct reclaim while holding + * wb->flushing.lock. Without it, btree node allocation inside + * the flush can enter reclaim → reclaim needs journal → journal + * needs write-buffer flush → blocked on our mutex → deadlock. */ - scoped_guard(memalloc_flags, PF_MEMALLOC_NOFS) { + scoped_guard(memalloc_flags, PF_MEMALLOC_NOFS|PF_MEMALLOC) { guard(mutex)(&wb->flushing.lock); ret = bch2_btree_write_buffer_flush_locked(trans, caller); } @@ -706,8 +711,10 @@ static int bch2_btree_write_buffer_flush_nocheck_rw(struct btree_trans *trans) int ret = 0; if (mutex_trylock(&wb->flushing.lock)) { - bch2_trans_unlock_long(trans); - ret = bch2_btree_write_buffer_flush_locked(trans, WB_FLUSH_tryflush); + scoped_guard(memalloc_flags, PF_MEMALLOC_NOFS|PF_MEMALLOC) { + bch2_trans_unlock_long(trans); + ret = bch2_btree_write_buffer_flush_locked(trans, WB_FLUSH_tryflush); + } mutex_unlock(&wb->flushing.lock); } @@ -793,7 +800,7 @@ static int bch2_btree_write_buffer_flush_thread(void *arg) if (kthread_should_stop()) break; - scoped_guard(memalloc_flags, PF_MEMALLOC_NOFS) { + scoped_guard(memalloc_flags, PF_MEMALLOC_NOFS|PF_MEMALLOC) { guard(mutex)(&wb->flushing.lock); CLASS(btree_trans, trans)(c); do { diff --git a/fs/bcachefs/data/ec/io.c b/fs/bcachefs/data/ec/io.c index bbd95248232dd..0c226b96173c2 100644 --- a/fs/bcachefs/data/ec/io.c +++ b/fs/bcachefs/data/ec/io.c @@ -494,7 +494,7 @@ int bch2_ec_read_extent(struct btree_trans *trans, struct bch_read_bio *rbio, } /* Don't hold btree locks for stripe buffer allocations, or IO */ - bch2_trans_unlock(trans); + bch2_trans_unlock_long(trans); ret = bch2_ec_stripe_buf_init(c, buf, offset, bio_sectors(&rbio->bio)); if (ret) diff --git a/fs/bcachefs/data/migrate.c b/fs/bcachefs/data/migrate.c index 24863bd828b28..3f66fb453d5f4 100644 --- a/fs/bcachefs/data/migrate.c +++ b/fs/bcachefs/data/migrate.c @@ -188,7 +188,7 @@ static int bch2_dev_metadata_drop(struct bch_fs *c, bch2_btree_iter_next_node(&iter); } - bch2_trans_unlock(trans); + bch2_trans_unlock_long(trans); bch2_btree_interior_updates_flush(c); BUG_ON(bch2_err_matches(ret, BCH_ERR_transaction_restart)); diff --git a/fs/bcachefs/data/move.c b/fs/bcachefs/data/move.c index 5bbcf95c4c298..2b720999b51b9 100644 --- a/fs/bcachefs/data/move.c +++ b/fs/bcachefs/data/move.c @@ -178,6 +178,9 @@ void bch2_moving_ctxt_init(struct moving_context *ctxt, ctxt->wp = wp; ctxt->wait_on_copygc = wait_on_copygc; + ctxt->max_sectors_in_flight = c->opts.move_bytes_in_flight >> 9; + ctxt->max_ios_in_flight = c->opts.move_ios_in_flight; + closure_init_stack(&ctxt->cl); mutex_init(&ctxt->lock); @@ -394,14 +397,14 @@ int bch2_move_ratelimit(struct moving_context *ctxt) } while (delay); /* - * XXX: these limits really ought to be per device, SSDs and hard drives - * will want different limits + * Per-context limits: set from c->opts by default, but may be + * overridden for device-aware limiting (e.g. rotational devices). */ move_ctxt_wait_event(ctxt, - atomic_read(&ctxt->write_sectors) < c->opts.move_bytes_in_flight >> 9 && - atomic_read(&ctxt->read_sectors) < c->opts.move_bytes_in_flight >> 9 && - atomic_read(&ctxt->write_ios) < c->opts.move_ios_in_flight && - atomic_read(&ctxt->read_ios) < c->opts.move_ios_in_flight); + atomic_read(&ctxt->write_sectors) < ctxt->max_sectors_in_flight && + atomic_read(&ctxt->read_sectors) < ctxt->max_sectors_in_flight && + atomic_read(&ctxt->write_ios) < ctxt->max_ios_in_flight && + atomic_read(&ctxt->read_ios) < ctxt->max_ios_in_flight); return 0; } diff --git a/fs/bcachefs/data/move.h b/fs/bcachefs/data/move.h index 6ffd1ff295c75..dea20748f2856 100644 --- a/fs/bcachefs/data/move.h +++ b/fs/bcachefs/data/move.h @@ -19,7 +19,9 @@ struct bch_read_bio; * - write_sectors/write_ios: read completion -> write completion * * bch2_move_ratelimit() blocks the caller until all counters are below - * c->opts.move_bytes_in_flight / move_ios_in_flight. + * max_sectors_in_flight / max_ios_in_flight. These default to the global + * fs options, but callers may override them for device-aware limiting + * (e.g. lower limits for rotational devices). * * Extent moves (bch2_move_extent) and stripe repairs (bch2_stripe_repair) * both account through these counters. @@ -38,6 +40,10 @@ struct moving_context { struct write_point_specifier wp; bool wait_on_copygc; + /* Per-context in-flight limits (default: from c->opts) */ + unsigned max_sectors_in_flight; + unsigned max_ios_in_flight; + /* For waiting on outstanding reads and writes: */ struct closure cl; diff --git a/fs/bcachefs/data/reconcile/work.c b/fs/bcachefs/data/reconcile/work.c index d4b18312a4738..10aea4a7b2b50 100644 --- a/fs/bcachefs/data/reconcile/work.c +++ b/fs/bcachefs/data/reconcile/work.c @@ -35,6 +35,11 @@ #include #include +static unsigned move_ios_in_flight_rotational = 8; +module_param(move_ios_in_flight_rotational, uint, 0644); +MODULE_PARM_DESC(move_ios_in_flight_rotational, + "Max in-flight IOs for background data moves on rotational devices (default 8)"); + #define RECONCILE_PHASE_TYPES() \ x(scan) \ x(btree) \ @@ -1166,6 +1171,14 @@ static CLOSURE_CALLBACK(do_reconcile_phys_thread) writepoint_ptr(&c->allocator.reconcile_write_point), true); + /* + * Rotational devices can only do ~100 random IOPS; a deep queue + * just causes head thrashing and holds btree transaction state + * for longer, blocking journal reclaim and sync. + */ + ctxt.max_ios_in_flight = move_ios_in_flight_rotational; + ctxt.max_sectors_in_flight = move_ios_in_flight_rotational << (17 - 9); /* ios * 128K */ + struct btree_trans *trans = ctxt.trans; CLASS(darray_reconcile_work, work)(); diff --git a/fs/bcachefs/data/update.c b/fs/bcachefs/data/update.c index 49ec333c59937..8a368d94fe435 100644 --- a/fs/bcachefs/data/update.c +++ b/fs/bcachefs/data/update.c @@ -629,6 +629,29 @@ void bch2_data_update_read_done(struct data_update *u) } } + if (unlikely(u->opts.checksum_only)) { + CLASS(btree_trans, trans)(c); + struct nonce nonce = extent_nonce(u->op.version, crc); + + crc.csum_type = u->op.csum_type; + crc.csum = bch2_checksum_bio(c, crc.csum_type, nonce, &rbio->bio); + + /* + * We don't need to rebuild the pointers manually here; + * data_update_index_update_nowrite will drop ptrs_kill (none in this case) + * and we just need it to write the updated CRC. + * But wait, data_update_index_update_nowrite only drops ptrs_kill. + * To update the CRC, we actually need to change the key we're giving it. + * u->k is the key that will be written. We should update the CRC in u->k + * before calling nowrite. + */ + bch2_bkey_narrow_crc(c, u->k.k, rbio->pick.crc, crc); + + u->op.error = data_update_index_update_nowrite(trans, u); + u->op.end_io(&u->op); + return; + } + /* write bio must own pages: */ BUG_ON(!u->op.wbio.bio.bi_vcnt); @@ -744,7 +767,9 @@ int bch2_update_unwritten_extent(struct btree_trans *trans, 0, &cl, &wp); if (bch2_err_matches(ret, BCH_ERR_operation_blocked)) { bch2_trans_unlock(trans); + unsigned long _start = jiffies; closure_sync(&cl); + bch2_trans_srcu_unlock_if_elapsed(trans, _start); continue; } @@ -777,7 +802,9 @@ int bch2_update_unwritten_extent(struct btree_trans *trans, if (closure_nr_remaining(&cl) != 1) { bch2_trans_unlock(trans); + unsigned long _start = jiffies; closure_sync(&cl); + bch2_trans_srcu_unlock_if_elapsed(trans, _start); } return ret; @@ -1273,7 +1300,7 @@ int bch2_data_update_init(struct btree_trans *trans, * replicas or durability settings have been changed since the extent * was written: */ - if (!m->op.nr_replicas) { + if (!m->op.nr_replicas && !m->opts.checksum_only) { /* if iter == NULL, it's just a promote */ if (iter) ret = bch2_extent_drop_ptrs(trans, iter, k, io_opts, &m->opts); @@ -1295,7 +1322,8 @@ int bch2_data_update_init(struct btree_trans *trans, * (i.e. trying to move a durability=2 replica to a target with a * single durability=2 device) */ - if (data_opts.type != BCH_DATA_UPDATE_copygc) { + if (data_opts.type != BCH_DATA_UPDATE_copygc && + !m->opts.checksum_only) { ret = __bch2_can_do_write(c, io_opts, &m->opts, &m->op.devs_have, k, NULL); if (ret) goto out; @@ -1307,7 +1335,8 @@ int bch2_data_update_init(struct btree_trans *trans, } } - if (!rhltable_insert_key(&c->update_table, &m->pos, &m->hash, bch_update_params)) + if (!m->opts.checksum_only && + !rhltable_insert_key(&c->update_table, &m->pos, &m->hash, bch_update_params)) m->on_hashtable = true; } else { if (unwritten) { @@ -1355,8 +1384,10 @@ int bch2_data_update_init(struct btree_trans *trans, } bch2_trans_unlock(trans); + unsigned long _start = jiffies; ret = bch2_data_update_bios_init(m, c, io_opts, buf_bytes); + bch2_trans_srcu_unlock_if_elapsed(trans, _start); if (ret) goto out_nocow_unlock; diff --git a/fs/bcachefs/data/update.h b/fs/bcachefs/data/update.h index 1f8dbd269562c..c5401d4a681d5 100644 --- a/fs/bcachefs/data/update.h +++ b/fs/bcachefs/data/update.h @@ -33,6 +33,7 @@ struct data_update_opts { u16 target; bool no_devs_have:1; bool checksum_paranoia:1; + bool checksum_only:1; unsigned read_dev; enum bch_read_flags read_flags; diff --git a/fs/bcachefs/data/write.c b/fs/bcachefs/data/write.c index 91910a2c532ec..b2550d5c028fc 100644 --- a/fs/bcachefs/data/write.c +++ b/fs/bcachefs/data/write.c @@ -25,6 +25,8 @@ #include "debug/async_objs.h" +#include "vfs/swap.h" + #include "fs/inode.h" #include "init/dev.h" @@ -373,7 +375,12 @@ static int bch2_write_index_default(struct bch_write_op *op) CLASS(btree_trans, trans)(c); struct bkey_buf sk __cleanup(bch2_bkey_buf_exit); - bch2_bkey_buf_init(&sk); + if (op->prealloc_bkey_buf) { + bch2_bkey_buf_init_prealloc(&sk, op->prealloc_bkey_buf); + op->prealloc_bkey_buf = NULL; + } else { + bch2_bkey_buf_init(&sk); + } do { bch2_trans_begin(trans); @@ -729,12 +736,55 @@ void bch2_write_point_do_index_updates(struct work_struct *work) op->flags |= BCH_WRITE_in_worker; + bool is_swap = (op->flags & BCH_WRITE_swap) && + bch2_swap_noreclaim_enabled; + + /* + * Pre-allocate the bkey spill buffer while we can still + * do normal allocations. Use GFP_NOWAIT to avoid entering + * direct reclaim — the kworker context can amplify the + * journal_write → reclaim → btree_shrinker → journal + * deadlock. If this fails, the old __GFP_NOFAIL path + * under PF_MEMALLOC will handle it. + */ + if (is_swap && !op->prealloc_bkey_buf) + op->prealloc_bkey_buf = kmalloc(2048, GFP_NOWAIT); + + /* + * Swap write ops must not enter direct reclaim — + * we're already in the swap writeback path and + * reclaim would try to swap more pages → deadlock. + */ + unsigned int noreclaim_flags = 0; + if (is_swap) + noreclaim_flags = memalloc_noreclaim_save(); + + u64 swap_start_ns = is_swap ? ktime_get_ns() : 0; + __bch2_write_index(op); + if (unlikely(op->prealloc_bkey_buf)) { + kfree(op->prealloc_bkey_buf); + op->prealloc_bkey_buf = NULL; + } + if (!(op->flags & BCH_WRITE_submitted)) __bch2_write(op); else bch2_write_done(&op->cl); + + if (is_swap) { + u64 elapsed = ktime_get_ns() - swap_start_ns; + memalloc_noreclaim_restore(noreclaim_flags); + + if (unlikely(elapsed > 2ULL * NSEC_PER_SEC)) { + pr_err("bcachefs: swap kworker STALL: %llu ms in index_updates", + elapsed / NSEC_PER_MSEC); + WARN_ON_ONCE(1); + } + if (unlikely(elapsed > 10ULL * NSEC_PER_SEC)) + BUG(); + } } } @@ -1436,8 +1486,10 @@ static bool bch2_nocow_write(struct bch_write_op *op) ptrs = bch2_bkey_ptrs_c(k); bch2_trans_unlock(trans); + unsigned long _start = jiffies; bch2_bkey_nocow_lock(c, ptrs, ~0U, BUCKET_NOCOW_LOCK_UPDATE); + bch2_trans_srcu_unlock_if_elapsed(trans, _start); /* * This could be handled better: If we're able to trylock the @@ -1560,7 +1612,15 @@ static void __bch2_write(struct bch_write_op *op) (!(op->flags & BCH_WRITE_submitted) && !(op->flags & BCH_WRITE_in_worker)); - guard(memalloc_flags)(PF_MEMALLOC_NOFS); + /* + * PF_MEMALLOC_NOFS: prevent filesystem re-entry from allocations. + * For swap writes (BCH_WRITE_swap): also set PF_MEMALLOC to + * prevent entering direct reclaim entirely — swap writes run + * during reclaim and must not recurse into it. + */ + guard(memalloc_flags)(PF_MEMALLOC_NOFS | + (((op->flags & BCH_WRITE_swap) && bch2_swap_noreclaim_enabled) + ? PF_MEMALLOC : 0)); if (unlikely(op->opts.nocow && c->opts.nocow_enabled) && @@ -1600,12 +1660,17 @@ static void __bch2_write(struct bch_write_op *op) PTR_ERR_OR_ZERO(req) ?: bch2_alloc_sectors_req(trans, req, op->write_point, &wp); })); - bch2_trans_unlock_long(trans); + bch2_trans_unlock(trans); + unsigned long _start = jiffies; + if (bch2_err_matches(ret, BCH_ERR_operation_blocked)) { - if (!wait_on_allocator_sync) + if (!wait_on_allocator_sync) { + bch2_trans_srcu_unlock_if_elapsed(trans, _start); break; + } bch2_wait_on_allocator(c, req, ret, &op->cl); + bch2_trans_srcu_unlock_if_elapsed(trans, _start); __bch2_write_index(op); op->wbio.failed.nr = 0; continue; diff --git a/fs/bcachefs/data/write.h b/fs/bcachefs/data/write.h index de409c2231934..e3eb24544b282 100644 --- a/fs/bcachefs/data/write.h +++ b/fs/bcachefs/data/write.h @@ -56,6 +56,7 @@ static inline void bch2_write_op_init(struct bch_write_op *op, struct bch_fs *c, op->new_i_size = U64_MAX; op->i_sectors_delta = 0; op->devs_need_flush = NULL; + op->prealloc_bkey_buf = NULL; } CLOSURE_CALLBACK(bch2_write); diff --git a/fs/bcachefs/data/write_types.h b/fs/bcachefs/data/write_types.h index 2f2ae1ddaa3e7..ecc04422679be 100644 --- a/fs/bcachefs/data/write_types.h +++ b/fs/bcachefs/data/write_types.h @@ -27,7 +27,8 @@ x(move) \ x(in_worker) \ x(submitted) \ - x(convert_unwritten) + x(convert_unwritten) \ + x(swap) enum __bch_write_flags { #define x(f) __BCH_WRITE_##f, @@ -120,6 +121,8 @@ struct bch_write_op { */ struct bch_devs_mask *devs_need_flush; + void *prealloc_bkey_buf; + /* Must be last: */ struct bch_write_bio wbio; }; diff --git a/fs/bcachefs/debug/tests.c b/fs/bcachefs/debug/tests.c index f80cfe4bcbc3a..935122cb27fb2 100644 --- a/fs/bcachefs/debug/tests.c +++ b/fs/bcachefs/debug/tests.c @@ -83,7 +83,9 @@ static int test_delete_written(struct bch_fs *c, u64 nr) return ret; bch2_trans_unlock(trans); + unsigned long _start = jiffies; bch2_journal_flush_all_pins(&c->journal); + bch2_trans_srcu_unlock_if_elapsed(trans, _start); ret = commit_do(trans, NULL, NULL, 0, bch2_btree_iter_traverse(&iter) ?: diff --git a/fs/bcachefs/init/error.c b/fs/bcachefs/init/error.c index 3361815a52eb4..bea72dbb1dd62 100644 --- a/fs/bcachefs/init/error.c +++ b/fs/bcachefs/init/error.c @@ -220,27 +220,16 @@ static enum ask_yn bch2_fsck_ask_yn(struct bch_fs *c, struct btree_trans *trans) return YN_NO; if (trans) - bch2_trans_unlock(trans); + bch2_trans_unlock_long(trans); - unsigned long unlock_long_at = trans ? jiffies + HZ * 2 : 0; darray_char line = {}; int ret; do { - unsigned long t; bch2_print(c, " (y,n, or Y,N for all errors of this type) "); -rewait: - t = unlock_long_at - ? max_t(long, unlock_long_at - jiffies, 0) - : MAX_SCHEDULE_TIMEOUT; - - int r = bch2_stdio_redirect_readline_timeout(stdio, &line, t); - if (r == -ETIME) { - bch2_trans_unlock_long(trans); - unlock_long_at = 0; - goto rewait; - } + int r = bch2_stdio_redirect_readline_timeout(stdio, &line, + MAX_SCHEDULE_TIMEOUT); if (r < 0) { ret = YN_NO; break; diff --git a/fs/bcachefs/journal/reclaim.c b/fs/bcachefs/journal/reclaim.c index 77b6b7eb573c4..18fa1f3e08be3 100644 --- a/fs/bcachefs/journal/reclaim.c +++ b/fs/bcachefs/journal/reclaim.c @@ -340,8 +340,12 @@ void bch2_journal_do_discards(struct journal *j) { struct bch_fs *c = container_of(j, struct bch_fs, journal); - for_each_member_device(c, ca) - bch2_journal_dev_do_discards(&ca->journal); + for_each_member_device(c, ca) { + if (test_bit(BCH_FS_rw_init_done, &c->flags)) + queue_work(j->discard_wq, &ca->journal.discard); + else + bch2_journal_dev_do_discards(&ca->journal); + } } /* diff --git a/fs/bcachefs/vfs/buffered.c b/fs/bcachefs/vfs/buffered.c index 0417b8bfd1e07..b4ca866d2d191 100644 --- a/fs/bcachefs/vfs/buffered.c +++ b/fs/bcachefs/vfs/buffered.c @@ -116,6 +116,8 @@ static int readpage_bio_extend(struct btree_trans *trans, { /* Don't hold btree locks while allocating memory: */ bch2_trans_unlock(trans); + unsigned long _start = jiffies; + bch2_trans_srcu_unlock_if_elapsed(trans, _start); while (bio_sectors(bio) < sectors_this_extent && bio->bi_vcnt < bio->bi_max_vecs) { diff --git a/fs/bcachefs/vfs/direct.c b/fs/bcachefs/vfs/direct.c index 2d82880cde6f7..ffe8e898e7485 100644 --- a/fs/bcachefs/vfs/direct.c +++ b/fs/bcachefs/vfs/direct.c @@ -366,7 +366,8 @@ static __always_inline long bch2_dio_write_done(struct dio_write *dio) return -EIOCBQUEUED; } - bch2_pagecache_block_put(inode); + if (!IS_SWAPFILE(&inode->v)) + bch2_pagecache_block_put(inode); kfree(dio->iov); @@ -497,7 +498,10 @@ static __always_inline long bch2_dio_write_loop(struct dio_write *dio) if (sync) dio->op.flags |= BCH_WRITE_sync; - dio->op.flags |= BCH_WRITE_check_enospc; + if (IS_SWAPFILE(&inode->v)) + dio->op.flags |= BCH_WRITE_swap; + else + dio->op.flags |= BCH_WRITE_check_enospc; ret = bch2_quota_reservation_add(c, inode, &dio->quota_res, bio_sectors(bio), true); @@ -589,6 +593,17 @@ ssize_t bch2_direct_write(struct kiocb *req, struct iov_iter *iter) if (!enumerated_ref_tryget(&c->writes, BCH_WRITE_REF_dio_write)) return -EROFS; + /* + * Swap I/O: the VFS sets IS_SWAPFILE on the inode during swapon. + * Skip generic_write_checks (rejects swap files with ETXTBSY), + * inode locking, and mtime updates. Swap I/O is serialized by + * the swap subsystem; PF_MEMALLOC is set by bch2_swap_rw(). + */ + if (IS_SWAPFILE(&inode->v)) { + locked = false; + goto swap_skip_checks; + } + inode_lock(&inode->v); ret = generic_write_checks(req, iter); @@ -603,17 +618,21 @@ ssize_t bch2_direct_write(struct kiocb *req, struct iov_iter *iter) if (unlikely(ret)) goto err_put_write_ref; +swap_skip_checks: if (unlikely((req->ki_pos|iter->count) & (block_bytes(c) - 1))) { ret = bch_err_throw(c, EINVAL_unaligned_io); goto err_put_write_ref; } inode_dio_begin(&inode->v); - bch2_pagecache_block_get(inode); + if (!IS_SWAPFILE(&inode->v)) + bch2_pagecache_block_get(inode); - extending = req->ki_pos + iter->count > inode->v.i_size; + extending = !IS_SWAPFILE(&inode->v) && + req->ki_pos + iter->count > inode->v.i_size; if (!extending) { - inode_unlock(&inode->v); + if (locked) + inode_unlock(&inode->v); locked = false; } @@ -637,7 +656,7 @@ ssize_t bch2_direct_write(struct kiocb *req, struct iov_iter *iter) dio->iter = *iter; dio->op.c = c; - if (unlikely(mapping->nrpages)) { + if (!IS_SWAPFILE(&inode->v) && unlikely(mapping->nrpages)) { ret = bch2_write_invalidate_inode_pages_range(mapping, req->ki_pos, req->ki_pos + iter->count - 1); @@ -651,7 +670,8 @@ ssize_t bch2_direct_write(struct kiocb *req, struct iov_iter *iter) inode_unlock(&inode->v); return ret; err_put_bio: - bch2_pagecache_block_put(inode); + if (!IS_SWAPFILE(&inode->v)) + bch2_pagecache_block_put(inode); bio_put(bio); inode_dio_end(&inode->v); err_put_write_ref: diff --git a/fs/bcachefs/vfs/fs.c b/fs/bcachefs/vfs/fs.c index c63f4118b1791..789b84ecaa2a3 100644 --- a/fs/bcachefs/vfs/fs.c +++ b/fs/bcachefs/vfs/fs.c @@ -32,6 +32,7 @@ #include "vfs/buffered.h" #include "vfs/direct.h" #include "vfs/pagecache.h" +#include "vfs/swap.h" #include #include @@ -355,7 +356,7 @@ static struct bch_inode_info *bch2_inode_hash_find(struct bch_fs *c, struct btre if (!trans) { __wait_on_freeing_inode(c, inode, inum); } else { - int ret = drop_locks_do(trans, + int ret = drop_locks_long_do(trans, (__wait_on_freeing_inode(c, inode, inum), 0)); if (ret) return ERR_PTR(ret); @@ -1541,6 +1542,9 @@ static const struct address_space_operations bch_address_space_operations = { .migrate_folio = filemap_migrate_folio, #endif .error_remove_folio = generic_error_remove_folio, + .swap_activate = bch2_swap_activate, + .swap_deactivate = bch2_swap_deactivate, + .swap_rw = bch2_swap_rw, }; struct bcachefs_fid { diff --git a/fs/bcachefs/vfs/fs.h b/fs/bcachefs/vfs/fs.h index 3d8cbf27ad816..80c9903c3cc53 100644 --- a/fs/bcachefs/vfs/fs.h +++ b/fs/bcachefs/vfs/fs.h @@ -43,6 +43,13 @@ struct bch_inode_info { struct bch_inode_unpacked ei_inode; struct delayed_work ei_writeback_timer; + + /* + * Disk reservation held while this inode is an active swap file. + * Sized to cover the full swap file so ENOSPC can't occur during + * swap COW writes. Zero when not a swap file. + */ + u64 ei_swap_reserved_sectors; }; #define bch2_pagecache_add_put(i) bch2_two_state_unlock(&(i)->ei_pagecache_lock, 0) diff --git a/fs/bcachefs/vfs/swap.c b/fs/bcachefs/vfs/swap.c new file mode 100644 index 0000000000000..737fd8ce69d7a --- /dev/null +++ b/fs/bcachefs/vfs/swap.c @@ -0,0 +1,310 @@ +// SPDX-License-Identifier: GPL-2.0 +#ifndef NO_BCACHEFS_FS + +#include "bcachefs.h" +#include "alloc/buckets.h" +#include "btree/cache.h" +#include "btree/iter.h" +#include "btree/update.h" +#include "data/extents.h" +#include "data/io_misc.h" +#include "data/write.h" +#include "vfs/fs.h" +#include "vfs/swap.h" +#include "vfs/direct.h" +#include "vfs/buffered.h" + +#include +#include +#include + +/* + * Swap file support for bcachefs. + * + * Uses the SWP_FS_OPS path (like NFS) so that bcachefs stays in the I/O + * loop for swap operations. This enables checksumming, encryption, + * replication, and multi-device support for swap data. + * + * Key design points: + * - Btree nodes are pinned (noevict) at swapon to avoid disk reads + * during memory reclaim + * - PF_MEMALLOC is set during swap I/O to prevent reclaim re-entry + * - BCH_WRITE_swap flag propagates the noreclaim context to the + * write index worker thread + */ + +/* + * Feature toggles for A/B testing. Disable via kernel cmdline: + * bcachefs.swap_nopin - disable btree node pinning + * bcachefs.swap_noreclaim - disable PF_MEMALLOC in swap_rw + */ +static bool bch2_swap_pin_enabled = true; +bool bch2_swap_noreclaim_enabled = true; /* also checked in data/write.c */ + +static int __init swap_nopin_setup(char *s) +{ + bch2_swap_pin_enabled = false; + return 1; +} +__setup("bcachefs.swap_nopin", swap_nopin_setup); + +static int __init swap_noreclaim_setup(char *s) +{ + bch2_swap_noreclaim_enabled = false; + return 1; +} +__setup("bcachefs.swap_noreclaim", swap_noreclaim_setup); + +/* + * Swap I/O diagnostics. + * + * Track in-flight swap ops and detect when they stall. Under memory + * pressure the write path can block indefinitely on allocation — + * we want to crash early with a useful stack trace rather than + * silently hang. + */ +static atomic_t bch2_swap_inflight = ATOMIC_INIT(0); +static atomic64_t bch2_swap_completed = ATOMIC64_INIT(0); +static atomic64_t bch2_swap_errors = ATOMIC64_INIT(0); + +/* Warn after 2 s, BUG after 10 s */ +#define SWAP_IO_WARN_NS (2ULL * NSEC_PER_SEC) +#define SWAP_IO_BUG_NS (10ULL * NSEC_PER_SEC) + +/* Pin leaf nodes in a btree covering a key range. */ +static int bch2_swap_pin_btree_range(struct btree_trans *trans, + enum btree_id btree, + struct bpos start, struct bpos end, + bool pin) +{ + int count = 0; + + int ret = __for_each_btree_node(trans, iter, btree, + start, 0, 0, BTREE_ITER_prefetch, b, ({ + if (bpos_gt(b->data->min_key, end)) + break; + + if (pin) + set_btree_node_noevict(b); + else + clear_btree_node_noevict(b); + count++; + 0; + })); + + return ret < 0 ? ret : count; +} + +static int bch2_swap_pin_unpin_nodes(struct bch_fs *c, + struct bch_inode_info *inode, + bool pin) +{ + int total = 0, ret; + + CLASS(btree_trans, trans)(c); + + u64 inum = inode->ei_inum.inum; + + ret = bch2_swap_pin_btree_range(trans, BTREE_ID_extents, + POS(inum, 0), POS(inum, U64_MAX), pin); + if (ret < 0) + return ret; + total += ret; + + ret = bch2_swap_pin_btree_range(trans, BTREE_ID_inodes, + POS(0, inum), POS(0, inum), pin); + if (ret < 0) + return ret; + total += ret; + + ret = bch2_swap_pin_btree_range(trans, BTREE_ID_alloc, + POS_MIN, SPOS_MAX, pin); + if (ret < 0) + return ret; + total += ret; + + return total; +} + +int bch2_swap_activate(struct swap_info_struct *sis, + struct file *file, sector_t *span) +{ + struct bch_inode_info *inode = file_bch_inode(file); + struct bch_fs *c = inode->v.i_sb->s_fs_info; + + if (!S_ISREG(inode->v.i_mode)) + return -EINVAL; + + int pinned = 0; + if (bch2_swap_pin_enabled) { + /* Pin after prefragmentation (more nodes to pin now) */ + pinned = bch2_swap_pin_unpin_nodes(c, inode, true); + if (pinned < 0) { + bch_err(c, "swap activate: failed to pin btree nodes: %s", + bch2_err_str(pinned)); + return pinned; + } + } + + /* + * Reserve disk space for the entire swap file. + * + * Each COW swap write allocates a new physical block before freeing + * the old one. Without a reservation, ENOSPC during reclaim is + * possible if the filesystem is near full — causing swap writes to + * fail, which prevents freeing memory, causing an OOM spiral. + * + * Reserving swap_pages × PAGE_SECTORS at swapon time guarantees + * that space can't be taken by other writers while swap is active. + */ + u64 swap_sectors = (u64)sis->pages * PAGE_SECTORS; + struct disk_reservation disk_res = + bch2_disk_reservation_init(c, c->opts.data_replicas); + int disk_ret = bch2_disk_reservation_get(c, &disk_res, + swap_sectors, + c->opts.data_replicas, 0); + if (disk_ret) { + bch_err(c, "swap activate: insufficient disk space for reservation (%llu sectors, %d replicas): %s", + swap_sectors, c->opts.data_replicas, + bch2_err_str(disk_ret)); + return disk_ret; + } + inode->ei_swap_reserved_sectors = disk_res.sectors; + + /* + * Pre-allocate btree node buffers on bc->freeable so that btree reads + * during swap I/O can steal a pre-allocated buffer rather than hitting + * the page allocator under PF_MEMALLOC. + * + * A single swap write traverses extents + inodes + alloc/freespace + * btrees (3 trees × BTREE_MAX_DEPTH levels × 256 KB = 3 MB minimum). + * We reserve 16 MB (= 16 MB / btree_node_size nodes) which covers + * ~5 concurrent fully-cold traversals with headroom. + */ + unsigned swap_reserve = (16 << 20) / c->opts.btree_node_size; + int reserved = bch2_btree_cache_add_reserve(c, swap_reserve); + + sis->flags |= SWP_FS_OPS; + *span = sis->pages; + + bch_info(c, "swap activated on inode %lu (%llu pages, %d nodes pinned, %d/%u btree nodes pre-reserved)", + inode->v.i_ino, (u64)sis->pages, pinned, reserved, swap_reserve); + + int ret = add_swap_extent(sis, 0, sis->max, 0); + if (ret < 0) { + bch2_btree_cache_remove_reserve(c, swap_reserve); + if (inode->ei_swap_reserved_sectors) { + struct disk_reservation dr = { + .sectors = inode->ei_swap_reserved_sectors, + }; + bch2_disk_reservation_put(c, &dr); + inode->ei_swap_reserved_sectors = 0; + } + if (bch2_swap_pin_enabled) + bch2_swap_pin_unpin_nodes(c, inode, false); + } + return ret; +} + +void bch2_swap_deactivate(struct file *file) +{ + struct bch_inode_info *inode = file_bch_inode(file); + struct bch_fs *c = inode->v.i_sb->s_fs_info; + + if (bch2_swap_pin_enabled) + bch2_swap_pin_unpin_nodes(c, inode, false); + + unsigned swap_reserve = (16 << 20) / c->opts.btree_node_size; + bch2_btree_cache_remove_reserve(c, swap_reserve); + + if (inode->ei_swap_reserved_sectors) { + struct disk_reservation disk_res = { + .sectors = inode->ei_swap_reserved_sectors, + }; + bch2_disk_reservation_put(c, &disk_res); + inode->ei_swap_reserved_sectors = 0; + } + + bch_info(c, "swap deactivated on inode %lu", inode->v.i_ino); +} + +/* + * Swap I/O callback — called for every swap read/write when SWP_FS_OPS + * is set. Returns bytes transferred or -EIOCBQUEUED for async I/O. + */ +int bch2_swap_rw(struct kiocb *iocb, struct iov_iter *iter) +{ + struct bch_fs *c = file_inode(iocb->ki_filp)->i_sb->s_fs_info; + u64 start_ns = ktime_get_ns(); + int rw = iov_iter_rw(iter); + + atomic_inc(&bch2_swap_inflight); + + iocb->ki_flags |= IOCB_DIRECT; + + /* + * Prevent reclaim re-entry for both writes AND reads. + * + * Writes: swap writeback runs during reclaim, so allocations in + * the write path must not trigger reclaim (circular dependency). + * + * Reads: swap-in happens during page fault. If a read-path + * allocation enters reclaim → reclaim tries to swap out other + * pages → those writes compete for the same btree locks as the + * read → deadlock. + * + * PF_MEMALLOC bypasses watermarks and skips direct reclaim. + */ + unsigned int noreclaim_flags = 0; + if (bch2_swap_noreclaim_enabled) + noreclaim_flags = memalloc_noreclaim_save(); + + ssize_t ret; + if (rw == READ) + ret = bch2_read_iter(iocb, iter); + else + ret = bch2_write_iter(iocb, iter); + + if (bch2_swap_noreclaim_enabled) + memalloc_noreclaim_restore(noreclaim_flags); + + atomic_dec(&bch2_swap_inflight); + + u64 elapsed_ns = ktime_get_ns() - start_ns; + + if (ret < 0 && ret != -EIOCBQUEUED) { + atomic64_inc(&bch2_swap_errors); + bch_err_ratelimited(c, "swap_rw %s error %li at pos %lld " + "(inflight=%d completed=%lld errors=%lld)", + rw == READ ? "read" : "write", + ret, iocb->ki_pos, + atomic_read(&bch2_swap_inflight), + atomic64_read(&bch2_swap_completed), + atomic64_read(&bch2_swap_errors)); + } else { + atomic64_inc(&bch2_swap_completed); + } + + /* + * Detect stalled swap I/O. If a single operation takes >2 s, + * something is badly wrong (likely PF_MEMALLOC reserves exhausted + * or deadlock). WARN at 2 s; in debug builds, BUG at 10 s to get + * a full crash dump with symbolized stacks instead of a silent hang. + */ + if (unlikely(elapsed_ns > SWAP_IO_WARN_NS)) { + bch_err(c, "swap_rw %s STALL: %llu ms at pos %lld " + "(inflight=%d completed=%lld errors=%lld)", + rw == READ ? "read" : "write", + elapsed_ns / NSEC_PER_MSEC, iocb->ki_pos, + atomic_read(&bch2_swap_inflight), + atomic64_read(&bch2_swap_completed), + atomic64_read(&bch2_swap_errors)); + WARN_ON_ONCE(1); + } + if (unlikely(elapsed_ns > SWAP_IO_BUG_NS)) + BUG_ON(IS_ENABLED(CONFIG_BCACHEFS_DEBUG)); + + return ret; +} + +#endif /* NO_BCACHEFS_FS */ diff --git a/fs/bcachefs/vfs/swap.h b/fs/bcachefs/vfs/swap.h new file mode 100644 index 0000000000000..648335c90ab9d --- /dev/null +++ b/fs/bcachefs/vfs/swap.h @@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _BCACHEFS_VFS_SWAP_H +#define _BCACHEFS_VFS_SWAP_H + +#ifndef NO_BCACHEFS_FS + +struct swap_info_struct; + +extern bool bch2_swap_noreclaim_enabled; + +int bch2_swap_activate(struct swap_info_struct *, struct file *, sector_t *); +void bch2_swap_deactivate(struct file *); +int bch2_swap_rw(struct kiocb *, struct iov_iter *); + +#endif /* NO_BCACHEFS_FS */ +#endif /* _BCACHEFS_VFS_SWAP_H */ diff --git a/prepare-vm-disks.sh b/prepare-vm-disks.sh new file mode 100755 index 0000000000000..49466d1ecbcce --- /dev/null +++ b/prepare-vm-disks.sh @@ -0,0 +1,66 @@ +#!/bin/bash +# Prepare pre-populated disk images for the bcachefs SRCU test VM. +# +# Run this ONCE on the host (needs root for loopback mount). +# Creates vm-disks/{fast,slow,swap}-pristine.img with ~60 MB of data +# already on the SSD tier, ready for reconcile when the VM boots. +# +# Usage: sudo ./prepare-vm-disks.sh +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +DISK_DIR="$SCRIPT_DIR/vm-disks" +MNT=$(mktemp -d) + +mkdir -p "$DISK_DIR" + +echo "=== Creating disk images ===" +truncate --size=256M "$DISK_DIR/fast-pristine.img" +truncate --size=256M "$DISK_DIR/slow-pristine.img" +truncate --size=512M "$DISK_DIR/swap-pristine.img" + +LOOP_FAST=$(losetup --find --show "$DISK_DIR/fast-pristine.img") +LOOP_SLOW=$(losetup --find --show "$DISK_DIR/slow-pristine.img") +LOOP_SWAP=$(losetup --find --show "$DISK_DIR/swap-pristine.img") + +cleanup() { + umount "$MNT" 2>/dev/null || true + losetup -d "$LOOP_FAST" 2>/dev/null || true + losetup -d "$LOOP_SLOW" 2>/dev/null || true + losetup -d "$LOOP_SWAP" 2>/dev/null || true + rmdir "$MNT" 2>/dev/null || true +} +trap cleanup EXIT + +echo "=== Formatting bcachefs (SSD=$LOOP_FAST, HDD=$LOOP_SLOW) ===" +bcachefs format \ + --label=ssd "$LOOP_FAST" \ + --label=hdd "$LOOP_SLOW" \ + --foreground_target=ssd \ + --background_target=hdd \ + --promote_target=ssd \ + --background_compression=zstd \ + --data_replicas=1 \ + --metadata_replicas=1 + +echo "=== Mounting bcachefs ===" +mount -t bcachefs "$LOOP_FAST:$LOOP_SLOW" "$MNT" + +echo "=== Writing 60 MB to SSD tier ===" +for i in $(seq 0 5); do + dd if=/dev/urandom of="$MNT/testfile$i" bs=1M count=10 status=none +done +sync + +echo "=== Filesystem usage ===" +bcachefs fs usage "$MNT" 2>/dev/null || true + +echo "=== Unmounting ===" +umount "$MNT" + +echo "=== Setting up swap ===" +mkswap "$LOOP_SWAP" + +echo "=== Done ===" +echo "Pristine images in $DISK_DIR/" +ls -lh "$DISK_DIR/"*.img diff --git a/run-vm-test.sh b/run-vm-test.sh new file mode 100755 index 0000000000000..f7e07be1d4d89 --- /dev/null +++ b/run-vm-test.sh @@ -0,0 +1,75 @@ +#!/bin/bash +# Run a bcachefs SRCU lock test VM. +# +# Usage: run-vm-test.sh