Implement Runtime NVMe Instance Storage Discovery Using AWS EBS Symlinks

[neddp]

Problem

On AWS Nitro-based instances with NVMe devices, the kernel's PCIe enumeration order is non-deterministic. This means:

• /dev/nvme0n1 could be the root EBS volume OR instance storage
• /dev/nvme1n1 could be instance storage OR the root EBS volume
• The order varies between boots and instance types
• There is no guaranteed ordering

Solution

Implemented runtime discovery to reliably identify instance storage by excluding EBS volumes.

Discovery Algorithm

1. Glob all NVMe devices: /dev/nvme*n1
2. Glob EBS symlinks: /dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_*
3. Resolve each symlink to its target device
4. Subtract EBS devices from all NVMe devices = instance storage
5. Validate count matches CPI expectations
6. Partition only the discovered instance storage devices

Why EBS Symlinks Are Reliable

AWS automatically creates persistent symlinks for all EBS volumes via udev rules:

/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol{volume_id}

Backwards Compatibility

Non-NVMe instances: No changes to behavior

• Traditional Xen instances (/dev/xvdb, /dev/sdb) use CPI paths directly
• Paravirtual instances work as before

This must be merged together with the CPI changes - cloudfoundry/bosh-aws-cpi-release#196

---

Pair @Ivaylogi98

[rkoster]

In general I would have expected this logic to go into the https://github.com/cloudfoundry/bosh-agent/tree/main/infrastructure/devicepathresolver package.

[neddp]

In general I would have expected this logic to go into the https://github.com/cloudfoundry/bosh-agent/tree/main/infrastructure/devicepathresolver package.

Thank you for the review! That's was a big oversight on my end, I'll look into it.

[rkoster]

No worries 🙂

[beyhan]

We discussed this during the FI WG meeting and this have to relay on the stemcell agent settings and agent strategy for disc handling.

[rkoster]

As discussed during the working group meeting, focus is now on validating: cloudfoundry/bosh-aws-cpi-release#196 (comment)

[rkoster]

As per: cloudfoundry/bosh-aws-cpi-release#196 (comment) this change is still needed. Please continue reviewing.

[rkoster]

Still think this PR could be done in an IaaS agnostic way.

[rkoster]

@neddp could you take a look at these failing unit tests?

[neddp]

Hi @rkoster,

We still haven't had the time to test the changes on an actual deployment. I will move the PR to draft until we can confirm everything is working fine.

We'll address the tests as well.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

Adds SymlinkDeviceResolver with NVMe constants, constructor, ResolveSymlinksToDevices, GetDevicesByPattern, and FilterDevices plus Ginkgo tests. Wires a SymlinkDeviceResolver into NewProvider and NewLinuxPlatform. Refactors Linux SetupRawEphemeralDisks to discover instance-storage devices (NVMe glob + symlink exclusion or identity resolution), sort/validate discovered devices, and partition discovered device paths. Also updates fake resolver recording and platform tests to inject and use the new resolver.

Suggested reviewers

• aramprice

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main change: implementing runtime NVMe instance storage discovery using AWS EBS symlinks as an exclusion mechanism.
Description check	✅ Passed	The description comprehensively explains the problem (non-deterministic NVMe enumeration), the solution (runtime discovery via EBS symlink exclusion), the algorithm, and backwards compatibility considerations.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix-nvme-instance-storage-discovery

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[neddp]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[neddp]

Both suggestions were addressed.

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

platform/linux_platform.go (1)

859-875: ⚠️ Potential issue | 🔴 Critical | ⚡ Quick win

Fail closed when managed-volume symlink resolution is incomplete.

managedDevices is treated as authoritative here, but the resolver upstream currently skips unreadable symlinks. If one EBS symlink is missed and the filtered count still matches len(devices), the code below can mklabel a managed volume or even the root disk. Please make unresolved managed-volume symlinks abort discovery instead of continuing with a partial exclusion set.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@platform/linux_platform.go` around lines 859 - 875, ResolveSymlinksToDevices
currently may skip unreadable symlinks, letting managedDevices be incomplete;
change the resolver to surface skipped/unresolved symlinks (e.g., change
ResolveSymlinksToDevices to return (devices []string, skipped int, err error) or
return an error when any symlink cannot be read) and update this call site in
linux_platform.go to treat any skipped/unresolved count or non-nil error as
fatal: after calling p.symlinkDeviceResolver.ResolveSymlinksToDevices, if
skipped>0 (or err != nil) return an error instead of continuing, so
managedDevices cannot be partial before calling FilterDevices/instanceStorage
and proceeding with mklabel operations.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@platform/linux_platform.go`:
- Around line 883-888: The loop calling devicePathResolver.GetRealDevicePath
ignores the boolean timedOut return; update the loop in linux_platform.go (where
devicePathResolver.GetRealDevicePath is invoked) to check the timedOut flag and
treat it as an explicit failure: if timedOut or realPath == "" return a wrapped
error (similar to other call sites) instead of continuing, so the function
returns a clear timeout error rather than allowing an empty path to be passed to
parted; use the same bosherr.WrapErrorf pattern and include context mentioning
the device and that resolution timed out.

---

Outside diff comments:
In `@platform/linux_platform.go`:
- Around line 859-875: ResolveSymlinksToDevices currently may skip unreadable
symlinks, letting managedDevices be incomplete; change the resolver to surface
skipped/unresolved symlinks (e.g., change ResolveSymlinksToDevices to return
(devices []string, skipped int, err error) or return an error when any symlink
cannot be read) and update this call site in linux_platform.go to treat any
skipped/unresolved count or non-nil error as fatal: after calling
p.symlinkDeviceResolver.ResolveSymlinksToDevices, if skipped>0 (or err != nil)
return an error instead of continuing, so managedDevices cannot be partial
before calling FilterDevices/instanceStorage and proceeding with mklabel
operations.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: ab6aff62-71d1-4d01-9a14-e284e52205e7

📥 Commits

Reviewing files that changed from the base of the PR and between c43c6a1 and 3b63c11.

📒 Files selected for processing (1)

• platform/linux_platform.go

[aramprice]

/coderabbitai review

[neddp]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@infrastructure/devicepathresolver/symlink_device_resolver_test.go`:
- Around line 88-90: The test currently expects an error from
resolver.ResolveSymlinksToDevices("/dev/disk/by-id/nvme-*") but doesn't assert
the returned mapping is nil; update the test to also assert that the returned
result (the value assigned from ResolveSymlinksToDevices) is nil on failure to
ensure no partial mapping is returned. Locate the call to
ResolveSymlinksToDevices and the variables holding its return values (e.g.,
result, err) and add an assertion like Expect(result).To(BeNil()) immediately
after Expect(err).To(HaveOccurred()) /
Expect(err.Error()).To(ContainSubstring("nvme-invalid")) so the contract on
error paths is enforced.

In `@platform/linux_platform.go`:
- Around line 884-889: The current call to
p.devicePathResolver.GetRealDevicePath handles err before checking timedOut,
which incorrectly treats cases where timedOut==true and err!=nil as a generic
resolver error; update the logic in the function containing this call so that
you check the timedOut boolean first and return the timeout-specific error
(bosherr.Errorf("Timed out resolving device path for %+v", device)) when
timedOut is true, otherwise handle err via bosherr.WrapErrorf; reference
GetRealDevicePath, devicePathResolver, timedOut, and the existing
bosherr.WrapErrorf/bosherr.Errorf calls to locate and adjust the branches
accordingly.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 8929d5c5-a5fd-4727-be1e-bbcf81c8484c

📥 Commits

Reviewing files that changed from the base of the PR and between c43c6a1 and 27f51d4.

📒 Files selected for processing (4)

• infrastructure/devicepathresolver/symlink_device_resolver.go
• infrastructure/devicepathresolver/symlink_device_resolver_test.go
• platform/linux_platform.go
• platform/linux_platform_test.go

[neddp]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.