Skip to content

Optimize REST calls for single-page virtual directory listings#2295

Open
syeleti-msft wants to merge 1 commit into
mainfrom
syeleti/list-optimize
Open

Optimize REST calls for single-page virtual directory listings#2295
syeleti-msft wants to merge 1 commit into
mainfrom
syeleti/list-optimize

Conversation

@syeleti-msft

Copy link
Copy Markdown
Member

Summary

Optimize Block Blob directory listings by avoiding per-directory marker lookups when the hierarchical listing completes in a single page.

  • Detect a single-page listing when both the incoming and outgoing continuation markers are empty.
  • Create attributes for markerless virtual directories directly from BlobPrefix entries.
  • Preserve the existing dirList marker-deduplication logic.
  • Continue using REST marker checks for paginated listings, where the marker blob may appear on another page.
  • Add unit tests and update the changelog.

Motivation

Previously, listing a directory containing multiple markerless virtual directories triggered one GetProperties request for every BlobPrefix.

For example, 200 markerless virtual directories could require:

1 List request + 200 GetProperties requests

@syeleti-msft syeleti-msft added this to the v2-2.5.5 milestone Jul 24, 2026
Copilot AI review requested due to automatic review settings July 24, 2026 06:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR optimizes Block Blob virtual directory listings by eliminating per-directory REST GetProperties marker checks when a hierarchical listing is confirmed to be single-page (no incoming marker and no outgoing continuation marker). This reduces REST call volume for directories containing many markerless virtual directories while preserving the existing paginated-listing correctness logic.

Changes:

  • Detect single-page hierarchical listings via a new isSinglePageListing(marker, nextMarker) helper and pass the result into processBlobPrefixes.
  • For single-page listings, create directory attributes directly from BlobPrefix entries instead of performing per-prefix marker lookups.
  • Add unit tests for isSinglePageListing and for skipping marker lookups on single-page listings; update the changelog.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
component/azstorage/block_blob.go Adds single-page detection and uses it to skip per-prefix marker REST checks when safe.
component/azstorage/block_blob_test.go Adds unit tests for the single-page logic and for skipping marker lookups.
CHANGELOG.md Notes the REST call reduction optimization in the unreleased changelog.

Comment on lines 827 to 828
// marker file not found in current iteration, so we need to manually check attributes via REST
_, err := bb.getAttrUsingRest(*blobInfo.Name)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants