fix(storage): retry throttled fts metadata listing

[BubbleCal]

Bug Fix

What is the bug?

Distributed FTS indexing can encounter transient object-store list failures during partition metadata discovery, especially Azure ServerBusy and account egress-limit responses. These failures were not retried consistently on the Lance side, and FTS metadata listing could swallow list stream errors and later report the misleading error No partition metadata files found.

What issues or incorrect behavior does the bug cause?

Transient cloud throttling can fail FTS index builds instead of backing off and resuming. When the list failure happens while discovering partition metadata, the user can lose the original service error body, request ID, or throttle reason and see an empty-directory error instead. List streams also did not feed the AIMD limiter before the underlying object-store list request started.

How does this PR fix the problem?

• Add delayed retry with conservative jittered exponential backoff to object-store list retry streams, while preserving immediate return for non-retryable errors.
• Resume retried list streams from the last successfully yielded key.
• Acquire the AIMD list token before creating list and list_with_offset delegate streams, then continue observing yielded items and errors.
• Expand throttle classification for targeted cloud throttling messages, including Azure ServerBusy, account egress limits, HTTP 429, and known rate-limit phrases.
• Remove the provider-side client_max_retries=0 AIMD-disable guard for AWS, Azure, and GCP. Users should set lance_aimd_max_retries=0 to explicitly disable Lance AIMD throttling.
• Propagate real FTS metadata list stream errors immediately while preserving the existing no-files error for genuinely empty metadata directories.

No public Rust, Python, or Java APIs are added.

Validation

• cargo test -p lance-io list_retry
• CARGO_TARGET_DIR=/tmp/lance-fc2c-target cargo test -p lance-io throttle
• CARGO_TARGET_DIR=/tmp/lance-fc2c-target cargo test -p lance-index list_metadata_files
• CARGO_TARGET_DIR=/tmp/lance-fc2c-target cargo test -p lance merge_existing_index_segments_supports_fts_segments
• cargo fmt --all
• CARGO_TARGET_DIR=/tmp/lance-fc2c-target cargo clippy --all --tests --benches -- -D warnings

[codecov]

Codecov Report

❌ Patch coverage is 80.92910% with 78 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-index/src/scalar/inverted/builder.rs	51.56%	31 Missing ⚠️
rust/lance-io/src/object_store/list_retry.rs	87.17%	24 Missing and 1 partial ⚠️
rust/lance-io/src/object_store/throttle.rs	85.33%	22 Missing ⚠️

📢 Thoughts on this report? Let us know!

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.

_{Tip: disable this comment in your organization's Code Review settings.}

[Xuanwo]

It is possible for us to not rely listing?

[BubbleCal]

let's remove listing in next PR!