refactor(index)!: move distributed BTree build to segmented index framework

[zhangyue19921010]

Overview

This is a refactor PR that wires the BTree scalar index's distributed build flow into
Lance's existing segmented-index framework, and retires the legacy BTree range_id sharding +
merge_index_metadata path.

The new distributed build flow: each worker builds one canonical segment for its assigned
fragment_ids without committing it, and the driver publishes them all at once via
commit_existing_index_segments(...). Passing fragment_ids is what triggers the skip-commit
segment build; the per-language entry points are:

• Python: create_index_uncommitted(column, "BTREE", fragment_ids=[...]) — returns the uncommitted
  segment Index. (create_scalar_index(..., "BTREE", fragment_ids=...) is no longer the distributed entry — see below.)
• Java: createIndex(IndexOptions.builder(...).withFragmentIds([...]).build()) — returns the
  segment Index.
• Rust: CreateIndexBuilder::new(...).fragments(vec![...]).execute_uncommitted().

The old "distributed write of part_* shards + a final merge_index_metadata" path is gone.

This PR does not include the segmented planning / k-way merge / optimize-append
implementations (plan_segments / build_segment / merge_segments, the BTree branches of
IndexSegmentBuilder / merge_existing_index_segments, etc.); those land in a follow-up PR2. In
PR1 the per-fragment segments are committed directly via commit_existing_index_segments(...),
with no build_all / segment-builder step (so IndexSegmentBuilder.with_index_type("BTREE") is
intentionally not wired yet).

Compatibility Impact (Review Focus)

• API soft-break: merge_index_metadata(BTree)
Dataset::merge_index_metadata(..., IndexType::BTree) (and Java Dataset.mergeIndexMetadata(BTREE),
  Python merge_index_metadata(index_type="BTREE")) now returns an error directly, with
  migration guidance. Callers that previously relied on "distributed write of part_* +
  merge_index_metadata to commit" are broken explicitly (rather than silently producing wrong
  results).
  Migration: build one segment per fragment group with create_index_uncommitted(..., "BTREE", fragment_ids=...) and publish them with commit_existing_index_segments(...); to consolidate
  multiple segments into fewer/one, use merge_existing_index_segments(...) (BTree support comes in
  PR2).

• Distributed BTree builds move to create_index_uncommitted (Python)
  Distributed BTree segments are now built through create_index_uncommitted(column, "BTREE", fragment_ids=[...]). create_scalar_index(..., "BTREE", fragment_ids=...) (and the IndexConfig equivalent) is now rejected with an error pointing to
  create_index_uncommitted. Non-distributed create_scalar_index("BTREE") (no fragment_ids) is
  unchanged and still commits a single canonical index. Java has no separate scalar entry, so its
  distributed entry stays createIndex(...withFragmentIds...).

• range_id deprecated
  BTree range_id (Rust BTreeParameters::range_id, Java BTreeIndexParams.rangeId) is
  deprecated: if Some(..) is detected at build time, a warn! migration hint is logged and the
  value is then ignored, still producing a canonical segment (lossless, no error). The field is
  retained (not physically removed) so that stale inputs are flagged loudly rather than silently
  dropped by serde; it will be removed in a future release.
  A pre-sorted data stream is still supported via Rust preprocessed_data / Java
  withPreprocessedData (the Python API does not take a reader). Pass it together with fragment_ids
  so the resulting segment is uncommitted and scoped to those fragments.

• Behavior change: distributed semantics change
  In the segmented case, distributed builds move from "a single index written in a distributed
  fashion, then a final merge" to "creating multiple independent segments in a distributed
  fashion" — each worker produces one canonical segment with exactly one page_lookup.lance +
  one page_data.lance per segment, and no more internal part_* shards. At query time the
  segments of one logical index are unioned automatically.

• Historical compatibility: old layouts remain readable and remappable
  The read and compaction remap paths for already-persisted range / part_* legacy index
  layouts are fully preserved, to maintain backward compatibility.

Example

import lance

ds = lance.dataset(uri)

# 1. Build one uncommitted BTree segment per fragment (fragment_ids => skip commit;
#    exactly one page_lookup.lance + one page_data.lance per segment, no part_* shards).
segments = [
    ds.create_index_uncommitted(
        column="id",
        index_type="BTREE",
        name="id_btree",
        fragment_ids=[frag.fragment_id],
    )
    for frag in ds.get_fragments()
]

# 2. Commit all segments as one logical index.
ds = ds.commit_existing_index_segments("id_btree", "id", segments)

# 3. Query (segments are unioned automatically).
ds.scanner(filter="id = 100").to_table()

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[codecov]

Codecov Report

❌ Patch coverage is 98.18182% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/index/create.rs	98.10%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!