Support Arrow View types in bulk copy

[gargsaumya]

Description

Adds native Arrow Utf8View and BinaryView support to mssql-py-core bulk copy. String views now map to SQL character, GUID, XML, and JSON destinations, while binary views map to binary destinations without requiring producers such as Polars to re-encode their buffers.

Includes Rust unit coverage, live PyArrow C-stream integration tests, and an unreleased changelog entry. The existing mssql-py-core version remains 0.1.9, which is reserved on main and is not yet published to the public wheel feed.

Related Issues

microsoft/mssql-python#708
https://sqlclientdrivers.visualstudio.com/mssql-python/_workitems/edit/47279

Validation

• cargo nextest run --lib arrow_bulkcopy::tests (53 passed)
• cargo clippy --frozen --all-features --all-targets -- -D warnings
• cargo fmt -- --check
• Live SQL Server tests for string_view and binary_view (2 passed)
• Live Polars 1.39.3 reproduction through mssql-python

Checklist

• cargo bfmt passes for the modified crate
• cargo bclippy passes for the modified crate
• cargo btest passes for the full workspace
• New/changed functionality has tests
• Public behavior changes are documented

[Copilot]

Pull request overview

Adds Arrow view-type support to mssql-py-core bulk copy, resolving Polars compatibility issue #708.

Changes:

• Maps Utf8View and BinaryView to compatible SQL types.
• Adds Rust unit and live integration coverage.
• Documents the change.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
mssql-py-core/src/arrow_bulkcopy.rs	Implements view-type planning and extraction.
mssql-py-core/tests/test_bulkcopy_arrow_varchar.py	Tests string_view round-tripping.
mssql-py-core/tests/test_bulkcopy_arrow_binary.py	Tests binary_view round-tripping.
CHANGELOG.md	Adds an unreleased changelog entry.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

[David Engel (David-Engel)]

Summary

Adds Arrow Utf8View/BinaryView support to mssql-py-core's Arrow bulk copy so Polars-exported C-stream columns load without a PyArrow re-encode. The change is small, correct, and well covered. No blocking findings — a couple of suggestions and nits inline.

Verified locally (worktree at 496d0c8e)

• cargo fmt -- --check — clean
• cargo clippy --all-features --all-targets -- -D warnings (mssql-py-core) — clean
• cargo nextest run --lib arrow_bulkcopy — 53 passed
• CI build 166711 Linux job: both new integration tests PASSED (test_cursor_bulkcopy_arrow_varbinary_binary_view, test_cursor_bulkcopy_arrow_varchar_string_view) — confirmed not skipped
• arrow 55.2 with features = ["ffi"] does support view types over the C data interface, and CI installs unpinned pyarrow, so pa.string_view() / pa.binary_view() are available

Correctness

The new resolve_kind arms cover NVarChar/NChar/NText, VarChar/Char/Text, VarBinary/Binary/Image, plus Xml/Json/UniqueIdentifier for Utf8View — mirroring the existing Utf8/LargeUtf8/Binary arms with no gaps I could find. NULL handling sits upstream of the match in extract_value, so view arrays get it for free, and the tests exercise both the inline (<=12 byte) and out-of-line buffer paths on each new array type.

CI

Both red checks trace to one infra flake, not this change:

• Build Stage / Build Linux ARM failed at Wait for cross-pool SQL host endpoint — "No SQL host endpoint after 10 min on stage attempt 3."
• coverage-report then timed out after 75 minutes waiting for the combined coverage artifact that the failed stage never produced.

The pipeline's own message applies here: use Rerun stage, not Rerun failed jobs — the latter won't restart the SQL host job and will time out again.

[github-actions]

📊 Code Coverage Report

🔥 Diff Coverage 89%

🎯 Overall Coverage 91.5%

📦 Project: mssql-tds + mssql-odbc + mssql-py-core ℹ️ Note: diff coverage is reported, not enforced.

---

Diff Coverage

Diff: main...HEAD, staged and unstaged changes

• mssql-py-core/src/arrow_bulkcopy.rs (89.4%): Missing lines 683-686,1195,1199,1226

Summary

• Total: 66 lines
• Missing: 7 lines
• Coverage: 89%

mssql-py-core/src/arrow_bulkcopy.rs

  679         Ok(a.value(row_idx))
  680     } else if let Some(a) = arr.as_any().downcast_ref::<StringViewArray>() {
  681         Ok(a.value(row_idx))
  682     } else {
! 683         Err(Error::UsageError(format!(
! 684             "Arrow array downcast failed: expected a UTF-8 string array (utf8/large_utf8/utf8_view), got {:?}",
! 685             arr.data_type()
! 686         )))
  687     }
  688 }
  689 
  690 /// Read a timestamp value as 100-ns ticks since UNIX epoch.

---

  1191             let dest = meta("name", sql_type, true);
  1192             let plan = one_col_plan(DataType::Utf8View, &dest);
  1193             match plan.extract_value(array.as_ref(), 0, &dest).unwrap() {
  1194                 ColumnValues::String(value) => assert_eq!(value.to_utf8_string(), "inline"),
! 1195                 other => panic!("expected String, got {other:?}"),
  1196             }
  1197             match plan.extract_value(array.as_ref(), 1, &dest).unwrap() {
  1198                 ColumnValues::String(value) => assert_eq!(value.to_utf8_string(), external),
! 1199                 other => panic!("expected String, got {other:?}"),
  1200             }
  1201             assert_eq!(
  1202                 plan.extract_value(array.as_ref(), 2, &dest).unwrap(),
  1203                 ColumnValues::Null

---

  1222                 .extract_value(array.as_ref(), 0, &dest)
  1223                 .unwrap()
  1224             {
  1225                 ColumnValues::Xml(_) | ColumnValues::Json(_) => {}
! 1226                 other => panic!("expected Xml/Json, got {other:?}"),
  1227             }
  1228         }
  1229     }

---

---

🔗 Quick Links

View Azure DevOps Build · Coverage Report

[Saurabh Singh (saurabh500)]

Adds native Arrow Utf8View and BinaryView support to Python bulk copy so Polars C-stream inputs no longer need a PyArrow table conversion. The implementation is correct and well scoped: destination mappings mirror the existing UTF-8 and binary paths, null handling is preserved, and tests cover inline and external view storage.

Verdict: Approve with comments. No blocking findings. One non-blocking unit-test assertion should be tightened; see the inline suggestion.

[Saurabh Singh (saurabh500)]

This accepts either Xml or Json for both iterations, so the test still passes if the planner swaps those mappings, and it never verifies the payload. Please match the destination and returned variant together and assert the content:

Suggested change

	match one_col_plan(DataType::Utf8View, &dest)
	.extract_value(array.as_ref(), 0, &dest)
	.unwrap()
	{
	ColumnValues::Xml(_) | ColumnValues::Json(_) => {}
	other => panic!("expected Xml/Json, got {other:?}"),
	}
	let value = one_col_plan(DataType::Utf8View, &dest)
	.extract_value(array.as_ref(), 0, &dest)
	.unwrap();
	match (sql_type, value) {
	(SqlDbType::Xml, ColumnValues::Xml(value)) => {
	assert_eq!(value.as_string(), text)
	}
	(SqlDbType::Json, ColumnValues::Json(value)) => {
	assert_eq!(value.bytes, text.as_bytes())
	}
	(_, other) => panic!("expected {sql_type:?}, got {other:?}"),
	}

[Saurabh Singh (saurabh500)]

One minor test change, ready to approve after that is incorporated