Publish mssql-py-core debug symbols to symbol servers (WI 45990)

[Saurabh Singh (saurabh500)]

Addresses WI 45990 — publishes mssql-py-core debug symbols (Windows PDB, Linux separate .debug + stripped .so, macOS .dSYM) to symbol servers so customer crash dumps can be symbolicated, while keeping shipped wheels small.

Mirrors the msodbcsql publishing pattern (msodbcsql-Symbol-Server pipeline in the sqlclientdrivers ADO org).

What runs where

Pipeline	Symbol destination	Retention
GH-Non-Official Python Wheels Publish (main / nightly, PRs skipped)	SqlClientDrivers ADO Artifacts symbol server (https://artifacts.dev.azure.com/sqlclientdrivers/_apis/symbol/symsrv)	30 days
Official Python Wheels Build (stable)	Debug artifacts published only as pipeline drops — no upload	n/a
ADO-Release Nuget Python Wheels (manual release)	1. SqlClientDrivers ADO Artifacts symbol server, 2. Microsoft Internal Symbol Server (default on), 3. Microsoft Public Symbol Server (default off)	Indefinite by default; Public always 0 per MS policy

Payload per build target

Windows x64/ARM64, Linux glibc/musl x64/ARM64, macOS universal2 — for each Python 3.10–3.14:

• Windows: mssql_py_core*.pdb
• Linux: mssql_py_core.cpython-3XX-*.so.debug (full DWARF, ~60 MB) plus the matching stripped .so that ships in the wheel — both carry the same GNU build-id, which is what the symbol server indexes on
• macOS: .dSYM bundles as pipeline artifacts only (not indexed pending Mach-O validation)

Each wheel is built into its own staging directory and the debug split runs against the .so inside that wheel, so the published .debug always belongs to exactly the binary we ship. Per-build staging is load-bearing: the manylinux and musllinux container steps of a Linux job share the same wheels output directory, so globbing it by Python tag would pick up the previous step's already-stripped wheel — yielding ~880 KB stub debug files and leaving the musllinux wheels unsplit at ~14.6 MB.

Cargo.toml release profile

Added to mssql-py-core/Cargo.toml:

[profile.release]
lto = true
debug = "full"
split-debuginfo = "packed"
strip = "none"

debug = "full" is the load-bearing setting; strip = "none" is technically redundant (Cargo already defaults strip to none when debug info is requested) but is kept explicit so the intent survives future edits. This is still a fully optimized release build that additionally carries debug metadata — not a debug build.

Linux overrides split-debuginfo to off (via CARGO_PROFILE_RELEASE_SPLIT_DEBUGINFO in the container build script) and uses the conventional objcopy --only-keep-debug / --strip-debug / --add-gnu-debuglink flow. Split-DWARF was prototyped and benchmarked in the production manylinux container and rejected on two counts: the .dwp carries no build-id at all (readelf -n is empty), so it cannot be indexed by build-id; and skeleton DWARF is not small — ~8.5 MB of .debug_* stays in the shipped .so, inflating the wheel by ~77%.

The wheel is rewritten in place by scripts/split-wheel-debuginfo.py, which copies every ZIP entry verbatim (name, permissions, timestamp, order, compression) and swaps only the .so payload and its RECORD line. This avoids wheel unpack/wheel pack, which loses Unix permission bits on older wheel versions and reconstructs the output filename from the WHEEL Tag: lines.

Validation — end-to-end against real published symbols

Validated on GH-Non-Official Python Wheels Publish build 164646 (succeeded), which includes the staging fix.

Symbol request

field	value
name	mssql_rs_ADOOnlySymbols_0.1.8-dev.20260805.164646_d3d43a43…
status	sealed
created	2026-08-05T02:19:17Z
expiration	2026-09-04T02:19:34Z
retention	30.0 days (confirms SymbolExpirationInDays is applied; the task default of ~36530 days would land in 2126)

All 20 Linux targets (glibc/musl × x64/arm64 × cp310–cp314) produced distinct build-ids and full debug files — no duplicates across libc variants, no ~880 KB stubs:

	.debug	shipped .so	wheel
manylinux x64	62,095–62,440 KB	7,822–7,837 KB	3.1 MB
musllinux x64	61,345–61,752 KB	7,816–7,840 KB	3.1 MB
manylinux arm64	63,596–64,004 KB	7,436–7,568 KB	2.9–3.0 MB
musllinux arm64	63,589–64,001 KB	7,429–7,502 KB	2.9 MB

Retrieval — all 20 build-ids resolve on the ADO symbol server, with published sizes matching the build logs:

• shipped binary: {so}/elf-buildid-{buildid}/{so} → 200
• full debug: _.debug/elf-buildid-sym-{buildid}/_.debug → 200
• alt debug: {so}.debug/elf-buildid-{buildid}/{so}.debug → 200

Deep check on musllinux x64 cp311 (c15e4899d4fb0da0453e0ebef2f3a0f0791e4036), downloaded from the symbol server:

• shipped .so — 0 .debug_* sections, .gnu_debuglink present, .symtab retained
• published .debug — same GNU build-id, 9 .debug_* sections, 621 compile units, 3.6 MB of .debug_line
• symbolication: 5,849 of 5,926 text symbols (98.7%) resolve to a source line; the 1.3% remainder are LTO-merged closures and stdlib intrinsics

mssql_py_core::connection::PyCoreConnection::extract_yes_no_bool -> /workspace/mssql-py-core/src/connection.rs:508
mssql_py_core::connection::PyCoreConnection::dict_to_client_config -> /workspace/mssql-py-core/src/connection.rs:230

Note that DWARF records container-absolute source paths such as /workspace/mssql-py-core/src/connection.rs. That is fine for the internal servers and substantiates keeping the Public symbol server stage off by default.

Known non-issue: the musllinux cp310 extension is named …-linux-gnu.so rather than …-linux-musl.so. This is upstream CPython behaviour — the cp310 interpreter in the musllinux image reports EXT_SUFFIX = .cpython-310-x86_64-linux-gnu.so (the musl triplet only landed in 3.11+), so this is exactly the name that interpreter looks for at import time. Build-ids confirm the binary is genuinely compiled separately from the glibc one.

Prerequisites (require ADO admin action — not in this PR)

• Create ADO service connection SymbolsPublishing-mssql-rs (Workload Identity Federation, patterned on SymbolsPublishing-ADO.Net in the ADO.Net project). ADO.Net and msodbcsql both have theirs; mssql-rs does not. Required only for release-time promotion to Microsoft Internal/Public — the non-official flow is validated and works today without it. Note that publishSymbolsInternal defaults to true in OfficialPythonWheelsRelease.yml, so the first official release will fail until this exists.
• Register the mssql-rs project namespace with the Microsoft Symbols Publishing service (symbolrequestprod.trafficmanager.net/projects/mssql-rs).

Follow-ups (deliberately out of scope)

• macOS .dSYM indexing — needs a validation pass for Mach-O UUID support on the Microsoft symbol services.
• Apply the same split-debuginfo treatment to mssql-js (and future BCP bindings) — WI 45990 covers those too, but the workspace-root [profile.release] still ships them stripped with no separate debug file.
• Public symbol sanitization — Rust debug="full" PDBs/DWARF contain source paths and private types. The Public stage is plumbed but defaults off; needs a stripping / source-path-rewriting flow first.
• Microsoft Internal/Public promotion is still unproven end-to-end, blocked on the service connection above.

Design bug caught & fixed

The existing OfficialPythonWheelsRelease.yml read version from the release-run checkout of Cargo.toml, not from the commit the Official Build actually ran against — so if stable moved between build and release, the NuGet version and shipped wheel version could disagree. Version is now read via git show :mssql-py-core/Cargo.toml and a wheel-filename cross-check aborts the release if they don't match.

[Saurabh Singh (saurabh500)]

See the inline comments on the diff for the actionable feedback.

[Copilot]

Pull request overview

Adds cross-platform debug-symbol capture and publishing for mssql-py-core wheels.

Changes:

• Enables full split debug information for release builds.
• Captures Windows, Linux, and macOS symbol artifacts.
• Adds ADO and Microsoft symbol-server publishing stages with retention controls.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
scripts/build-python-wheels-in-container.sh	Collects Linux .dwp and .so artifacts.
mssql-py-core/Cargo.toml	Configures release debug information.
.pipeline/templates/publish-symbols.yml	Adds reusable symbol publishing and promotion.
.pipeline/templates/build-python-wheels-template.yml	Captures platform-specific symbols.
.pipeline/OneBranch/stages.yml	Adds non-official symbol publishing.
.pipeline/OneBranch/OfficialPythonWheelsRelease.yml	Adds official symbol promotion and version validation.
.pipeline/OneBranch/NonOfficialPythonWheelsPublish.yml	Enables ADO symbol publishing by default.

[Saurabh Singh (saurabh500)]

Inline review notes are anchored on the relevant lines below.

[github-actions]

📊 Code Coverage Report

🔥 Diff Coverage 100%

🎯 Overall Coverage 90.2%

📦 Project: mssql-tds + mssql-odbc + mssql-py-core ℹ️ Note: diff coverage is reported, not enforced.

---

Diff Coverage

Diff: main...HEAD, staged and unstaged changes

No lines with coverage information in this diff.

---

🔗 Quick Links

View Azure DevOps Build · Coverage Report