From 263c6560061d232698b06502d85adc9b846281f8 Mon Sep 17 00:00:00 2001 From: Saurabh Singh <1623701+saurabh500@users.noreply.github.com> Date: Sat, 8 Aug 2026 12:14:28 -0700 Subject: [PATCH 1/2] Add fetch-throughput A/B ODBC benchmark harness (mssql-odbc vs msodbcsql18) Add a standalone C++ benchmark that times the SQLFetch + SQLGetData drain over a large result set and runs an A/B between the native ODBC Driver 18 for SQL Server and the Rust mssql-odbc dev driver in one invocation. It reuses the e2e harness config/diagnostics plumbing but is a plain executable (not a CTest test). Each leg loads its driver DLL directly (its own tiny driver manager) so the unregistered dev driver runs with no admin/registry and both legs share an identical, DM-free code path. Reports raw per-rep ms/rows-per-sec, per-driver medians, and the rust/native ratio; a checksum over decoded values guards against elision and confirms both legs did equal work. run_bench.ps1 builds the Rust driver + benchmark and runs both legs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a3c262b9-1962-4d93-b08e-e3bc84aa4ac0 --- mssql-odbc/tests/e2e/CMakeLists.txt | 10 + mssql-odbc/tests/e2e/README.md | 68 +++ mssql-odbc/tests/e2e/bench/fetch_bench.cpp | 516 +++++++++++++++++++++ mssql-odbc/tests/e2e/run_bench.ps1 | 170 +++++++ 4 files changed, 764 insertions(+) create mode 100644 mssql-odbc/tests/e2e/bench/fetch_bench.cpp create mode 100644 mssql-odbc/tests/e2e/run_bench.ps1 diff --git a/mssql-odbc/tests/e2e/CMakeLists.txt b/mssql-odbc/tests/e2e/CMakeLists.txt index e8ee5379..0d24e539 100644 --- a/mssql-odbc/tests/e2e/CMakeLists.txt +++ b/mssql-odbc/tests/e2e/CMakeLists.txt @@ -125,3 +125,13 @@ add_odbc_test(more_results_test tests/more_results_test.cpp) add_odbc_test(execute_test tests/execute_test.cpp) add_odbc_test(get_type_info_test tests/get_type_info_test.cpp) add_odbc_test(row_count_test tests/row_count_test.cpp) + +# --------------------------------------------------------------------------- +# Benchmarks (not registered with CTest — perf tools, not pass/fail tests) +# --------------------------------------------------------------------------- +# fetch_bench times the SQLFetch drain over a large result set for one or more +# ODBC drivers selected by name and prints an A/B comparison. It reuses +# odbc_test_lib for connection config + diagnostics but is a plain executable so +# a normal `ctest` run does not invoke it. Drive it via run_bench.ps1. +add_executable(fetch_bench bench/fetch_bench.cpp) +target_link_libraries(fetch_bench PRIVATE odbc_test_lib) diff --git a/mssql-odbc/tests/e2e/README.md b/mssql-odbc/tests/e2e/README.md index a6d15f83..572e6a7e 100644 --- a/mssql-odbc/tests/e2e/README.md +++ b/mssql-odbc/tests/e2e/README.md @@ -35,6 +35,9 @@ tests/e2e/ ├── run_e2e.sh # Build + test runner (Linux / macOS) ├── build_e2e.sh # Build-only half for CI artifact reuse (Linux / macOS) ├── run_e2e.ps1 # Build + test runner (Windows, requires admin) +├── run_bench.ps1 # Build + run the fetch-throughput A/B benchmark (Windows, no admin) +├── bench/ +│ └── fetch_bench.cpp # Fetch/row-throughput A/B benchmark (mssql-odbc vs msodbcsql18) └── README.md # This file ``` @@ -311,3 +314,68 @@ across matching distros: A glibc-2.35 binary may fail to load on older glibc (e.g. RHEL 8's 2.28), and an OpenSSL 3 binary won't find `libssl.so.1.1`, which is why the glibc-2.28 track exists as a separate build. + +## Fetch-throughput A/B benchmark + +`bench/fetch_bench.cpp` is a standalone perf harness (not a pass/fail CTest +test) that times the **fetch / row-decode path** — the `SQLFetch` + +`SQLGetData` drain over a large result set — and runs an A/B between the native +**ODBC Driver 18 for SQL Server** and the Rust `mssql-odbc` dev driver in one +invocation. It deliberately does **not** benchmark `SQLExecDirect` (that path is +I/O bound; see PR #186); the query returns many rows so row decode dominates. + +### What it measures + +- The query is `SELECT TOP (N)` of an `int` / `bigint` / `varchar` / `nvarchar` + / `float` row set materialised entirely server-side (a `sys.all_objects` + self cross-join), so the client spends its time decoding rows. +- Every column is pulled with `SQLGetData` as character data — the one column + path both drivers share (the Rust dev driver does not export `SQLBindCol`, and + its Phase-1 `SQLGetData` converts to `SQL_C_CHAR` / `SQL_C_WCHAR` only, so + `datetime2` is intentionally excluded). +- Timing uses `QueryPerformanceCounter` around each execute+drain rep. The first + rep per driver is a discarded warmup; every remaining rep is printed (ms and + rows/sec) along with the median. A checksum over all decoded values is summed + so nothing is optimised away — it matches across both legs, confirming an + identical, fair workload. + +### Driver-manager bypass (no admin, no registry) + +Each leg loads a driver **DLL directly** (`LoadLibrary` + `GetProcAddress`, +acting as its own tiny driver manager) via the `dll:` leg syntax, so the +unregistered Rust dev driver runs with **zero admin rights** and both legs +execute on an identical, DM-free code path. A leg given a bare name instead of +`dll:` routes through `odbc32.dll` (the real DM) and resolves that name from the +registry as usual. + +### Running it (Windows) + +```powershell +# From mssql-odbc\tests\e2e\ — builds the Rust driver + fetch_bench, then runs +# both legs. No administrator required. +.\run_bench.ps1 + +# Tune the workload +.\run_bench.ps1 -Rows 500000 -Reps 11 -Warmup 1 + +# Point at specific driver DLLs (defaults: system32 msodbcsql18.dll for native, +# target\debug\msodbcsql18.dll for Rust) +.\run_bench.ps1 -NativeDll 'C:\Windows\System32\msodbcsql18.dll' -RustDll '...\target\debug\msodbcsql18.dll' +``` + +Connection info is resolved the same way as the e2e tests (`ODBC_TEST_*` env +vars, else `mssql-tds/.env` for host/port/user and `$env:SQL_PASSWORD` / +`C:\tmp\password` for the password — never echoed). + +The binary can also be driven directly once built: + +```powershell +build\fetch_bench.exe --rows 200000 --reps 9 --warmup 1 ` + --driver "native=dll:C:\Windows\System32\msodbcsql18.dll" ` + --driver "rust=dll:...\target\debug\msodbcsql18.dll" +``` + +The final line prints `ratio (rust median / native median)` — `1.00x` is +parity, `>1` means Rust is slower. Per-driver deltas under ~15% are within +run-to-run noise. + diff --git a/mssql-odbc/tests/e2e/bench/fetch_bench.cpp b/mssql-odbc/tests/e2e/bench/fetch_bench.cpp new file mode 100644 index 00000000..e243d1b8 --- /dev/null +++ b/mssql-odbc/tests/e2e/bench/fetch_bench.cpp @@ -0,0 +1,516 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. +// +// fetch_bench.cpp – Fetch/row-throughput A/B benchmark for ODBC drivers. +// +// Times the SQLFetch drain over a large, wide result set for one or more ODBC +// drivers selected by name, then prints raw per-rep numbers and an A/B ratio. +// This benchmarks the FETCH / decode path ONLY (execute+fetch drain) — it is +// deliberately NOT an ExecDirect micro-benchmark (that path is I/O bound, see +// PR #186). The query returns many rows so row decode dominates the timing. +// +// Reuses the e2e harness plumbing (ODBCTestConfig for server/uid/pwd/database, +// ODBCTestUtils for diagnostics) but builds its own per-driver connection +// string so the SAME binary can drive both the native "ODBC Driver 18 for SQL +// Server" and the Rust mssql-odbc dev driver back-to-back in one invocation. +// +// Usage: +// fetch_bench [--rows N] [--reps R] [--warmup W] +// [--driver LABEL=DRIVER_NAME | LABEL=dll:PATH] ... +// +// --rows N rows the query returns (default 200000) +// --reps R timed reps per driver, incl warmup (default 9) +// --warmup W leading reps discarded per driver (default 1) +// --driver LABEL=TARGET register a driver leg; may be repeated. TARGET is +// either a DM-registered driver name (routed via odbc32.dll) or +// `dll:` to load an unregistered driver DLL directly (no admin, +// no registry). If omitted, defaults to two legs: +// native = env ODBC_BENCH_NATIVE_DLL (direct) or +// ODBC_BENCH_NATIVE_DRIVER (default "ODBC Driver 18 for SQL Server") +// rust = env ODBC_BENCH_RUST_DLL (direct) or +// ODBC_BENCH_RUST_DRIVER (default "mssql-odbc") + +#include "odbc_test_fixture.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#ifdef _WIN32 +#include +#else +#include +#endif + +namespace { + +// --------------------------------------------------------------------------- +// High-resolution wall-clock timer. +// Windows: QueryPerformanceCounter (per the benchmark spec). +// POSIX: steady_clock, so the harness still builds/runs cross-platform. +// --------------------------------------------------------------------------- +class Stopwatch { +public: + void Start() { start_ = Now(); } + double ElapsedMs() const { return (Now() - start_) * 1000.0; } + +private: +#ifdef _WIN32 + static double Now() { + LARGE_INTEGER c, f; + QueryPerformanceCounter(&c); + QueryPerformanceFrequency(&f); + return static_cast(c.QuadPart) / static_cast(f.QuadPart); + } +#else + static double Now() { + return std::chrono::duration_cast>( + std::chrono::steady_clock::now().time_since_epoch()) + .count(); + } +#endif + double start_ = 0.0; +}; + +struct DriverLeg { + std::string label; + std::string target; // registered driver name (DM leg) or DLL path (direct leg) + bool direct; // true: LoadLibrary(target); false: route via odbc32 DM +}; + +struct RepResult { + double ms = 0.0; + long long rows = 0; +}; + +// --------------------------------------------------------------------------- +// Direct-load ODBC entry points (bypass the Driver Manager). +// +// The Windows DM only loads drivers registered in HKLM\...\ODBCINST.INI, which +// requires admin. To exercise an *unregistered* dev driver — and to keep both +// A/B legs on an identical, DM-free code path — every leg resolves the handful +// of ODBC entry points it needs from an explicit module: +// * a `dll:` leg loads the driver DLL itself and calls it directly; +// * a registered-name leg loads odbc32.dll (the DM), which then resolves the +// driver from the registry as usual. +// Wide (…W) entry points are used throughout to match the UNICODE harness build. +// --------------------------------------------------------------------------- +using Fn_AllocHandle = SQLRETURN(SQL_API*)(SQLSMALLINT, SQLHANDLE, SQLHANDLE*); +using Fn_SetEnvAttr = SQLRETURN(SQL_API*)(SQLHENV, SQLINTEGER, SQLPOINTER, SQLINTEGER); +using Fn_DriverConnect = SQLRETURN(SQL_API*)(SQLHDBC, SQLHWND, SQLWCHAR*, SQLSMALLINT, + SQLWCHAR*, SQLSMALLINT, SQLSMALLINT*, + SQLUSMALLINT); +using Fn_ExecDirect = SQLRETURN(SQL_API*)(SQLHSTMT, SQLWCHAR*, SQLINTEGER); +using Fn_GetData = SQLRETURN(SQL_API*)(SQLHSTMT, SQLUSMALLINT, SQLSMALLINT, SQLPOINTER, + SQLLEN, SQLLEN*); +using Fn_Fetch = SQLRETURN(SQL_API*)(SQLHSTMT); +using Fn_FreeStmt = SQLRETURN(SQL_API*)(SQLHSTMT, SQLUSMALLINT); +using Fn_Disconnect = SQLRETURN(SQL_API*)(SQLHDBC); +using Fn_FreeHandle = SQLRETURN(SQL_API*)(SQLSMALLINT, SQLHANDLE); +using Fn_GetDiagRec = SQLRETURN(SQL_API*)(SQLSMALLINT, SQLHANDLE, SQLSMALLINT, SQLWCHAR*, + SQLINTEGER*, SQLWCHAR*, SQLSMALLINT, + SQLSMALLINT*); + +struct OdbcApi { +#ifdef _WIN32 + HMODULE mod = nullptr; +#else + void* mod = nullptr; +#endif + Fn_AllocHandle AllocHandle = nullptr; + Fn_SetEnvAttr SetEnvAttr = nullptr; + Fn_DriverConnect DriverConnect = nullptr; + Fn_ExecDirect ExecDirect = nullptr; + Fn_GetData GetData = nullptr; + Fn_Fetch Fetch = nullptr; + Fn_FreeStmt FreeStmt = nullptr; + Fn_Disconnect Disconnect = nullptr; + Fn_FreeHandle FreeHandle = nullptr; + Fn_GetDiagRec GetDiagRec = nullptr; + + bool Complete() const { + return AllocHandle && SetEnvAttr && DriverConnect && ExecDirect && GetData && + Fetch && FreeStmt && Disconnect && FreeHandle && GetDiagRec; + } +}; + +void* ResolveSym(const OdbcApi& a, const char* name) { +#ifdef _WIN32 + return reinterpret_cast(GetProcAddress(a.mod, name)); +#else + return dlsym(a.mod, name); +#endif +} + +// Load |modulePath| and bind the ODBC entry points. Returns false (with a +// message on stderr) if the module or any required symbol is missing. +bool LoadOdbcApi(OdbcApi& a, const std::string& modulePath) { +#ifdef _WIN32 + SqlTString w = ODBCTestUtils::ToSqlTStr(modulePath); + a.mod = LoadLibraryW(reinterpret_cast(w.c_str())); +#else + a.mod = dlopen(modulePath.c_str(), RTLD_NOW | RTLD_LOCAL); +#endif + if (!a.mod) { + std::cerr << "ERROR: cannot load module '" << modulePath << "'\n"; + return false; + } + a.AllocHandle = reinterpret_cast(ResolveSym(a, "SQLAllocHandle")); + a.SetEnvAttr = reinterpret_cast(ResolveSym(a, "SQLSetEnvAttr")); + a.DriverConnect = + reinterpret_cast(ResolveSym(a, "SQLDriverConnectW")); + a.ExecDirect = reinterpret_cast(ResolveSym(a, "SQLExecDirectW")); + a.GetData = reinterpret_cast(ResolveSym(a, "SQLGetData")); + a.Fetch = reinterpret_cast(ResolveSym(a, "SQLFetch")); + a.FreeStmt = reinterpret_cast(ResolveSym(a, "SQLFreeStmt")); + a.Disconnect = reinterpret_cast(ResolveSym(a, "SQLDisconnect")); + a.FreeHandle = reinterpret_cast(ResolveSym(a, "SQLFreeHandle")); + a.GetDiagRec = reinterpret_cast(ResolveSym(a, "SQLGetDiagRecW")); + if (!a.Complete()) { + std::cerr << "ERROR: module '" << modulePath + << "' is missing required ODBC entry points\n"; + return false; + } + return true; +} + +// Walk the diagnostic records for |handle| and join them into one string. +std::string DiagMessage(const OdbcApi& a, SQLSMALLINT ht, SQLHANDLE h) { + std::string out; + for (SQLSMALLINT rec = 1; rec <= 8; ++rec) { + SQLWCHAR state[6] = {}; + SQLWCHAR msg[1024] = {}; + SQLINTEGER native = 0; + SQLSMALLINT len = 0; + SQLRETURN rc = a.GetDiagRec(ht, h, rec, state, &native, msg, + static_cast(1024), &len); + if (rc != SQL_SUCCESS && rc != SQL_SUCCESS_WITH_INFO) break; + SqlTString ss(reinterpret_cast(state)); + SqlTString ms(reinterpret_cast(msg)); + if (!out.empty()) out += " | "; + out += "[" + ODBCTestUtils::ToNarrow(ss) + "] " + ODBCTestUtils::ToNarrow(ms); + } + return out.empty() ? "(no diagnostic)" : out; +} + +// Build a DSN-less connection string, pulling server/uid/pwd/etc from the shared +// e2e config so both legs hit the same live server. Direct-load legs omit the +// Driver= keyword (the driver DLL is already selected by module path). +SqlTString BuildConnStr(bool includeDriver, const std::string& driver) { + auto& cfg = ODBCTestConfig::Instance(); + std::ostringstream cs; + if (includeDriver) { + cs << "Driver={" << driver << "};"; + } + cs << "Server=" << cfg.Server() << ";"; + cs << "Database=" << cfg.Database() << ";"; + cs << "TrustServerCertificate=" << cfg.TrustCert() << ";"; + if (!cfg.Encrypt().empty()) { + cs << "Encrypt=" << cfg.Encrypt() << ";"; + } + if (cfg.HasCredentials()) { + // Brace uid/pwd so values with ODBC connection-string metacharacters + // (spaces, ';', '=') survive DM parsing. + cs << "Uid={" << cfg.Uid() << "};"; + cs << "Pwd={" << cfg.Pwd() << "};"; + } else { + cs << "Trusted_Connection=Yes;"; + } + return ODBCTestUtils::ToSqlTStr(cs.str()); +} + +// A wide, representative row set: int / bigint / varchar / nvarchar / float. +// TOP (N) over a self cross-join of sys.all_objects generates N rows entirely +// server-side, so the client spends its time decoding rows (the path we time). +// Every column is retrieved as character data (SQLGetData -> SQL_C_CHAR/WCHAR), +// the one column-retrieval path both drivers share. datetime2 is intentionally +// avoided: the Rust dev driver's Phase-1 SQLGetData cannot yet convert it to +// text, so including it would make the A/B legs diverge. +std::string BuildQuery(long long rows) { + std::ostringstream q; + q << "SELECT TOP (" << rows << ") " + "CAST(ROW_NUMBER() OVER (ORDER BY (SELECT 1)) AS int) AS n, " + "CAST(CAST(ROW_NUMBER() OVER (ORDER BY (SELECT 1)) AS bigint) * 2654435761 AS bigint) AS big, " + "CONVERT(varchar(50), 'row-' + CAST(ROW_NUMBER() OVER (ORDER BY (SELECT 1)) AS varchar(20))) AS vc, " + "CONVERT(nvarchar(50), N'row-' + CAST(ROW_NUMBER() OVER (ORDER BY (SELECT 1)) AS nvarchar(20))) AS nvc, " + "CAST(ROW_NUMBER() OVER (ORDER BY (SELECT 1)) * 1.5 AS float) AS f " + "FROM sys.all_objects a CROSS JOIN sys.all_objects b"; + return q.str(); +} + +[[noreturn]] void Fail(const OdbcApi& a, const std::string& what, SQLSMALLINT ht, + SQLHANDLE h) { + std::cerr << "ERROR: " << what << " -> " << DiagMessage(a, ht, h) << "\n"; + std::exit(2); +} + +// Execute the query and drain every row/column, returning rows fetched. +// |checksum| is accumulated across reps so the compiler/driver can't elide the +// decode work. Timing is the caller's responsibility (wraps this call). +// Execute the query and drain every row/column via SQLFetch + SQLGetData, +// returning rows fetched. SQLGetData (rather than bound columns) is used so the +// same path exercises both drivers — the Rust dev driver retrieves columns via +// SQLGetData and does not export SQLBindCol. |checksum| is accumulated across +// reps so the compiler/driver can't elide the decode work. Timing is the +// caller's responsibility (wraps this call). +long long ExecuteAndDrain(const OdbcApi& a, SQLHSTMT stmt, const std::string& sql, + uint64_t& checksum) { + SqlTString tsql = ODBCTestUtils::ToSqlTStr(sql); + if (!SQL_SUCCEEDED(a.ExecDirect( + stmt, reinterpret_cast(const_cast(tsql.c_str())), + SQL_NTS))) + Fail(a, "SQLExecDirect", SQL_HANDLE_STMT, stmt); + + long long rows = 0; + uint64_t sum = checksum; + for (;;) { + SQLRETURN rc = a.Fetch(stmt); + if (rc == SQL_NO_DATA) break; + if (!SQL_SUCCEEDED(rc)) Fail(a, "SQLFetch", SQL_HANDLE_STMT, stmt); + + SQLCHAR n[32] = {}; + SQLCHAR big[32] = {}; + SQLCHAR vc[128] = {}; + SQLWCHAR nvc[128] = {}; + SQLCHAR f[64] = {}; + SQLLEN nInd = 0, bigInd = 0, vcInd = 0, nvcInd = 0, fInd = 0; + + if (!SQL_SUCCEEDED(a.GetData(stmt, 1, SQL_C_CHAR, n, sizeof(n), &nInd))) + Fail(a, "SQLGetData(n)", SQL_HANDLE_STMT, stmt); + if (!SQL_SUCCEEDED(a.GetData(stmt, 2, SQL_C_CHAR, big, sizeof(big), &bigInd))) + Fail(a, "SQLGetData(big)", SQL_HANDLE_STMT, stmt); + if (!SQL_SUCCEEDED(a.GetData(stmt, 3, SQL_C_CHAR, vc, sizeof(vc), &vcInd))) + Fail(a, "SQLGetData(vc)", SQL_HANDLE_STMT, stmt); + if (!SQL_SUCCEEDED(a.GetData(stmt, 4, SQL_C_WCHAR, nvc, sizeof(nvc), &nvcInd))) + Fail(a, "SQLGetData(nvc)", SQL_HANDLE_STMT, stmt); + if (!SQL_SUCCEEDED(a.GetData(stmt, 5, SQL_C_CHAR, f, sizeof(f), &fInd))) + Fail(a, "SQLGetData(f)", SQL_HANDLE_STMT, stmt); + + ++rows; + sum += (nInd == SQL_NULL_DATA) ? 0u : static_cast(n[0]); + sum += (bigInd == SQL_NULL_DATA) ? 0u : static_cast(big[0]); + sum += (vcInd == SQL_NULL_DATA) ? 0u : static_cast(vc[0]); + sum += (nvcInd == SQL_NULL_DATA) ? 0u : static_cast(nvc[0]); + sum += (fInd == SQL_NULL_DATA) ? 0u : static_cast(f[0]); + } + checksum = sum; + + a.FreeStmt(stmt, SQL_CLOSE); + return rows; +} + +double Median(std::vector v) { + if (v.empty()) return 0.0; + std::sort(v.begin(), v.end()); + size_t m = v.size() / 2; + return (v.size() % 2) ? v[m] : (v[m - 1] + v[m]) / 2.0; +} + +// Run warmup+timed reps for one driver leg; prints each retained rep. +std::vector RunLeg(const DriverLeg& leg, const std::string& sql, + int reps, int warmup) { + std::string module = leg.direct ? leg.target : std::string("odbc32.dll"); + OdbcApi api; + if (!LoadOdbcApi(api, module)) { + std::cerr << "ERROR: leg '" << leg.label << "' could not load '" << module + << "'\n"; + std::exit(2); + } + + SQLHENV env = SQL_NULL_HENV; + SQLHDBC dbc = SQL_NULL_HDBC; + + if (!SQL_SUCCEEDED(api.AllocHandle(SQL_HANDLE_ENV, SQL_NULL_HANDLE, &env))) + Fail(api, "SQLAllocHandle(ENV)", SQL_HANDLE_ENV, env); + api.SetEnvAttr(env, SQL_ATTR_ODBC_VERSION, + reinterpret_cast(SQL_OV_ODBC3_80), 0); + if (!SQL_SUCCEEDED(api.AllocHandle(SQL_HANDLE_DBC, env, &dbc))) + Fail(api, "SQLAllocHandle(DBC)", SQL_HANDLE_ENV, env); + + SqlTString connstr = BuildConnStr(!leg.direct, leg.target); + SQLWCHAR outStr[1024] = {}; + SQLSMALLINT outLen = 0; + SQLRETURN rc = api.DriverConnect( + dbc, nullptr, + reinterpret_cast(const_cast(connstr.c_str())), + static_cast(connstr.size()), outStr, + static_cast(sizeof(outStr) / sizeof(SQLWCHAR)), &outLen, + SQL_DRIVER_NOPROMPT); + if (!SQL_SUCCEEDED(rc)) + Fail(api, "SQLDriverConnect [" + leg.label + " / " + leg.target + "]", + SQL_HANDLE_DBC, dbc); + + SQLHSTMT stmt = SQL_NULL_HSTMT; + if (!SQL_SUCCEEDED(api.AllocHandle(SQL_HANDLE_STMT, dbc, &stmt))) + Fail(api, "SQLAllocHandle(STMT)", SQL_HANDLE_DBC, dbc); + + std::cout << "\n=== Leg: " << leg.label << " (" + << (leg.direct ? "direct-load " : "DM driver=") << "\"" << leg.target + << "\") ===\n"; + + std::vector kept; + uint64_t checksum = 0; + for (int i = 0; i < reps; ++i) { + Stopwatch sw; + sw.Start(); + long long rows = ExecuteAndDrain(api, stmt, sql, checksum); + double ms = sw.ElapsedMs(); + bool isWarmup = i < warmup; + double rps = (ms > 0.0) ? (rows / (ms / 1000.0)) : 0.0; + std::cout << " rep " << std::setw(2) << (i + 1) + << (isWarmup ? " [warmup]" : " ") << " " + << std::fixed << std::setprecision(2) << std::setw(10) << ms + << " ms " << std::setw(12) + << static_cast(rps) << " rows/s (" << rows + << " rows)\n"; + if (!isWarmup) kept.push_back({ms, rows}); + } + std::cout << " checksum=" << checksum << "\n"; + + api.FreeHandle(SQL_HANDLE_STMT, stmt); + api.Disconnect(dbc); + api.FreeHandle(SQL_HANDLE_DBC, dbc); + api.FreeHandle(SQL_HANDLE_ENV, env); + return kept; +} + +std::string GetEnvOr(const char* name, const char* fallback) { +#ifdef _WIN32 + char* buf = nullptr; + size_t len = 0; + if (_dupenv_s(&buf, &len, name) == 0 && buf) { + std::string v(buf); + free(buf); + return v; + } + return fallback; +#else + const char* v = std::getenv(name); + return (v && v[0]) ? std::string(v) : std::string(fallback); +#endif +} + +} // namespace + +int main(int argc, char** argv) { + long long rows = 200000; + int reps = 9; + int warmup = 1; + std::vector legs; + + for (int i = 1; i < argc; ++i) { + std::string a = argv[i]; + auto next = [&](const char* what) -> std::string { + if (i + 1 >= argc) { + std::cerr << "ERROR: " << what << " needs a value\n"; + std::exit(1); + } + return argv[++i]; + }; + if (a == "--rows") { + rows = std::stoll(next("--rows")); + } else if (a == "--reps") { + reps = std::stoi(next("--reps")); + } else if (a == "--warmup") { + warmup = std::stoi(next("--warmup")); + } else if (a == "--driver") { + std::string spec = next("--driver"); + auto eq = spec.find('='); + std::string label = (eq == std::string::npos) ? spec : spec.substr(0, eq); + std::string value = (eq == std::string::npos) ? spec : spec.substr(eq + 1); + bool direct = false; + if (value.rfind("dll:", 0) == 0) { + direct = true; + value = value.substr(4); + } + legs.push_back({label, value, direct}); + } else { + std::cerr << "ERROR: unknown arg '" << a << "'\n"; + return 1; + } + } + + if (warmup >= reps) { + std::cerr << "ERROR: --warmup (" << warmup << ") must be < --reps (" + << reps << ")\n"; + return 1; + } + + if (legs.empty()) { + // Prefer direct-load legs when DLL paths are supplied (no DM registration + // needed); otherwise fall back to DM-registered driver names. + std::string ndll = GetEnvOr("ODBC_BENCH_NATIVE_DLL", ""); + std::string rdll = GetEnvOr("ODBC_BENCH_RUST_DLL", ""); + if (!ndll.empty() && !rdll.empty()) { + legs.push_back({"native", ndll, true}); + legs.push_back({"rust", rdll, true}); + } else { + legs.push_back({"native", + GetEnvOr("ODBC_BENCH_NATIVE_DRIVER", + "ODBC Driver 18 for SQL Server"), + false}); + legs.push_back( + {"rust", GetEnvOr("ODBC_BENCH_RUST_DRIVER", "mssql-odbc"), false}); + } + } + + auto& cfg = ODBCTestConfig::Instance(); + if (!cfg.HasConnection()) { + std::cerr << "ERROR: no connection configured — set ODBC_TEST_SERVER " + "(and ODBC_TEST_UID / ODBC_TEST_PWD for SQL auth)\n"; + return 1; + } + + std::string sql = BuildQuery(rows); + std::cout << "Fetch-throughput A/B benchmark\n" + << " server = " << cfg.Server() << "\n" + << " database = " << cfg.Database() << "\n" + << " rows = " << rows << "\n" + << " reps = " << reps << " (warmup " << warmup + << " discarded)\n" + << " query = " << sql << "\n"; + + std::vector> medians; + for (const auto& leg : legs) { + auto kept = RunLeg(leg, sql, reps, warmup); + std::vector ms; + for (const auto& r : kept) ms.push_back(r.ms); + double med = Median(ms); + double medRps = (kept.empty() || med <= 0.0) + ? 0.0 + : (kept.front().rows / (med / 1000.0)); + medians.push_back({leg.label, med}); + std::cout << " --> median " << std::fixed << std::setprecision(2) + << med << " ms (" << static_cast(medRps) + << " rows/s)\n"; + } + + std::cout << "\n=== Summary (median ms, lower is faster) ===\n"; + for (const auto& m : medians) { + std::cout << " " << std::setw(10) << std::left << m.first << " " + << std::fixed << std::setprecision(2) << m.second << " ms\n"; + } + + // A/B ratio: rust median / native median when both labels are present. + auto find = [&](const std::string& lbl) -> double { + for (const auto& m : medians) + if (m.first == lbl) return m.second; + return -1.0; + }; + double nativeMed = find("native"); + double rustMed = find("rust"); + if (nativeMed > 0.0 && rustMed > 0.0) { + std::cout << "\n=== A/B: Rust vs native ===\n" + << " ratio (rust median / native median) = " << std::fixed + << std::setprecision(3) << (rustMed / nativeMed) << "x\n" + << " (1.00x = parity; >1 = Rust slower; deltas <15% are " + "noise)\n"; + } + return 0; +} diff --git a/mssql-odbc/tests/e2e/run_bench.ps1 b/mssql-odbc/tests/e2e/run_bench.ps1 new file mode 100644 index 00000000..652d5a12 --- /dev/null +++ b/mssql-odbc/tests/e2e/run_bench.ps1 @@ -0,0 +1,170 @@ +# Copyright (c) Microsoft Corporation. +# Licensed under the MIT License. +# +# Build and run the fetch-throughput A/B benchmark: native +# "ODBC Driver 18 for SQL Server" vs the Rust mssql-odbc dev driver. +# +# Mirrors run_e2e.ps1's toolchain for building the Rust cdylib (msodbcsql18.dll), +# then runs the benchmark WITHOUT touching the registry: fetch_bench loads each +# driver DLL directly (its own tiny driver manager) via the `dll:` leg +# syntax, so the unregistered Rust dev driver runs with zero admin rights and +# both A/B legs execute on an identical, DM-free code path (a fairer compare). +# +# Requires: a live SQL Server, msodbcsql18 installed, MSVC + CMake (VS "C++ +# CMake tools" component). NO administrator needed. +# +# Usage: .\run_bench.ps1 [-Release] [-Rows N] [-Reps R] [-Warmup W] +# [-VcVarsVer 14.44] [-NativeDll PATH] [-RustDll PATH] +# +# Connection info resolution (matches the P8 harness contract): +# ODBC_TEST_SERVER = , from mssql-tds/.env +# ODBC_TEST_UID = from mssql-tds/.env +# ODBC_TEST_PWD = $env:SQL_PASSWORD, else contents of C:\tmp\password +# ODBC_TEST_DATABASE = tempdb ODBC_TEST_TRUST_CERT = Yes +# Any ODBC_TEST_* already set in the environment is respected and not overwritten. + +param( + [switch]$Release, + [long]$Rows = 200000, + [int]$Reps = 9, + [int]$Warmup = 1, + # MSVC toolset to select in vcvars64. Some machines ship a default toolset + # missing the x64 CRT import libs; pin a known-good one here. Set to "" to + # use the environment's default toolset. + [string]$VcVarsVer = "14.44", + [string]$NativeDll = "", + [string]$RustDll = "" +) + +$ErrorActionPreference = "Stop" + +$ScriptDir = Split-Path -Parent $MyInvocation.MyCommand.Definition +$OdbcCrateDir = Resolve-Path (Join-Path $ScriptDir "..\..") +$WorkspaceDir = Resolve-Path (Join-Path $OdbcCrateDir "..") +$BuildType = if ($Release) { "release" } else { "debug" } +$CMakeBuildType = if ($Release) { "Release" } else { "Debug" } + +# Parse KEY=VALUE lines out of an .env file into a hashtable (ignores # comments). +function Read-DotEnv([string]$Path) { + $map = @{} + if (-not (Test-Path $Path)) { return $map } + foreach ($line in Get-Content -Path $Path) { + $t = $line.Trim() + if (-not $t -or $t.StartsWith("#")) { continue } + $eq = $t.IndexOf("=") + if ($eq -lt 1) { continue } + $k = $t.Substring(0, $eq).Trim() + $v = $t.Substring($eq + 1).Trim().Trim('"').Trim("'") + $map[$k] = $v + } + return $map +} + +# Resolve the SQL password without ever echoing it: env var first, then file. +function Resolve-Password { + if ($env:SQL_PASSWORD) { return $env:SQL_PASSWORD } + $pwFile = "C:\tmp\password" + if (Test-Path $pwFile) { return (Get-Content -Raw -Path $pwFile).TrimEnd("`r", "`n") } + throw "No password: set `$env:SQL_PASSWORD or provide $pwFile" +} + +function Get-VsRoot { + $vswhere = Join-Path ${env:ProgramFiles(x86)} 'Microsoft Visual Studio\Installer\vswhere.exe' + if (Test-Path $vswhere) { + $root = & $vswhere -latest -products '*' -property installationPath 2>$null | Select-Object -First 1 + if ($root) { return $root } + } + throw "Visual Studio not found (vswhere). Install VS 2022 with the C++ workload." +} + +# Resolve the native msodbcsql18.dll path: explicit arg, then env override, then +# the HKLM ODBCINST registration, then the system32 default. +function Resolve-NativeDll { + if ($NativeDll) { return (Resolve-Path $NativeDll).Path } + if ($env:ODBC_BENCH_NATIVE_DLL) { return $env:ODBC_BENCH_NATIVE_DLL } + $reg = "HKLM:\Software\ODBC\ODBCINST.INI\ODBC Driver 18 for SQL Server" + if (Test-Path $reg) { + $d = Get-ItemProperty -Path $reg -Name "Driver" -ErrorAction SilentlyContinue + if ($d -and $d.Driver -and (Test-Path $d.Driver)) { return $d.Driver } + } + $sys = Join-Path $env:WINDIR "system32\msodbcsql18.dll" + if (Test-Path $sys) { return $sys } + throw "Could not locate native msodbcsql18.dll; pass -NativeDll ." +} + +# --- Build the Rust driver ---------------------------------------------------- +Write-Host "=== Building mssql-odbc ($BuildType) ===" +Push-Location $OdbcCrateDir +if ($Release) { cargo build --release } else { cargo build } +$TargetDir = $null +try { + $meta = cargo metadata --format-version 1 --no-deps 2>$null | ConvertFrom-Json + if ($meta -and $meta.target_directory) { $TargetDir = $meta.target_directory } +} catch { } +Pop-Location +if (-not $TargetDir) { $TargetDir = Join-Path $WorkspaceDir "target" } + +$RustDriverPath = if ($RustDll) { $RustDll } else { Join-Path $TargetDir "$BuildType\msodbcsql18.dll" } +if (-not (Test-Path $RustDriverPath)) { Write-Error "Rust driver not found at $RustDriverPath" } +$RustDriverPath = (Resolve-Path $RustDriverPath).Path +$NativeDriverPath = Resolve-NativeDll +Write-Host "Native driver: $NativeDriverPath" +Write-Host "Rust driver: $RustDriverPath" + +# --- Resolve connection env (only fill what isn't already set) ---------------- +$envFile = Join-Path $WorkspaceDir "mssql-tds\.env" +$dotenv = Read-DotEnv $envFile + +if (-not $env:ODBC_TEST_SERVER) { + $h = $dotenv["DB_HOST"]; if (-not $h) { $h = "127.0.0.1" } + # The Windows ODBC driver on this host fails to resolve the literal + # "localhost" ("No such host is known"); 127.0.0.1 connects reliably. + if ($h -eq "localhost") { $h = "127.0.0.1" } + $p = $dotenv["DB_PORT"]; if (-not $p) { $p = "1433" } + $env:ODBC_TEST_SERVER = "$h,$p" +} +if (-not $env:ODBC_TEST_UID) { + $u = $dotenv["DB_USERNAME"]; if (-not $u) { $u = "sa" } + $env:ODBC_TEST_UID = $u +} +if (-not $env:ODBC_TEST_PWD) { $env:ODBC_TEST_PWD = Resolve-Password } +if (-not $env:ODBC_TEST_DATABASE) { $env:ODBC_TEST_DATABASE = "tempdb" } +if (-not $env:ODBC_TEST_TRUST_CERT) { $env:ODBC_TEST_TRUST_CERT = "Yes" } + +Write-Host "Connection: server=$($env:ODBC_TEST_SERVER) uid=$($env:ODBC_TEST_UID) db=$($env:ODBC_TEST_DATABASE) (password hidden)" + +# --- Configure + build the benchmark target (MSVC/Ninja inside vcvars64) ------ +$VsRoot = Get-VsRoot +$VcVars = Join-Path $VsRoot "VC\Auxiliary\Build\vcvars64.bat" +$VsCMake = Join-Path $VsRoot "Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe" +$CMakeExe = if (Test-Path $VsCMake) { $VsCMake } elseif (Get-Command cmake -ErrorAction SilentlyContinue) { (Get-Command cmake).Source } else { throw "cmake not found." } +$Installer = Join-Path ${env:ProgramFiles(x86)} 'Microsoft Visual Studio\Installer' +$VerFlag = if ($VcVarsVer) { "-vcvars_ver=$VcVarsVer" } else { "" } + +Write-Host "" +Write-Host "=== Configuring + building fetch_bench (MSVC $VcVarsVer / Ninja) ===" +Push-Location $ScriptDir +$cfg = "set `"PATH=$Installer;%PATH%`" && call `"$VcVars`" $VerFlag >nul && " + + "`"$CMakeExe`" -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=$CMakeBuildType -DODBC_E2E_FORCE_UNICODE=ON && " + + "`"$CMakeExe`" --build build --target fetch_bench" +& $env:ComSpec /c $cfg +$buildExit = $LASTEXITCODE +Pop-Location +if ($buildExit -ne 0) { throw "fetch_bench build FAILED (exit $buildExit)" } + +$BenchExe = $null +foreach ($cand in @( + (Join-Path $ScriptDir "build\fetch_bench.exe"), + (Join-Path $ScriptDir "build\$CMakeBuildType\fetch_bench.exe"))) { + if (Test-Path $cand) { $BenchExe = $cand; break } +} +if (-not $BenchExe) { Write-Error "fetch_bench.exe not found under build/" } + +# --- Run the A/B benchmark (both legs direct-load, no registry) --------------- +Write-Host "" +Write-Host "=== Running fetch-throughput A/B ===" +& $BenchExe --rows $Rows --reps $Reps --warmup $Warmup ` + --driver "native=dll:$NativeDriverPath" ` + --driver "rust=dll:$RustDriverPath" +$exit = $LASTEXITCODE +if ($exit -ne 0) { throw "benchmark FAILED (exit $exit)" } From 2e0a57fbb07a0c402a7695cdff37d50ee126259a Mon Sep 17 00:00:00 2001 From: Saurabh Singh <1623701+saurabh500@users.noreply.github.com> Date: Sat, 8 Aug 2026 12:20:55 -0700 Subject: [PATCH 2/2] Note release-build requirement for the Rust benchmark leg A debug mssql-odbc build is 10-30x slower purely from codegen and swamps the real driver-path cost; document that the Rust leg must be a --release build so the A/B baseline reflects the driver, not debug overhead. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a3c262b9-1962-4d93-b08e-e3bc84aa4ac0 --- mssql-odbc/tests/e2e/README.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mssql-odbc/tests/e2e/README.md b/mssql-odbc/tests/e2e/README.md index 572e6a7e..ed4ccc13 100644 --- a/mssql-odbc/tests/e2e/README.md +++ b/mssql-odbc/tests/e2e/README.md @@ -339,6 +339,12 @@ I/O bound; see PR #186); the query returns many rows so row decode dominates. so nothing is optimised away — it matches across both legs, confirming an identical, fair workload. +> **Always benchmark the Rust leg as a `--release` build.** A debug `mssql-odbc` +> build is 10–30x slower purely from codegen (no inlining, overflow checks), which +> swamps the real driver-path cost and produces a badly misleading gap. Pass +> `-Release` to `run_bench.ps1` (it builds `cargo build --release` and points the +> Rust leg at `target\release\msodbcsql18.dll`). + ### Driver-manager bypass (no admin, no registry) Each leg loads a driver **DLL directly** (`LoadLibrary` + `GetProcAddress`,