Patch Doltlite to preserve original bytes on lossy-collation index entries

prollyBtCursorInsert stored no value for non-INTKEY inserts where
every record field fits into the sort key (splitKey=0), relying on
getCursorPayload to reconstruct the record from the sort key bytes
on read. That round-trip is lossless for BINARY but the sort-key
encoder in sortkey.c folds A-Z to a-z under NOCASE and strips
trailing spaces under RTRIM, so when SQLite picks the auto-index as
a covering index for a plain column read (e.g. `SELECT pkcol FROM t`
on a composite-PK rowid table with a NOCASE pkcol), OP_Column emits
the folded bytes instead of the original column value.

Reproduces testAlterTableModifyColumnComplexChange — the driver
rebuilds the table via CREATE/INSERT/RENAME and then reads back
'johnny' where it stored 'Johnny'.

Patch: when any pKeyInfo collation is non-identity (NOCASE or
RTRIM), preserve pPayload->pKey as the value alongside the sort
key. getCursorPayload prefers the stored value over reconstruction,
so reads return the original bytes. No behavior change for BINARY-
only records.

Author: JanJakes

.github/doltlite-patches/preserve-original-value-when-sortkey-lossy.patch

@@ -0,0 +1,72 @@
+Preserve the original record when the sort-key encoding is lossy
+
+For a non-INTKEY cursor, prollyBtCursorInsert normally picks between
+two storage shapes:
+
+  - splitKey=1 — sort key holds a prefix of the record and the rest of
+    the record is stored as the value (WITHOUT ROWID tables, secondary
+    indexes);
+  - splitKey=0 — sort key holds every field and no value is stored.
+
+On read, getCursorPayload prefers the stored value; if none is
+present it reconstructs a SQLite record from the sort key bytes. That
+round-trip is lossless for BINARY collations but the sort-key encoder
+in sortkey.c lower-cases A-Z under NOCASE and strips trailing spaces
+under RTRIM.
+
+When SQLite picks the auto-index as a covering index for a plain
+column read — e.g. `SELECT pkcol FROM t` on a composite-PK rowid
+table with a NOCASE pkcol — OP_Column reads the reconstructed (folded)
+bytes from getCursorPayload and returns them in place of the original
+column value. Reproduces on:
+
+  CREATE TABLE t (ID INT, pkcol TEXT COLLATE NOCASE, extra TEXT,
+                  PRIMARY KEY (ID, pkcol));
+  INSERT INTO t VALUES (1, 'Johnny', 'x');
+  SELECT pkcol FROM t;                 -- 'johnny' (wrong) without fix
+  SELECT * FROM t;                     -- 'Johnny' (right) both ways
+
+Fix: when any field in pKeyInfo uses a collation whose sort-key
+encoding is not a bijection over the original bytes (NOCASE, RTRIM),
+preserve the original record as the value alongside the sort key.
+getCursorPayload then prefers the original bytes over reconstruction.
+No behavior change when all collations are BINARY: the value side
+stays empty and storage footprint is unchanged.
+
+--- a/src/prolly_btree.c
++++ b/src/prolly_btree.c
+@@ -4716,7 +4716,33 @@
+                                       pCur->pKeyInfo,
+                                       &pSortKey, &nSortKey);
+     if( rc==SQLITE_OK ){
+-      if( splitKey ){
++      /* When every field of the record is folded into the sort key
++      ** (splitKey==0) we would normally store no value and, on read,
++      ** reconstruct the SQLite record from the sort key. That is
++      ** lossless for BINARY collations but the sort-key encoder
++      ** lower-cases A-Z under NOCASE and strips trailing spaces
++      ** under RTRIM. When SQLite picks the auto-index as a covering
++      ** index for a plain column read (e.g. `SELECT pkcol FROM t`
++      ** on a composite-PK rowid table with NOCASE pkcol), OP_Column
++      ** reads the reconstructed (folded) bytes and returns them to
++      ** the caller in place of the original column value. Preserve
++      ** the original record as the value whenever any field in
++      ** pKeyInfo uses a lossy collation so the read path prefers
++      ** the original bytes over the reconstructed ones. */
++      int lossy = 0;
++      if( pCur->pKeyInfo ){
++        int i;
++        for( i = 0; i < pCur->pKeyInfo->nAllField; i++ ){
++          CollSeq *pColl = pCur->pKeyInfo->aColl[i];
++          if( pColl && pColl->zName
++           && ( sqlite3StrICmp(pColl->zName, "NOCASE")==0
++             || sqlite3StrICmp(pColl->zName, "RTRIM")==0 ) ){
++            lossy = 1;
++            break;
++          }
++        }
++      }
++      if( splitKey || lossy ){
+         rc = prollyMutMapInsert(pCur->pMutMap,
+                                  pSortKey, nSortKey, 0,
+                                  (const u8*)pPayload->pKey, (int)pPayload->nKey);

.github/workflows/phpunit-tests-doltlite.yml

@@ -32,7 +32,7 @@ jobs:
           path: doltlite-src
 
       - name: Patch Doltlite for mysql-on-sqlite compatibility
-        # Narrow, verifiable source-level patches applied before build:
+        # Four narrow, verifiable source-level patches applied before build:
         #
         # 1) src/build.c — neutralize the auto-conversion of composite/TEXT-PK
         #    tables to WITHOUT ROWID, so the driver's `ORDER BY ROWID`
@@ -55,6 +55,13 @@ jobs:
         #    entry check the mutmap for a DELETE on that tree key; if
         #    found, skip and advance. Without this, INSERT+DELETE+UPDATE
         #    in one transaction trips "database disk image is malformed".
+        #
+        # 4) src/prolly_btree.c — preserve the original record bytes as
+        #    the index value when any pKeyInfo collation is lossy
+        #    (NOCASE, RTRIM). Without this, auto-index covering reads
+        #    reconstruct a SQLite record from the sort key and return
+        #    folded bytes (e.g. 'johnny' instead of 'Johnny') for plain
+        #    column reads on composite-PK rowid tables.
         run: |
           BUILD=doltlite-src/src/build.c
           MOVETO=doltlite-src/src/prolly_btree.c
@@ -74,8 +81,12 @@ jobs:
           patch -d doltlite-src -p1 --no-backup-if-mismatch \
             < .github/doltlite-patches/mergeScan-check-tree-delete.patch
 
+          # --- Patch 4: preserve original record bytes on lossy-collation index entries ---
+          patch -d doltlite-src -p1 --no-backup-if-mismatch \
+            < .github/doltlite-patches/preserve-original-value-when-sortkey-lossy.patch
+
           echo "--- Patched lines ---"
-          grep -n 'patched off in CI\|savedEqSeen\|walked past' "$BUILD" "$MOVETO"
+          grep -n 'patched off in CI\|savedEqSeen\|walked past\|lossy = 1' "$BUILD" "$MOVETO"
 
       - name: Build Doltlite shared library
         run: |
@@ -120,7 +131,7 @@ jobs:
           echo "ldd pdo_sqlite:"
           ldd "$(php -r 'echo ini_get("extension_dir");')/pdo_sqlite.so"
 
-      - name: Verify Doltlite is active and patches took effect
+      - name: Verify Doltlite is active and both patches took effect
         run: |
           VERSION=$(php -r 'echo (new PDO("sqlite::memory:"))->query("SELECT SQLITE_VERSION()")->fetch()[0];')
           ENGINE=$(php -r 'echo (new PDO("sqlite::memory:"))->query("SELECT doltlite_engine()")->fetch()[0];')
@@ -183,6 +194,20 @@ jobs:
             echo "::error::mergeScan patch did not take effect: expected 'x', got '${VAL}'"
             exit 1
           fi
+          # 4) lossy-collation patch: a covering-index read on a
+          # composite-PK rowid table with a NOCASE text column no
+          # longer returns folded bytes.
+          NAME=$(php -r '
+            $db = new PDO("sqlite::memory:");
+            $db->exec("CREATE TABLE t (id INT, pkcol TEXT COLLATE NOCASE, extra TEXT, PRIMARY KEY (id, pkcol))");
+            $db->exec("INSERT INTO t VALUES (1, \"Johnny\", \"x\")");
+            echo $db->query("SELECT pkcol FROM t")->fetch()[0];
+          ' 2>&1)
+          echo "covering-index pkcol = ${NAME}"
+          if [ "${NAME}" != "Johnny" ]; then
+            echo "::error::lossy-collation patch did not take effect: expected 'Johnny', got '${NAME}'"
+            exit 1
+          fi
 
       - name: Install Composer dependencies (root)
         uses: ramsey/composer-install@v3