Fix ANSI (PT_STRING8) text extraction: folder names, subjects, senders, bodies#62
Open
mekinney wants to merge 2 commits into
Open
Fix ANSI (PT_STRING8) text extraction: folder names, subjects, senders, bodies#62mekinney wants to merge 2 commits into
mekinney wants to merge 2 commits into
Conversation
PidTagDisplayName (folder name, prop 12289) was read with GetString(), which requires PropertyTypeString (PT_UNICODE). ANSI PSTs store it as PropertyTypeString8 (a codepage string), so GetString() returned ErrPropertyTypeMismatch on the first folder and aborted the entire WalkFolders — making ANSI .pst files unreadable. Branch on the actual property type and decode PT_STRING8 via GetString8 (the existing path, defaulting to Windows-1252). Resolves the in-code TODO at this site. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Building on the folder-name fix, ANSI .pst files still surfaced every message string (subject, sender, body, ...) as empty. Two causes: 1. The generated property structs key each text field by "<ID><Type>" using the Unicode string type 31 (e.g. Subject is msg:"5531"). On ANSI PSTs those properties carry PropertyTypeString8 (type 30), so WriteMessagePackValue wrote them under a "<ID>30" key that no struct field matched -- the values were read from the heap correctly but then dropped on decode. Normalize the String8 key to the Unicode string type so the value lands in the same field the Unicode path populates. Also default String8 decoding to Windows-1252 (a superset of ASCII covering Western European text) instead of UTF-8, matching the folder-name path. 2. GetMessage read PidTagMessageClass with GetString, which rejects String8, so every ANSI item fell back to properties.Message and appointments/contacts were misclassified. Add GetStringValue, a type-agnostic getter (String or String8), and use it for the class. Adds TestANSIStringProperties over data/32-bit.pst asserting a PT_STRING8 PidTagSubject decodes to its expected text. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ANSI
.pstfiles (Outlook 97-2002) are unreadable today: folder names and every message string property (subject, sender, body, …) come back empty, andWalkFoldersaborts on the first folder.ANSI stores text as
PropertyTypeString8(a code-page string) rather than thePropertyTypeString(UTF-16) used by Unicode PSTs. Two places assume Unicode:Folder walk aborts.
GetSubFoldersreadsPidTagDisplayName(prop 12289) withGetString(), which returnsErrPropertyTypeMismatchon aString8value — aborting the entire walk on the first folder.String values silently dropped on decode. The generated property structs key each text field by
"<ID><Type>"using the Unicode string type 31 (e.g.Subjectismsg:"5531").WriteMessagePackValuewrites aString8value under a"<ID>30"key, which no struct field matches — so the value is read from the heap correctly and then dropped during msgp decode. Integer/time properties survive because their type is identical across ANSI and Unicode; only strings are affected.Additionally,
GetMessagereadsPidTagMessageClasswithGetString(), so on ANSI every item fails that read and falls back toproperties.Message{}— misclassifying appointments/contacts.Fix
folder.go— branch on the actual property type forPidTagDisplayName: decodeString8viaGetString8(Windows-1252), elseGetString. The walk no longer aborts on ANSI.property_reader.go— inWriteMessagePackValue, normalize theString8message-pack key to the Unicode string type (31) so the value lands in the same struct field the Unicode path populates. DefaultString8decoding to Windows-1252 (a superset of ASCII covering Western European text) rather than UTF-8.property_reader.go— addGetStringValue(), a type-agnostic getter (StringorString8), for callers that don't control the PST format.message.go— read the message class viaGetStringValue()so ANSI items classify correctly.Windows-1252is the pragmatic default; a fully correct implementation would honorPidTagMessageCodepage/PidTagInternetCodepage, but 1252 decodes the common case (ASCII + Western European) correctly and matches the existing folder-name path.Test
Adds
TestANSIStringPropertiesover the existingdata/32-bit.pst(ANSI) sample, asserting that aPT_STRING8PidTagSubjectdecodes to its expected text. The existing test suite still passes.Notes
These two commits were developed for and validated by a downstream consumer (private-recall) against real ANSI archives — subjects, senders, sender emails, and bodies all extract; no regression on Unicode PSTs.