fix: require an exact-match key field for update/delete document#2
Merged
federico-martinez-invgate merged 1 commit intoJul 9, 2026
Merged
Conversation
updateDocument/deleteDocument accepted ANY field as key_field and ran delete_term(Term::from_field_text(field, value)) with no check that the field is a non-tokenized exact key. Passing a tokenized `text` field (or a STORED-only `attribute`) silently corrupted the index: the whole value is never stored as a single term, so delete_term either matched nothing (update -> the delete is a no-op and the add creates a DUPLICATE document for the key) or matched a shared token (delete -> MASS DELETION of every document containing that token). Add field_is_exact_key() in schema.rs -- a field is a valid key iff it is a STRING str field, i.e. indexed with the "raw" tokenizer (STRING uses "raw", TEXT uses "default", attributes have no indexing options) -- and a resolve_key_field() guard used by both delete_by_id and update_document that rejects tokenized/attribute fields with a clear, actionable error BEFORE any writer mutation. The pre-existing "field does not exist" path is preserved. Two tests reproduce both corruption modes (update duplication, delete mass-deletion) against the old behavior and now assert the operations are rejected without corrupting the index. Validated end-to-end against the real Service Desk indexes (Incidents / KnowledgeBase / CatalogCategories), whose update/delete paths only ever use id_key / doc_key (both in the `keys` bucket), so the fail-loud behavior never triggers for correct usage.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
updateDocument/deleteDocumentaccepted any field askey_fieldand randelete_term(Term::from_field_text(field, value))with no check that the field is a non-tokenizedexact key. Passing a tokenized
textfield (or a STORED-onlyattribute) silently corrupted theindex:
delete_termmatches nothing, the delete is a no-op, and the add creates a duplicate document for the key.
that token, not one.
Fix
field_is_exact_key()inschema.rs: a field is a valid key iff it's aSTRINGstr field —indexed with the
"raw"tokenizer (STRING uses"raw", TEXT uses"default", attributes have noindexing options). This rejects both tokenized
textfields and non-indexedattributefields.resolve_key_field()guard inwriter.rs, called by bothdelete_by_idandupdate_document,which returns a clear error before any writer mutation. The existing "field does not exist"
path is preserved. Plain
TantivyExceptionin PHP (no new exception type) — this is a programmererror, not a runtime-recoverable condition.
Tests
Two new tests reproduce both corruption modes against the old behavior (update → 2 copies;
delete → 0 docs) and now assert the operations are rejected without corrupting the index. Full suite:
20/20 pass (17 core + 3 ext),
clippy -D warningsandfmt --checkclean.Real-product validation
Audited
service-desk-mainbefore merge: the onlyupdateDocument/deleteDocumentcall-sites (thewrite feed) derive
key_fieldfromtantivyIdField()=id_key/doc_key, both in thekeysbucket for all three indexes (Incidents / KnowledgeBase / CatalogCategories). Ran the real engine:
update-by-key leaves no duplicate, delete-by-key removes only the target (including composite
doc_key), and the negative cases (tokenized/attribute key) throw the expected exception. Nocall-sites need changes.
🤖 Generated with Claude Code