Write pending queue during online indexing#4293
Draft
jjezra wants to merge 27 commits into
Draft
Conversation
jjezra
force-pushed
the
vector_index_pending_queue
branch
from
9b7b422 to
df51931
Compare
Appears to be unused
…onDB#4285) ### Summary - The `DistinctRecordsProperty` visitor previously returned `false` unconditionally for `RecordQueryExplodePlan`, treating all explode results as potentially non-distinct. - In the `WITH ORDINALITY` variant, each output row includes a unique 1-based ordinal alongside the array element. Even if the source array contains duplicate values, the (element, ordinal) tuples are always distinct because the ordinal is unique. - Updated `visitExplodePlan` to return `element.isWithOrdinality()` so the planner can recognize distinctness.
Major updates: These have large changes and some breakages, but no source changes are necessary on our end. * jline 3.30.4 → 4.1.3 (released 2026-05-28). * mockito 3.7.7 → 5.23.0 (released 2026-03-11). * hamcrest 2.2 → 3.0 (released 2024-08-01). Minor updates: * JUnit Jupiter → 5.14.4 (released 2026-04-26) and JUnit Platform → 1.14.4 (released 2026-04-26). (The major upgrade to JUnit 6.0 is left for later.) * assertj 3.26.3 → 3.27.7 (released 2026-01-24). * java-diff-utils 4.12 → 4.15 (released 2024-11-23). * httpclient 5.2.1 → 5.6.1 (released 2026-04-15). * snakeyaml 2.2 → 2.6 (released 2026-02-26). * jcommander 1.81 → 3.00 (released 2025-10-05).
…4286) ## Summary - Adds `AGENTS.md` (shared context file for all AI assistants) with project overview, test strategy, and agent routing rules; `CLAUDE.md`, `GEMINI.md`, and `.github/copilot-instructions.md` are symlinks to it - Adds Claude Code-specific tooling under `.claude/`: skills for code review, coding standards, docs writing, query processing, test running, and session management - Skills are invocable via `/skill-name` in Claude Code and provide reusable, focused instruction sets for common tasks ## Test plan - [ ] Verify `/code-reviewer`, `/using-gradle`, `/test-runner`, and other skills load correctly in Claude Code - [ ] Confirm `CLAUDE.md` and `GEMINI.md` resolve correctly as symlinks to `AGENTS.md` - [ ] Check that the routing rules in `AGENTS.md` match the available skills
) ## Summary Extends the deserialization retry path in `TransformedRecordSerializer` to cover failures raised by `decrypt`, building on the retry mechanism added in FoundationDB#4009. ### Background — what FoundationDB#4009 added FoundationDB#4009 introduced retry-on-deserialization for `TransformedRecordSerializer`: - When `decompress` throws after a successful `decrypt`, the serializer rebuilds a fresh `TransformedRecordSerializerState` from the original raw bytes and re-runs the entire decrypt + decompress pipeline, up to `deserializeReattemptCount` times. - A `failOnDeserializeReattempt` builder flag, when set, throws a `RecordSerializationException` with `RESULT="success"` and `RETRY_COUNT` even when retries succeed — useful for surfacing transient faults instead of silently swallowing them. In that PR retries were scoped to `decompress` only — a `decrypt` that threw propagated immediately, and there was no retry path at all when compression was disabled. ### What this PR adds - `decryptOrThrow` failures are now caught into the same `failure` slot as `decompressOrThrow`, so they trigger the existing retry-from-raw-bytes loop. The first failing step short-circuits the second. - `failOnDeserializeReattempt` now also fires when decryption succeeded only on retry, matching the diagnostic intent for the decompression case. - The success-on-retry exception message changed from `"decompression error"` to the more accurate `"deserialization error"`. The original failure exception (when retries are exhausted) still carries the specific decrypt-vs-decompress identity in its log info. - This also enables retries on encryption-only serializers (compression disabled). Previously such a configuration had no retry path at all.
… fields can be converted appropriately (FoundationDB#4243) This fixes a gap that would block the use of UUID-valued fields through the direct access API. It adds a new test fixture for record round tripping (currently validated on all the scalar types but not struct or array types). It also makes use of the new UUID-valued fields to extend a `UniqueIndexTest` with `UUID`s. (The actual unique indexes show correct semantics in this case.) This fixes FoundationDB#4240. Co-authored-by: Pranjal Gupta <g31.pranjal@gmail.com>
This PR adds a step to the `gradle-test` action, which is used in the `nightly`, `pull-request` and `release` workflows, to cache build assets that are downloaded manually from our gradle build files. To make this simpler, a new project-level property `downloadsDirectory` is added to each gradle project at a known top-level path, and that path is cached via GitHub Actions dependency caching. The cached assets are invalidated if any of the gradle build files are changed, and the cache key includes the gradle command passed to the `gradle-test` action.
…B#4289) When a transaction fails with `NOT_COMMITTED` (conflict), it can be difficult to know whether the conflict was caused by a background index build via OnlineIndexer. Indexes in WRITE_ONLY state are actively being written to by both the OnlineIndexer and normal record save operations — making them a potential source of contention. This change tracks which write-only indexes were touched during a transaction and, if that transaction conflicts, these values can be inspected. In support of this, a mechanism to store and retrieve values in a `FdbRecordContext` was added in a generic fashion using the `session` field. It is meant for external consumption, with a protected typed API that prevents access to other pre-existing values. Resolves FoundationDB#4288
This indexing option will cause user updates to accumulate in a special queue during ongoing online indexing. The indexer will drain this queue after every transaction. This resolves 4292
jjezra
force-pushed
the
vector_index_pending_queue
branch
from
e9d76fb to
990398c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
During online indexing, under certain circumstances, short indexer transactions will repeatedly cause conflicts with user transactions (while the user attempts to update the index). To avoid this, we should add a new index state: WRITE_ONLY_WITH_QUEUE. In this index state, user updates will be added to a write-pending queue, and the indexer will drain the queue.