Skip to content

Optimize memory allocations in query execution#20

Merged
emicklei merged 2 commits into
mainfrom
perf-alloc-optimization
Aug 22, 2025
Merged

Optimize memory allocations in query execution#20
emicklei merged 2 commits into
mainfrom
perf-alloc-optimization

Conversation

@google-labs-jules
Copy link
Copy Markdown
Contributor

This change optimizes the main query execution paths to reduce memory
allocations and improve performance.

  • In QuerySet.Exec, the slice for scanning row values is now
    allocated only once outside the loop.
  • In execIntoMaps, the slice for scannable values is also created
    only once.
  • In resultIterator.Next, the slice for scan values is now part of the
    iterator struct and is reused for each row, avoiding reallocation
    on each call.

These changes significantly reduce the garbage collector pressure when
processing large result sets. I also fixed a unit test that was failing due to these changes.

This change optimizes the main query execution paths to reduce memory
allocations and improve performance.

- In `QuerySet.Exec`, the slice for scanning row values is now
  allocated only once outside the loop.
- In `execIntoMaps`, the slice for scannable values is also created
  only once.
- In `resultIterator.Next`, the slice for scan values is now part of the
  iterator struct and is reused for each row, avoiding reallocation
  on each call.

These changes significantly reduce the garbage collector pressure when
processing large result sets.
@emicklei emicklei marked this pull request as ready for review August 22, 2025 08:39
@emicklei emicklei requested a review from Copilot August 22, 2025 08:39

This comment was marked as outdated.

This change optimizes the main query execution paths to reduce memory
allocations and improve performance.

- In `QuerySet.Exec`, the slice for scanning row values is now
  allocated only once outside the loop.
- In `resultIterator.Next`, the slice for scan values is now part of the
  iterator struct and is reused for each row, avoiding reallocation
  on each call.

These changes significantly reduce the garbage collector pressure when
processing large result sets.

An incorrect optimization for `execIntoMaps` was reverted.
@emicklei emicklei requested a review from Copilot August 22, 2025 09:13
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR optimizes memory allocations in the main query execution paths to reduce garbage collector pressure when processing large result sets. The changes focus on reusing slice allocations rather than creating new ones for each row iteration.

  • Pre-allocates scan value slices outside of loops in QuerySet.Exec and resultIterator
  • Reuses the scanValues slice as a struct field in resultIterator to avoid per-row allocations
  • Updates test setup to include the new scanValues field initialization

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
queryset.go Pre-allocates scan values slice outside the row iteration loop and adds scanValues field to resultIterator initialization
iterator.go Adds scanValues field to resultIterator struct and reuses it in Next() method instead of creating new slice each call
iterator_test.go Updates test setup to initialize the new scanValues field in resultIterator

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@emicklei emicklei merged commit 9509730 into main Aug 22, 2025
1 check passed
@emicklei emicklei deleted the perf-alloc-optimization branch August 22, 2025 19:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants