`.go({ data: 'raw', pages: 'all' })` only returns the first DynamoDB page

[jonnedeprez]

Summary

When a query is configured with both data: 'raw' and pages: 'all' (or pages > 1), only the first DynamoDB Query/Scan response is returned. Subsequent pages are never fetched, even though LastEvaluatedKey is present on the underlying DDB response.

This is silently lossy: the caller asked for "all pages" but gets a single page, with no warning or error. It's especially dangerous when combined with a selective FilterExpression, because DynamoDB applies the filter after reading ~1 MB of raw partition data — so the first page can return Items: [] while matches exist deeper in the partition. ElectroDB sees the empty page, returns it, and the caller silently gets zero results.

Versions affected

At least 3.5.0 through 3.7.5 (latest at time of writing). Same code in both.

Reproduction (real-world)

We hit this in a data-correction backfill task where we used 'raw' to bypass a getter. The query:

const result = await InvoiceEntity.query
  .primary({ userId })
  .where(
    ({ deleted, layoutSettings }, { eq, exists, notExists }) =>
      eq(deleted, false) +
      ' AND ' +
      exists(layoutSettings) +
      ' AND ' +
      notExists(layoutSettings.brandId)
  )
  .go({
    data: 'raw',
    attributes: [...],
    pages: 'all',
  });

const result = await InvoiceEntity.query
  .primary({ userId })
  .where(
    ({ deleted, layoutSettings }, { eq, exists, notExists }) =>
      eq(deleted, false) +
      ' AND ' +
      exists(layoutSettings) +
      ' AND ' +
      notExists(layoutSettings.brandId)
  )
  .go({
    data: 'raw',
    attributes: [...],
    pages: 'all',
  });

For a user with a couple hundred invoices, of which ~23 match the filter, the query returned Items: []. The compiled FilterExpression was correct — confirmed via .params() and by running the identical query against DynamoDB with the AWS CLI, which returned the matches.

Removing only data: 'raw' (keeping pages: 'all') makes the same query return all 23 matches.

Root cause — src/entity.js, multi-page loop

In 3.7.5 around lines 703–775 (and the same shape in 3.5.0 ~line 711):

do {
  let response = await this._exec(method, { ExclusiveStartKey, ...parameters }, config);
  ExclusiveStartKey = response.LastEvaluatedKey;
  response = this.formatResponse(response, parameters.IndexName, { ... });
  if (config.data === DataOptions.raw) {
    return response;          // ← early return regardless of ExclusiveStartKey / pages
  } else if (config._isCollectionQuery) {
    ...
  } else if (Array.isArray(response.data)) {
    results = [...results, ...items];
    ...
  }
  iterations++;
} while (
  ExclusiveStartKey &&
  (pages === AllPages || config.count !== undefined || iterations < pages) &&
  (config.count === undefined || count < config.count)
);

The if (config.data === DataOptions.raw) { return response; } is inside the do/while, so the loop exits after iteration 1. The pages value and ExclusiveStartKey are evaluated only at the while condition, which is unreachable for raw queries.

Expected behaviour

One of:

1. Accumulate raw Items across pages (mirroring how non-raw results are concatenated into results) and return a single response with merged Items and the final LastEvaluatedKey. This is what most users would expect from pages: 'all'.
2. Or, if (1) isn't desirable for the data: 'raw' shape, throw / log a warning when data: 'raw' is combined with pages ≠ 1, so callers aren't silently misled.

Suggested doc note (regardless of fix)

The Query section of the docs should call out that data: 'raw' is incompatible with pages: 'all'. Currently both options are described independently with no warning that combining them silently drops pages.

[thealpha93]

I'd like to work on this.
I've traced the root cause to an early return inside the pagination loop in entity.js that exits after the first page when data: 'raw' is set, originally intentional when auto-pagination was added, but it makes pages: 'all' a silent no-op. I'll open a PR with a fix.