Skip to content

OXY-17: Add IN / NOT IN value-list predicates to the sql query DSL - #300

Open
Kalin-Rudnicki wants to merge 3 commits into
mainfrom
OXY-17
Open

OXY-17: Add IN / NOT IN value-list predicates to the sql query DSL#300
Kalin-Rudnicki wants to merge 3 commits into
mainfrom
OXY-17

Conversation

@Kalin-Rudnicki

Copy link
Copy Markdown
Owner

What

Adds IN / NOT IN value-list support to the oxygen-sql @compile query DSL:

@compile
val peopleByIds: QueryIO[Seq[UUID], Person] =
  for {
    ids <- input[Seq[UUID]]
    p   <- select[Person]
    _   <- where if p.id.in(ids)   // or p.id.notIn(ids)
  } yield p

Filters a single column against a runtime Seq[A] with expanded placeholders col IN (?, ?, ...) / NOT IN (...).

How

The ? count depends on the runtime list length, which is unknown when QueryContext.sql is baked at construction. So the compiled SQL carries a unique sentinel token where the predicate goes, plus a runtime InClause expander that renders the final SQL per-execution:

  • empty IN -> FALSE, empty NOT IN -> TRUE (never the illegal IN ()).
  • Each element binds as its own scalar via a new InputEncoder.SeqEncoder, embedded in the normal encoder chain, so multi-input interleaving stays correct with no execute-time special-casing.

Kept deliberately distinct from OXY-6's single-array = ANY(?) / UNNEST.

Decisions (see report for full list)

  • Spelling col.in(coll) / col.notIn(coll); collection is Seq[A] (Set -> .toSeq).
  • notIn first-class (clean empty-list rewrite). Subquery IN (SELECT ...) out of scope.
  • batched + dynamic IN unsupported (throws); value-list must be a runtime input, not const.

Verification

  • oxygen-sql + sql-it/Test compile clean.
  • New CustomQuerySpec "in / notIn (OXY-17)" runs against a real Postgres (testcontainers) and passes: non-empty / single / empty(FALSE) / large lists, NOT IN, NOT IN empty(TRUE), composition, delete-by-IN, multi-input interleaving.
  • Full CustomQuerySpec (13 tests) passes — no regressions.
  • Docs updated (docs/docs/sql/queries.md), cross-linked to OXY-6.

Confidence: 8.5/10

High confidence in the implemented scope (green integration tests, localized design). Residual risk: IN embedded in subqueries / INSERT-FROM-SELECT not explicitly tested; const-list & Set inputs intentionally unsupported.

Full notes + assumptions: report/OXY-17.md.

🤖 Generated with Claude Code

@Kalin-Rudnicki

Copy link
Copy Markdown
Owner Author

💬 Feedback from Kalin

A core design goal of oxygen-sql is 100% static SQL strings — a query's SQL text should be fixed at compile time, not vary at runtime. This PR's IN implementation uses a sentinel token + a runtime InClause expander that emits a different number of ? placeholders depending on the collection size, which breaks that invariant.

Is that goal unreasonable here? I don't think so — can we tweak it to keep the SQL string static? The obvious candidate is to bind the whole collection as a single array parameter and emit col = ANY(?) (and col <> ALL(?) / NOT (col = ANY(?)) for NOT IN), so the SQL text is constant regardless of list size. That also sidesteps the JDBC 32,767-param limit and plan-cache churn.

Note this would converge IN with the array/ANY mechanism from the sibling PR (#298 / OXY-6) — worth deciding whether IN should just be sugar over that. Spinning up an agent to investigate feasibility + edge cases (empty list, NOT IN, NULL semantics, types) and, if viable, rework it to keep the SQL static.

@Kalin-Rudnicki

Copy link
Copy Markdown
Owner Author

Reworked IN / NOT IN to keep SQL 100% STATIC

Re: the feedback that the goal of static SQL strings is a core design goal, and that the sentinel-token + runtime InClause expander (which rewrote the SQL per execution to emit N ? placeholders) violated it.

The goal is not unreasonable, and yes — it can be tweaked. Done.

What changed

col.in(coll) / col.notIn(coll) now bind the whole collection as a single java.sql.Array parameter and emit a constant string:

DSL SQL (static, one ?)
p.id.in(ids) p.id = ANY(?)
p.id.notIn(ids) p.id <> ALL(?)

The ? count is always 1 regardless of list size, so QueryContext.sql is a plain constant again. Removed the sentinel token, the InClause runtime expander, the per-execution ctx.sql rewrite, InputEncoder.SeqEncoder, the GeneratedFragment inClauses channel, and GeneratedInClause. batched now works with IN (it was previously disabled because the ? count varied per input).

Reuses OXY-6 / OXY-18 machinery

Encoding goes through RowRepr.ArrayRepr[A](col).encoder (ArraySeqEncoderInputWriter.unsafeWriteArraycreateArrayOf), the same array-bind path introduced by OXY-18 and surfaced as ids.contains(col) in OXY-6 (#298). The runtime Seq[A] is adapted with ArraySeq.untagged.from. No ::type[] cast — the JDBC typed array carries its element type.

⚠️ Convergence flag: this makes in / notIn effectively sugar over OXY-6's = ANY(?). #298 and this PR now overlap heavily on mechanism and differ only in DSL surface (contains vs in/notIn, plus this adds <> ALL(?) negation). Suggest merging one, then rebasing the other onto a shared generation helper and deciding whether both spellings should coexist.

Truth table / edge cases (verified)

= ANY / <> ALL reproduce the exact 3-valued logic of IN / NOT IN:

  • Empty (native, no special-casing): = ANY('{}')FALSE (like IN ()), <> ALL('{}')TRUE (like NOT IN ()).
  • NOT IN + NULL footgun: preserved identically, not worsened — and unreachable here (array columns are non-nullable; optional value-lists are rejected).
  • Single-column element only; const value-lists rejected.

Verification

  • @compile(debug=true) confirms static templates: WHERE p.id = ANY(?) and WHERE p.groupId = ? AND p.id <> ALL(?).
  • New static-SQL assertions in the test: ctx.sql contains "p.id = ANY(?)", !contains "IN (", contains "p.id <> ALL(?)".
  • CustomQuerySpec 13/13 pass against real Postgres (testcontainers) — non-empty / single / empty / large / NOT IN / NOT IN empty / composition / delete / multi-input interleaving.

Full rationale + decisions in report/OXY-17.md. Confidence: 9/10.

Kalin-Rudnicki and others added 3 commits August 14, 2026 22:15
Adds `col.in(coll)` / `col.notIn(coll)` to the `@compile` query DSL, filtering a
single column against a runtime `Seq[A]` with expanded placeholders
`col IN (?, ?, ...)` / `NOT IN (...)`.

Because the `?` count depends on the runtime list length (which is not known when
`QueryContext.sql` is baked at construction), the compiled SQL carries a unique
sentinel token where the predicate goes, plus an `InClause` expander that renders
the final SQL per-execution (empty list -> `FALSE` for IN, `TRUE` for NOT IN --
never the illegal `IN ()`). Each element binds as its own scalar parameter via a
new `InputEncoder.SeqEncoder`, embedded in the normal encoder chain so multi-input
interleaving stays correct with no execute-time special-casing.

Kept deliberately distinct from OXY-6's single-array `= ANY(?)` / `UNNEST`.

- DSL: `Q.in` / `Q.notIn` extension methods.
- Parsing: dedicated `RawQueryExpr.InList` + `QueryExpr.InList` (single-column LHS,
  runtime-input RHS).
- Generation: `GeneratedInClause` third channel through `GeneratedFragment`;
  `FragmentBuilder.inList`; `makeQuery` threads clauses into `QueryI`/`QueryIO`.
- Runtime: `InClause` renderer; `QueryI`/`QueryIO.Simple` render SQL per input;
  `batched` guarded against dynamic IN.
- Tests: `CustomQuerySpec` "in / notIn (OXY-17)" (non-empty/empty/single/large,
  NOT IN, composition, delete) + compiled queries in queries.scala.
- Docs: queries.md DSL vocabulary + IN/NOT IN section cross-linked to OXY-6.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…<> ALL(?))

Owner feedback: oxygen-sql aims for 100% STATIC SQL strings, but the first `IN`
cut emitted a runtime-variable number of `?` placeholders (sentinel token +
per-execution `InClause` expander that rewrote `ctx.sql` on every run).

Rework: bind the whole collection as ONE `java.sql.Array` parameter and emit a
constant string -- `col = ANY(?)` for `in`, `col <> ALL(?)` for `notIn`. The `?`
count is always 1, independent of list size, so `ctx.sql` is static again. No
sentinel, no per-exec SQL rewrite; `batched` now works with IN.

Reuses OXY-6/OXY-18 array machinery (`RowRepr.ArrayRepr` / `ArraySeqEncoder` /
`InputWriter.unsafeWriteArray`); the runtime `Seq[A]` is adapted via
`ArraySeq.untagged.from`. `= ANY` / `<> ALL` reproduce the exact 3-valued truth
table of `IN` / `NOT IN`, including empty-collection (`= ANY('{}')` -> FALSE,
`<> ALL('{}')` -> TRUE) and the NOT-IN-with-NULL footgun (unreachable here: array
columns are non-nullable, optional value-lists rejected). This converges `in` /
`notIn` with OXY-6's `= ANY(?)` -- flagged in report for human reconciliation.

Removed: query/InClause.scala, generation/GeneratedInClause.scala,
InputEncoder.SeqEncoder, GeneratedFragment's inClauses channel, QueryI/QueryIO
inClauses+execCtx, ParsedQuery makeQuery inClauses threading.

Tests: same behavioral matrix (non-empty/single/empty/large/NOT IN/composition/
delete/multi-input) + static-SQL assertions (`= ANY(?)`, no `IN (`, `<> ALL(?)`).
Full CustomQuerySpec 13/13 green on real Postgres.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011YxWKdsz97QT9BD7AdpSq6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant