From f4827cf47f29b5a209ffa44b522323cfe072b675 Mon Sep 17 00:00:00 2001 From: Kalin Rudnicki Date: Fri, 14 Aug 2026 22:04:04 -0600 Subject: [PATCH] OXY-6: array/set input bound as a single param (= ANY(?) / UNNEST(?)) Add first-class collection inputs to the sql query DSL, bound as a single `java.sql.Array` param instead of expanding to N placeholders (avoids the JDBC ~32767-param limit and plan-cache blowup). - `input.array[A]` (a `Seq[A]`) and `input.set[A]` (a `Set[A]`) + `arr.contains(col)` generate `col = ANY(?)`. - `select.unnest(arr)` uses the collection as a `FROM UNNEST(?::t[]) a(a)` table source (one row per element), aliased/qualified so it can be joined/filtered/selected. - `RowRepr.seqRepr` / `RowRepr.setRepr` wrap `RowRepr[A]` into `RowRepr[Seq[A]]` / `RowRepr[Set[A]]` (single Array column via `ArraySeqEncoder`); the runtime collection is adapted to a JDBC array at bind time, so callers pass their collection directly. - single-column element types only; array/set inputs are non-optional. - docs + it-test coverage (empty / single / large / NOT-found / composition / Set). Report captured on the OXY-6 Jira issue. Co-Authored-By: Claude Opus 4.8 --- docs/docs/sql/queries.md | 71 +++++++++++++++++ .../generic/generation/DecoderBuilder.scala | 11 +-- .../generic/generation/FragmentBuilder.scala | 45 +++++++++++ .../sql/generic/model/ParsedQuery.scala | 2 + .../oxygen/sql/generic/model/QueryExpr.scala | 26 ++++++- .../sql/generic/model/TypeclassExpr.scala | 16 ++++ .../sql/generic/model/VariableReference.scala | 22 ++++++ .../sql/generic/model/part/InputPart.scala | 2 + .../sql/generic/model/part/SelectPart.scala | 50 +++++++++++- .../sql/generic/parsing/RawQueryExpr.scala | 30 +++++++ .../main/scala/oxygen/sql/query/dsl/Q.scala | 21 +++++ .../main/scala/oxygen/sql/query/dsl/T.scala | 35 +++++++++ .../scala/oxygen/sql/schema/RowRepr.scala | 16 ++++ .../scala/oxygen/sql/CustomQuerySpec.scala | 78 +++++++++++++++++++ .../src/test/scala/oxygen/sql/queries.scala | 77 ++++++++++++++++++ 15 files changed, 494 insertions(+), 8 deletions(-) diff --git a/docs/docs/sql/queries.md b/docs/docs/sql/queries.md index bbac3f14..b516fb60 100644 --- a/docs/docs/sql/queries.md +++ b/docs/docs/sql/queries.md @@ -118,6 +118,8 @@ input makes it a `QueryO`/`Query`. Pass `debug = true` (`@compile(debug = true)` | Form | Purpose | |------|---------| | `input[I]` / `input.optional[I]` / `input.const(i)` | bind a runtime / optional / compile-time-constant parameter | +| `input.array[I]` / `input.set[I]` + `ids.contains(col)` | bind a whole `Seq` / `Set` as one array parameter, expanded to `col = ANY(?)` | +| `select.unnest(ids)` | use a collection input as a `UNNEST(?)` join/table source (one row per element) | | `select[A]` | select all columns of table `A` | | `join[A] if ` / `leftJoin[A] if ` | inner / left join (`leftJoin` yields `Option[A]`) | | `where if ` | filter | @@ -143,6 +145,75 @@ val personJoinNotes: QueryIO[UUID, (Person, Note)] = > Custom column types flow through automatically: `input[Email]` and `select[UserRow]` use the > `RowRepr`/encoder/decoder for `Email` you defined in [Models](models.md). +### Array input (`= ANY(?)`) + +To filter against a collection of values, bind the whole collection as a **single** array parameter +with `input.array[I]` (a `Seq[I]`) or `input.set[I]` (a `Set[I]`) and test membership with +`ids.contains(col)`. This generates Postgres `col = ANY(?)` — one bind parameter carrying a +`java.sql.Array`, instead of expanding an `IN (…)` list to N placeholders (which hits the JDBC +~32767-param limit and blows up the plan cache). + +```scala +@compile +val selectByIdArray: QueryIO[Seq[UUID], Person] = + for { + ids <- input.array[UUID] + p <- select[Person] + _ <- where if ids.contains(p.id) + } yield p +``` + +```scala +selectByIdArray.execute(Seq(id1, id2, id3)) +``` + +Notes: + +- `input.array[I]` takes a `Seq[I]`, `input.set[I]` takes a `Set[I]`; either is bound as one JDBC + array, so callers pass their collection directly (no `ArraySeq.from(…)` conversion needed). +- An empty collection yields `col = ANY('{}')`, i.e. no matches. +- `I` must be a single-column type (e.g. `UUID`, `Long`, `String`, or a `RowRepr` newtype over one). + Composite/multi-column element types are not supported yet. +- Array inputs compose with other inputs, e.g. `where if ids.contains(p.id) && p.groupId == groupId`. +- No explicit `::type[]` cast is emitted: the JDBC driver builds a typed array via + `createArrayOf(, …)`, so `= ANY(?)` already knows the element type. + +### Array input as a join table (`UNNEST(?)`) + +The primary way to use a collection input is as a **table source** you can join against: `select.unnest(ids)` +turns an `input.array[I]` / `input.set[I]` collection into a `UNNEST(?)` from-item that yields one row +per element, aliased so it can be joined/filtered/selected like any other table. The whole collection +still binds as a single `java.sql.Array` parameter. + +```scala +@compile +val notesByPersonIdUnnest: QueryIO[Seq[UUID], Note] = + for { + ids <- input.array[UUID] + id <- select.unnest(ids) // FROM UNNEST(?::uuid[]) id(id) + n <- join[Note] if n.personId == id + } yield n +``` + +generates (roughly): + +```sql +SELECT n.id, n.person_id, n.note + FROM UNNEST(?::uuid[]) id(id) + JOIN note n ON n.person_id = id.id +``` + +Notes: + +- The unnested element is a normal query variable — reference it in the join/where (`n.personId == id`) + or select it (`yield (id, n)`). It is emitted as a named, qualified column (`id.id`) so it never + collides with a same-named column of a joined table. +- An empty array produces zero rows (the join matches nothing). +- `I` must be a single-column type, same restriction as `input.array` / `input.set`. +- Composes with other inputs/joins/wheres, e.g. an extra `input[String]` used in a `where`. +- Currently supported as the root `FROM` source; `JOIN UNNEST(?)` (unnest as a non-root join item) + is not yet supported. + ## A real repo method ```scala diff --git a/modules/sql/core/src/main/scala/oxygen/sql/generic/generation/DecoderBuilder.scala b/modules/sql/core/src/main/scala/oxygen/sql/generic/generation/DecoderBuilder.scala index d4524f0f..32919dee 100644 --- a/modules/sql/core/src/main/scala/oxygen/sql/generic/generation/DecoderBuilder.scala +++ b/modules/sql/core/src/main/scala/oxygen/sql/generic/generation/DecoderBuilder.scala @@ -16,11 +16,12 @@ final class DecoderBuilder { case (queryExpr: QueryExpr.ConstValue, Some(parentContext)) => convert.const(queryExpr, parentContext) case (queryExpr: QueryExpr.InputVariableReferenceLike, Some(parentContext)) => convert.input(queryExpr, parentContext) case (queryExpr: QueryExpr.QueryVariableReferenceLike, _) => convert.query(queryExpr) - case (queryExpr: QueryExpr.Binary, _) => convert.binary(queryExpr) - case (queryExpr: QueryExpr.BuiltIn, _) => convert.builtIn(queryExpr) - case (queryExpr: QueryExpr.Composite, _) => convert.composite(queryExpr, parentContext) - case (queryExpr: QueryExpr.ConstValue, None) => ParseResult.error(queryExpr.fullTerm, "No RowRepr to compare with") - case (queryExpr: QueryExpr.InputVariableReferenceLike, None) => ParseResult.error(queryExpr.fullTerm, "No RowRepr to compare with") + case (_: QueryExpr.ArrayContains, _) => ParseResult.success(GeneratedResultDecoder.single(TypeclassExpr.RowRepr.boolean.resultDecoder, TypeRepr.of[Boolean])) + case (queryExpr: QueryExpr.Binary, _) => convert.binary(queryExpr) + case (queryExpr: QueryExpr.BuiltIn, _) => convert.builtIn(queryExpr) + case (queryExpr: QueryExpr.Composite, _) => convert.composite(queryExpr, parentContext) + case (queryExpr: QueryExpr.ConstValue, None) => ParseResult.error(queryExpr.fullTerm, "No RowRepr to compare with") + case (queryExpr: QueryExpr.InputVariableReferenceLike, None) => ParseResult.error(queryExpr.fullTerm, "No RowRepr to compare with") def const(queryExpr: QueryExpr.ConstValue, parentContext: TypeclassExpr.RowRepr): ParseResult[GeneratedResultDecoder] = ParseResult.success(GeneratedResultDecoder.single(parentContext.resultDecoder, queryExpr.fullTerm.tpe.widen)) diff --git a/modules/sql/core/src/main/scala/oxygen/sql/generic/generation/FragmentBuilder.scala b/modules/sql/core/src/main/scala/oxygen/sql/generic/generation/FragmentBuilder.scala index f7f412e9..a4ffb87d 100644 --- a/modules/sql/core/src/main/scala/oxygen/sql/generic/generation/FragmentBuilder.scala +++ b/modules/sql/core/src/main/scala/oxygen/sql/generic/generation/FragmentBuilder.scala @@ -18,6 +18,7 @@ final case class FragmentBuilder(inputs: List[InputPart])(using Quotes) { private val nonConstInputParams: List[VariableReference.NonConstInput] = inputs.map(_.mapQueryRef).flatMap { case p: VariableReference.FromInput => p.some + case p: VariableReference.ArrayFromInput => p.some case p: VariableReference.OptionalFromInput => p.some case _: VariableReference.FromConstInput => None } @@ -60,6 +61,11 @@ final case class FragmentBuilder(inputs: List[InputPart])(using Quotes) { case (Some(inputTpeTupGen), VariableReference.FromInput(param)) => val idx: Int = symIdxMap.getOrElse(param.sym, report.errorAndAbort("sym not found?", param.tree.pos)) InputRepr.NonConst(TermTransformer.FromProductGenericField(inputTpeTupGen, inputTpeTupGen.fields(idx)), false) + case (None, VariableReference.ArrayFromInput(_, _)) => + InputRepr.NonConst(TermTransformer.Id, false) + case (Some(inputTpeTupGen), VariableReference.ArrayFromInput(param, _)) => + val idx: Int = symIdxMap.getOrElse(param.sym, report.errorAndAbort("sym not found?", param.tree.pos)) + InputRepr.NonConst(TermTransformer.FromProductGenericField(inputTpeTupGen, inputTpeTupGen.fields(idx)), false) case (None, VariableReference.OptionalFromInput(_)) => InputRepr.NonConst(TermTransformer.Id, true) case (Some(inputTpeTupGen), VariableReference.OptionalFromInput(param)) => @@ -83,6 +89,7 @@ final case class FragmentBuilder(inputs: List[InputPart])(using Quotes) { case (queryExpr: QueryExpr.ConstValue, Some(parentContext)) => queryExprToFragment.constOrInput(queryExpr, parentContext) case (queryExpr: QueryExpr.InputVariableReferenceLike, Some(parentContext)) => queryExprToFragment.constOrInput(queryExpr, parentContext) case (queryExpr: QueryExpr.QueryVariableReferenceLike, _) => queryExprToFragment.query(queryExpr) + case (queryExpr: QueryExpr.ArrayContains, _) => queryExprToFragment.arrayContains(queryExpr) case (queryExpr: QueryExpr.Binary, _) => queryExprToFragment.binary(queryExpr) case (queryExpr: QueryExpr.BuiltIn, _) => queryExprToFragment.builtIn(queryExpr) case (queryExpr: QueryExpr.Composite, _) => queryExprToFragment.composite(queryExpr, parentContext) @@ -220,6 +227,17 @@ final case class FragmentBuilder(inputs: List[InputPart])(using Quotes) { } yield GeneratedFragment.of(lhsFrag, op.sqlPadded, rhsFrag) } + def arrayContains(queryExpr: QueryExpr.ArrayContains)(using ParseContext, GenerationContext, Quotes): ParseResult[GeneratedFragment] = + for { + colFrag <- queryExprToFragment.query(queryExpr.queryCol) + // wrap the compared column's `RowRepr[A]` into `RowRepr[Seq[A]]` / `RowRepr[Set[A]]`, so the + // whole collection binds as a single `?` param (via `ArraySeqEncoder` -> `java.sql.Array`). + arrRowRepr <- collectionAsArrayRepr(queryExpr.arrInput, queryExpr.queryCol.rowRepr) + inputFrag <- GenerationContext.updated(input = GenerationContext.Parens.Never) { + queryExprToFragment.input(queryExpr.arrInput, arrRowRepr) + } + } yield GeneratedFragment.of(colFrag, " = ANY(", inputFrag, ")") + def builtIn(queryExpr: QueryExpr.BuiltIn)(using ParseContext, GenerationContext, Quotes): ParseResult[GeneratedFragment] = queryExpr match case QueryExpr.Static(_, out, _) => ParseResult.Success(GeneratedFragment.sql(out)) @@ -306,6 +324,33 @@ final case class FragmentBuilder(inputs: List[InputPart])(using Quotes) { s"\n ) AS ${s.subQueryTableName}", ) + /** Wrap an element `RowRepr[A]` into the collection `RowRepr` matching how `input` was declared (`array` -> `Seq`, `set` -> `Set`). */ + private def collectionAsArrayRepr( + input: QueryExpr.InputVariableReferenceLike, + elemRowRepr: TypeclassExpr.RowRepr, + )(using ParseContext): ParseResult[TypeclassExpr.RowRepr] = + input.queryRef match + case VariableReference.ArrayFromInput(_, kind) => ParseResult.success(elemRowRepr.collectionAsArray(kind)) + case _ => ParseResult.error(input.fullTerm, "expected an `input.array` / `input.set` collection") + + def select(s: SelectPart.FromUnnest)(using ParseContext, GenerationContext, Quotes): ParseResult[GeneratedFragment] = + for { + // bind the referenced collection input as a single `?` param via `RowRepr[Seq[A]]` / `RowRepr[Set[A]]` (`ArraySeqEncoder`) + arrRowRepr <- collectionAsArrayRepr(s.arrInput, s.elemRowRepr) + inputFrag <- GenerationContext.updated(input = GenerationContext.Parens.Never) { + queryExprToFragment.input(s.arrInput, arrRowRepr) + } + // cast `?::[]` so postgres knows the array/element type of the table source, then + // `()` names the single output column so it can be referenced as `.`. + alias = s.mapQueryRef.sqlString + castType: Expr[String] = '{ "::" + ${ s.elemRowRepr.expr }.columns.columns.head.columnType.baseType + "[]" } + } yield GeneratedFragment.of( + "\n FROM UNNEST(", + inputFrag, + GeneratedFragment.sql(castType), + s") $alias($alias)", + ) + def update(u: UpdatePart)(using Quotes): ParseResult[GeneratedFragment] = ParseResult.success( GeneratedFragment.of( diff --git a/modules/sql/core/src/main/scala/oxygen/sql/generic/model/ParsedQuery.scala b/modules/sql/core/src/main/scala/oxygen/sql/generic/model/ParsedQuery.scala index f8c7f519..f2cf2118 100644 --- a/modules/sql/core/src/main/scala/oxygen/sql/generic/model/ParsedQuery.scala +++ b/modules/sql/core/src/main/scala/oxygen/sql/generic/model/ParsedQuery.scala @@ -185,6 +185,7 @@ private[sql] object ParsedQuery extends Parser[Term, ParsedQuery] { select match { case select: SelectPart.FromTable => Growable(select.mapQueryRef) case select: SelectPart.FromSubQuery => Growable.many(select.mapQueryRefs) + case select: SelectPart.FromUnnest => select.queryRefs }, Growable.many(joins).flatMap(_.queryRefs), Growable.option(where).flatMap(_.filterExpr.queryRefs), @@ -201,6 +202,7 @@ private[sql] object ParsedQuery extends Parser[Term, ParsedQuery] { selectFrag <- select match case select: SelectPart.FromTable => fragmentBuilder.select(select) case select: SelectPart.FromSubQuery => fragmentBuilder.select(select) + case select: SelectPart.FromUnnest => fragmentBuilder.select(select) joinFrag <- joins.traverse(fragmentBuilder.join).map(GeneratedFragment.flatten(_)) whereFrag <- where.traverse(fragmentBuilder.where).map(GeneratedFragment.option) orderByFrag <- orderBy.traverse(fragmentBuilder.orderBy).map(GeneratedFragment.option) diff --git a/modules/sql/core/src/main/scala/oxygen/sql/generic/model/QueryExpr.scala b/modules/sql/core/src/main/scala/oxygen/sql/generic/model/QueryExpr.scala index 46200958..d851b611 100644 --- a/modules/sql/core/src/main/scala/oxygen/sql/generic/model/QueryExpr.scala +++ b/modules/sql/core/src/main/scala/oxygen/sql/generic/model/QueryExpr.scala @@ -35,6 +35,7 @@ private[generic] sealed trait QueryExpr { this match { case unary: QueryExpr.QueryVariableReferenceLike => ParseResult.success(unary.rowRepr) case _: QueryExpr.Binary => ParseResult.success(TypeclassExpr.RowRepr.boolean) + case _: QueryExpr.ArrayContains => ParseResult.success(TypeclassExpr.RowRepr.boolean) case _ => ParseResult.error(fullTerm, "Unable to extract RowRepr") } @@ -297,6 +298,22 @@ private[generic] object QueryExpr extends Parser[RawQueryExpr, QueryExpr] { rhs: QueryExpr, ) extends Binary + ////////////////////////////////////////////////////////////////////////////////////////////////////// + // ArrayContains + ////////////////////////////////////////////////////////////////////////////////////////////////////// + + /** + * Represents `arr.contains(col)`, generated as `col = ANY(?)` where `?` binds the whole array. + */ + final case class ArrayContains( + fullTerm: Term, + queryCol: QueryExpr.QueryVariableReferenceLike, + arrInput: QueryExpr.InputVariableReferenceLike, + ) extends QueryExpr { + override def queryRefs: Growable[VariableReference] = queryCol.queryRefs ++ arrInput.queryRefs + override def show(using Quotes): String = s"${arrInput.show}.contains(${queryCol.show})" + } + ////////////////////////////////////////////////////////////////////////////////////////////////////// // BuiltIn ////////////////////////////////////////////////////////////////////////////////////////////////////// @@ -368,8 +385,13 @@ private[generic] object QueryExpr extends Parser[RawQueryExpr, QueryExpr] { override def parse(expr: RawQueryExpr)(using ParseContext, Quotes): ParseResult[QueryExpr] = expr match { - case expr: RawQueryExpr.VariableReferenceLike => VariableReferenceLike.parse(expr) - case expr: RawQueryExpr.Binary => Binary.parse(expr) + case expr: RawQueryExpr.VariableReferenceLike => VariableReferenceLike.parse(expr) + case expr: RawQueryExpr.Binary => Binary.parse(expr) + case RawQueryExpr.ArrayContains(fullTerm, arr, elem) => + for { + arrInput <- QueryExpr.InputVariableReferenceLike.parse(arr) + queryCol <- QueryExpr.QueryVariableReferenceLike.parse(elem) + } yield QueryExpr.ArrayContains(fullTerm, queryCol, arrInput) case RawQueryExpr.InstantiateTable(fullTerm, gen, givenTableRepr, args) => args.traverse(parse).map(QueryExpr.InstantiateTable(fullTerm, gen, givenTableRepr, _)) case RawQueryExpr.RandomUUID(fullTerm) => ParseResult.success(QueryExpr.Static(fullTerm, "gen_random_uuid()", TypeclassExpr.RowRepr.uuid)) case RawQueryExpr.InstantNow(fullTerm) => ParseResult.success(QueryExpr.Static(fullTerm, "NOW()", TypeclassExpr.RowRepr.instant)) diff --git a/modules/sql/core/src/main/scala/oxygen/sql/generic/model/TypeclassExpr.scala b/modules/sql/core/src/main/scala/oxygen/sql/generic/model/TypeclassExpr.scala index 5e314d7c..dfdc09ee 100644 --- a/modules/sql/core/src/main/scala/oxygen/sql/generic/model/TypeclassExpr.scala +++ b/modules/sql/core/src/main/scala/oxygen/sql/generic/model/TypeclassExpr.scala @@ -54,6 +54,22 @@ object TypeclassExpr { def optional: TypeclassExpr.RowRepr = TypeclassExpr.RowRepr { '{ $expr.optional } } + /** Names this schema's (single) column, so an otherwise-unnamed scalar can be referenced as `alias.name`. */ + def prefixedInline(prefix: String): TypeclassExpr.RowRepr = + TypeclassExpr.RowRepr { '{ $expr.prefixedInline(${ Expr(prefix) }) } } + + /** + * Wraps this element `RowRepr[A]` into a `RowRepr[Seq[A]]` / `RowRepr[Set[A]]` (see + * [[oxygen.sql.schema.RowRepr.seqRepr]] / [[oxygen.sql.schema.RowRepr.setRepr]]), so the whole + * collection binds as one Array-typed `?` param. + */ + def collectionAsArray(kind: VariableReference.ArrayInputKind): TypeclassExpr.RowRepr = + TypeclassExpr.RowRepr { + kind match + case VariableReference.ArrayInputKind.Seq => '{ $expr.seqRepr } + case VariableReference.ArrayInputKind.Set => '{ $expr.setRepr } + } + def productSchemaField(term: Term, field: String): TypeclassExpr.RowRepr = TypeclassExpr.RowRepr { type T diff --git a/modules/sql/core/src/main/scala/oxygen/sql/generic/model/VariableReference.scala b/modules/sql/core/src/main/scala/oxygen/sql/generic/model/VariableReference.scala index 8085ec70..9529708f 100644 --- a/modules/sql/core/src/main/scala/oxygen/sql/generic/model/VariableReference.scala +++ b/modules/sql/core/src/main/scala/oxygen/sql/generic/model/VariableReference.scala @@ -15,6 +15,7 @@ private[generic] sealed trait VariableReference { final def show: String = this match case VariableReference.FromInput(param) => param.name.greenFg.toString + case VariableReference.ArrayFromInput(param, kind) => s"${kind.show}(${param.name.greenFg})" case VariableReference.OptionalFromInput(param) => s"optional(${param.name.greenFg})" case VariableReference.FromConstInput(param, term, _) => s"const(${param.name.greenFg} = ${term.showAnsiCode})" case VariableReference.FromQuery(_, _, _, true, sqlString) => sqlString.hexFg("#7EB77F").toString @@ -37,6 +38,27 @@ private[generic] object VariableReference { override val nonConstInputType: TypeRepr = param.tpe } + /** The collection kind of an [[ArrayFromInput]] -- both bind as a single `java.sql.Array` param. */ + enum ArrayInputKind { + case Seq, Set + def show: String = this match + case ArrayInputKind.Seq => "array" + case ArrayInputKind.Set => "set" + } + + /** + * A collection input bound as a single `?` param (`java.sql.Array`). + * `param.tpe` is the user-facing collection type -- `Seq[A]` (`kind = Seq`) or `Set[A]` + * (`kind = Set`) -- so the query input type matches what the user declared; it is converted to a + * JDBC array at bind time. + */ + final case class ArrayFromInput( + param: Function.NamedParam, + kind: ArrayInputKind, + ) extends NonConstInput { + override val nonConstInputType: TypeRepr = param.tpe + } + final case class OptionalFromInput( param: Function.NamedParam, ) extends NonConstInput { diff --git a/modules/sql/core/src/main/scala/oxygen/sql/generic/model/part/InputPart.scala b/modules/sql/core/src/main/scala/oxygen/sql/generic/model/part/InputPart.scala index 3343ca0b..3a87054e 100644 --- a/modules/sql/core/src/main/scala/oxygen/sql/generic/model/part/InputPart.scala +++ b/modules/sql/core/src/main/scala/oxygen/sql/generic/model/part/InputPart.scala @@ -21,6 +21,8 @@ object InputPart extends MapChainParser[InputPart] { for { (mapAAFC, input) <- AppliedAnonFunctCall.parseTyped[T.InputLike](term, "map function").parseLhsAndFunct { case ('{ Q.input.apply[a] }, mapAAFC) => mapAAFC.funct.parseParam1.map { mapParam => InputPart(VariableReference.FromInput(mapParam)) } + case ('{ Q.input.array[a] }, mapAAFC) => mapAAFC.funct.parseParam1.map { mapParam => InputPart(VariableReference.ArrayFromInput(mapParam, VariableReference.ArrayInputKind.Seq)) } + case ('{ Q.input.set[a] }, mapAAFC) => mapAAFC.funct.parseParam1.map { mapParam => InputPart(VariableReference.ArrayFromInput(mapParam, VariableReference.ArrayInputKind.Set)) } case ('{ Q.input.optional[a] }, mapAAFC) => mapAAFC.funct.parseParam1.map { mapParam => InputPart(VariableReference.OptionalFromInput(mapParam)) } case ('{ Q.input.const[a](${ expr }) }, mapAAFC) => mapAAFC.funct.parseParam1.map { mapParam => InputPart(VariableReference.FromConstInput(mapParam, expr.toTerm, TypeRepr.of[Any])) } } diff --git a/modules/sql/core/src/main/scala/oxygen/sql/generic/model/part/SelectPart.scala b/modules/sql/core/src/main/scala/oxygen/sql/generic/model/part/SelectPart.scala index 764fb6b3..bcbf1fa3 100644 --- a/modules/sql/core/src/main/scala/oxygen/sql/generic/model/part/SelectPart.scala +++ b/modules/sql/core/src/main/scala/oxygen/sql/generic/model/part/SelectPart.scala @@ -14,6 +14,7 @@ sealed trait SelectPart { final def optTableRepr: Option[TypeclassExpr.TableRepr] = this match case SelectPart.FromTable(_, tableRepr) => tableRepr.some case _: SelectPart.FromSubQuery => None + case _: SelectPart.FromUnnest => None } object SelectPart extends MapChainParser.Deferred[SelectPart] { @@ -87,6 +88,53 @@ object SelectPart extends MapChainParser.Deferred[SelectPart] { } - override lazy val deferTo: MapChainParser[SelectPart] = FromTable || FromSubQuery + ////////////////////////////////////////////////////////////////////////////////////////////////////// + // FromUnnest + ////////////////////////////////////////////////////////////////////////////////////////////////////// + + /** + * `FROM UNNEST(?::type[]) alias` : uses an array/collection input (`arrInput`) as a table source + * that yields one row per element. The element is exposed as the query var `mapQueryRef` + * (a single, alias-named column) so it can be joined/filtered/selected against. + */ + final case class FromUnnest( + mapQueryRef: VariableReference.FromQuery, + arrInput: QueryExpr.InputVariableReferenceLike, + elemRowRepr: TypeclassExpr.RowRepr, + ) extends SelectPart { + + override def show(using Quotes): String = + s"FROM UNNEST(${arrInput.show}) ${mapQueryRef.show}" + + def queryRefs: Growable[VariableReference] = + mapQueryRef +: arrInput.queryRefs + + } + object FromUnnest extends MapChainParser[FromUnnest] { + + override def parse(term: Term, refs: RefMap, prevFunction: String)(using ParseContext, Quotes): MapChainParseResult[FromUnnest] = + for { + (mapAAFC, (elemRowRepr, arrTerm)) <- AppliedAnonFunctCall.parseTyped[T.SelectUnnest[?]](term, "map function").parseLhs { // + case '{ Q.select.unnest[a]($arr)(using $rowRepr) } => ParseResult.success((TypeclassExpr.RowRepr(rowRepr), arr.toTerm.underlyingArgument)) + } + mapParam <- mapAAFC.funct.parseParam1 + mapFunctName <- functionNames.mapOrFlatMap.parse(mapAAFC.nameRef).unknownAsError + + // resolve the array input that is being unnested (must be a previously-declared `input.array[_]`) + rawArr <- RawQueryExpr.VariableReferenceLike.parse((arrTerm, refs)).unknownAsError + arrInput <- QueryExpr.InputVariableReferenceLike.parse(rawArr) + + // the element becomes a query var; name its single column = the alias, so it is referenced as + // `alias.alias` (qualified), avoiding ambiguity with same-named columns of joined tables. + alias = mapParam.name.camelToSnake + namedElemRowRepr = elemRowRepr.prefixedInline(alias) + mapQueryRef = VariableReference.FromQuery(mapParam, namedElemRowRepr, true) + newRefs = refs.add(mapQueryRef) + + } yield MapChainResult(FromUnnest(mapQueryRef, arrInput, namedElemRowRepr), mapFunctName, newRefs, mapAAFC.appliedFunctionBody) + + } + + override lazy val deferTo: MapChainParser[SelectPart] = FromTable || FromSubQuery || FromUnnest } diff --git a/modules/sql/core/src/main/scala/oxygen/sql/generic/parsing/RawQueryExpr.scala b/modules/sql/core/src/main/scala/oxygen/sql/generic/parsing/RawQueryExpr.scala index 5a5c8266..af6723d1 100644 --- a/modules/sql/core/src/main/scala/oxygen/sql/generic/parsing/RawQueryExpr.scala +++ b/modules/sql/core/src/main/scala/oxygen/sql/generic/parsing/RawQueryExpr.scala @@ -35,6 +35,7 @@ private[generic] sealed trait RawQueryExpr { case RawQueryExpr.OptionNullability(_, inner, showScala, _) => s"${inner.show}.${showScala.hexFg("#35A7FF")}" case RawQueryExpr.SelectPrimaryKey(_, inner, _) => s"${inner.show}.${"tablePK".hexFg("#35A7FF")}" case RawQueryExpr.SelectNonPrimaryKey(_, inner, _) => s"${inner.show}.${"tableNPK".hexFg("#35A7FF")}" + case RawQueryExpr.ArrayContains(_, arr, elem) => s"${arr.show}.${"contains".magentaFg}(${elem.show})" case bin: RawQueryExpr.Binary if bin.lhs.isBin || bin.rhs.isBin => s"(${bin.lhs.show}) ${bin.op.show} (${bin.rhs.show})" case bin: RawQueryExpr.Binary => s"${bin.lhs.show} ${bin.op.show} ${bin.rhs.show}" case RawQueryExpr.InstantiateTable(_, gen, _, args) => args.map(_.show).mkString(s"${gen.typeRepr.showCode}.apply(", ", ", ")") @@ -166,6 +167,34 @@ private[generic] object RawQueryExpr extends Parser[(Term, RefMap), RawQueryExpr } + ////////////////////////////////////////////////////////////////////////////////////////////////////// + // ArrayContains + ////////////////////////////////////////////////////////////////////////////////////////////////////// + + final case class ArrayContains( + fullTerm: Term, + arr: RawQueryExpr.VariableReferenceLike, + elem: RawQueryExpr.VariableReferenceLike, + ) extends RawQueryExpr + object ArrayContains extends Parser[(Term, RefMap), ArrayContains] { + + override def parse(input: (Term, RefMap))(using ParseContext, Quotes): ParseResult[ArrayContains] = { + val (rootTerm, refs) = input + rootTerm match { + case singleApply(Select(lhs, "contains"), rhs) => + VariableReferenceLike.parse((lhs, refs)) match { + case ParseResult.Success(arr @ ReferencedVariable(_, _: VariableReference.ArrayFromInput)) => + VariableReferenceLike.parse((rhs, refs)).unknownAsError.map(ArrayContains(rootTerm, arr, _)) + case _ => + ParseResult.unknown(rootTerm, "not an array `.contains(_)`") + } + case _ => + ParseResult.unknown(rootTerm, "not an array `.contains(_)`") + } + } + + } + ////////////////////////////////////////////////////////////////////////////////////////////////////// // Binary ////////////////////////////////////////////////////////////////////////////////////////////////////// @@ -358,6 +387,7 @@ private[generic] object RawQueryExpr extends Parser[(Term, RefMap), RawQueryExpr override def parse(input: (Term, RefMap))(using ParseContext, Quotes): ParseResult[RawQueryExpr] = input match case ConstValue.optional(res) => res + case ArrayContains.optional(res) => res case ReferencedVariable.optional(res) => res case StaticCount.optional(res) => res case CountWithArg.optional(res) => res diff --git a/modules/sql/core/src/main/scala/oxygen/sql/query/dsl/Q.scala b/modules/sql/core/src/main/scala/oxygen/sql/query/dsl/Q.scala index e34aa6c5..af137b38 100644 --- a/modules/sql/core/src/main/scala/oxygen/sql/query/dsl/Q.scala +++ b/modules/sql/core/src/main/scala/oxygen/sql/query/dsl/Q.scala @@ -15,6 +15,12 @@ object Q { def const[I](i: I): T.ConstInput[I] = macroOnly + /** A `Seq` input bound as a single `?` param. Use `arr.contains(col)` for `col = ANY(?)`. */ + def array[I]: T.ArrayInput[I] = macroOnly + + /** A `Set` input bound as a single `?` param. Use `arr.contains(col)` for `col = ANY(?)`. */ + def set[I]: T.SetInput[I] = macroOnly + } object select { @@ -23,6 +29,21 @@ object Q { def subQuery[A](subQueryTableName: String)(q: QueryO[A]): T.SelectSubQuery[A] = macroOnly + /** + * Use an array/collection input as a `UNNEST(?)` table source (one row per element), + * aliased so it can be JOINed/filtered/selected against. + * + * {{{ + * for { + * ids <- input.array[UUID] + * id <- select.unnest(ids) + * n <- join[Note] if n.personId == id + * } yield n + * // SELECT ... FROM UNNEST(?::uuid[]) id JOIN note n ON n.person_id = id + * }}} + */ + def unnest[A](arr: Iterable[A])(using r: RowRepr[A]): T.SelectUnnest[A] = macroOnly + } object insert { diff --git a/modules/sql/core/src/main/scala/oxygen/sql/query/dsl/T.scala b/modules/sql/core/src/main/scala/oxygen/sql/query/dsl/T.scala index fd8f55a3..46b0ae86 100644 --- a/modules/sql/core/src/main/scala/oxygen/sql/query/dsl/T.scala +++ b/modules/sql/core/src/main/scala/oxygen/sql/query/dsl/T.scala @@ -32,6 +32,30 @@ object T { def flatMap[I2, O](f: I => QueryIO[I2, O]): QueryIO[I2, O] = macroOnly } + /** + * A `Seq` bound as a single `?` parameter (`java.sql.Array`). + * The body param has type `Seq[A]`; use `arr.contains(col)` to generate `col = ANY(?)`. + * `A` must be a single-column type. + */ + final class ArrayInput[A] private extends InputLike { + def flatMap(f: Seq[A] => Query): QueryI[Seq[A]] = macroOnly + def flatMap[I2](f: Seq[A] => QueryI[I2])(using zip: Zip[Seq[A], I2]): QueryI[zip.Out] = macroOnly + def flatMap[O](f: Seq[A] => QueryO[O]): QueryIO[Seq[A], O] = macroOnly + def flatMap[I2, O](f: Seq[A] => QueryIO[I2, O])(using zip: Zip[Seq[A], I2]): QueryIO[zip.Out, O] = macroOnly + } + + /** + * A `Set` bound as a single `?` parameter (`java.sql.Array`). + * The body param has type `Set[A]`; use `arr.contains(col)` to generate `col = ANY(?)`. + * `A` must be a single-column type. + */ + final class SetInput[A] private extends InputLike { + def flatMap(f: Set[A] => Query): QueryI[Set[A]] = macroOnly + def flatMap[I2](f: Set[A] => QueryI[I2])(using zip: Zip[Set[A], I2]): QueryI[zip.Out] = macroOnly + def flatMap[O](f: Set[A] => QueryO[O]): QueryIO[Set[A], O] = macroOnly + def flatMap[I2, O](f: Set[A] => QueryIO[I2, O])(using zip: Zip[Set[A], I2]): QueryIO[zip.Out, O] = macroOnly + } + ////////////////////////////////////////////////////////////////////////////////////////////////////// // CRUD ////////////////////////////////////////////////////////////////////////////////////////////////////// @@ -51,6 +75,17 @@ object T { def flatMap[B](f: A => Ret[B]): QueryO[B] = macroOnly } + /** + * A `UNNEST(?)` table source: the array/collection `arr` is bound as a single `?` param and + * expanded into one row per element, aliased so it can be joined/filtered/selected against. + * The body param has type `A` (the element type); the whole array binds as one `java.sql.Array`. + * Generates `FROM UNNEST(?::type[]) alias`. `A` must be a single-column type. + */ + final class SelectUnnest[A] private { + def map[B](f: A => B): QueryO[B] = macroOnly + def flatMap[B](f: A => Ret[B]): QueryO[B] = macroOnly + } + final class SelectSubQuery[A] private { def map[B](f: A => B): QueryO[B] = macroOnly def flatMap[B](f: A => Ret[B]): QueryO[B] = macroOnly diff --git a/modules/sql/core/src/main/scala/oxygen/sql/schema/RowRepr.scala b/modules/sql/core/src/main/scala/oxygen/sql/schema/RowRepr.scala index 177297de..f1e80b94 100644 --- a/modules/sql/core/src/main/scala/oxygen/sql/schema/RowRepr.scala +++ b/modules/sql/core/src/main/scala/oxygen/sql/schema/RowRepr.scala @@ -41,6 +41,22 @@ trait RowRepr[A] { final def optional: RowRepr[Option[A]] = RowRepr.OptionalRepr(this) + /** + * A `RowRepr` for a `Seq[A]` bound as a single Array-typed column (`ArraySeqEncoder` -> `java.sql.Array`). + * The runtime `Seq` is adapted to `ArraySeq` for encoding; decoding widens the `ArraySeq` back to `Seq`. + * `A` must be a single, non-nullable column (enforced by [[RowRepr.ArrayRepr]]). + */ + final def seqRepr: RowRepr[Seq[A]] = + RowRepr.ArrayRepr(this).transform(as => as, ArraySeq.untagged.from) + + /** + * A `RowRepr` for a `Set[A]` bound as a single Array-typed column (`ArraySeqEncoder` -> `java.sql.Array`). + * The runtime `Set` is adapted to `ArraySeq` for encoding; decoding collects the `ArraySeq` into a `Set`. + * `A` must be a single, non-nullable column (enforced by [[RowRepr.ArrayRepr]]). + */ + final def setRepr: RowRepr[Set[A]] = + RowRepr.ArrayRepr(this).transform(_.toSet, ArraySeq.untagged.from) + final def transform[B](ab: A => B, ba: B => A): RowRepr[B] = RowRepr.Transform(this, ab, ba) final def transformOrFail[B](ab: A => Either[String, B], ba: B => A): RowRepr[B] = RowRepr.TransformOrFail(this, ab, ba) diff --git a/modules/sql/it-test/src/test/scala/oxygen/sql/CustomQuerySpec.scala b/modules/sql/it-test/src/test/scala/oxygen/sql/CustomQuerySpec.scala index 97f2afa5..429cf175 100644 --- a/modules/sql/it-test/src/test/scala/oxygen/sql/CustomQuerySpec.scala +++ b/modules/sql/it-test/src/test/scala/oxygen/sql/CustomQuerySpec.scala @@ -321,6 +321,84 @@ object CustomQuerySpec extends OxygenSpec[Database] { res2 == Set(n3, n4), ) }, + test("array input (= ANY(?))") { + for { + groupId1 <- Random.nextUUID + groupId2 <- Random.nextUUID + p1s <- Person.generate(groupId1)().replicateZIO(5) + p2s <- Person.generate(groupId2)().replicateZIO(5) + + _ <- Person.insert.batched(p1s ++ p2s).unit + + all = p1s ++ p2s + // a plain `Seq` (not `ArraySeq`) is bound directly -- no conversion needed + someIds = (p1s.take(2) ++ p2s.take(1)).map(_.id).toSeq + + // basic: single array param expands to `= ANY(?)` + res1 <- queries.selectByIdArray(someIds).to[Set] + // `input.set` variant: same `= ANY(?)`, but the caller passes a `Set` + resSet <- queries.selectByIdSet((p1s.take(2) ++ p2s.take(1)).map(_.id).toSet).to[Set] + // empty collection -> `x = ANY('{}')` -> no matches + resEmpty <- queries.selectByIdArray(Seq.empty[java.util.UUID]).to[Set] + // large array bound as ONE param (no N-placeholder expansion / plan-cache blowup) + randomIds <- Random.nextUUID.replicateZIO(1000) + bigIds = all.map(_.id).toSeq ++ randomIds + resBig <- queries.selectByIdArray(bigIds).to[Set] + // composition with a scalar input + resGroup <- queries.selectByIdArrayAndGroup(all.map(_.id).toSeq, groupId1).to[Set] + } yield assertTrue( + res1 == (p1s.take(2) ++ p2s.take(1)).toSet, + resSet == (p1s.take(2) ++ p2s.take(1)).toSet, + resEmpty.isEmpty, + resBig == all.toSet, + resGroup == p1s.toSet, + ) + }, + test("UNNEST(?) as an input join table") { + for { + groupId <- Random.nextUUID + // 5 people, each with 2 notes + people <- Person.generate(groupId)().replicateZIO(5) + _ <- Person.insert.batched(people).unit + notes <- ZIO.foreach(people)(p => Note.generate(p.id)().replicateZIO(2)).map(_.flatten) + _ <- Note.insert.batched(notes).unit + + notesByPerson = notes.groupBy(_.personId) + + // basic: JOIN only returns notes for the person-ids present in the array + selectedPeople = people.take(2) + selectedIds = selectedPeople.map(_.id).toSeq + expectedNotes = selectedPeople.flatMap(p => notesByPerson.getOrElse(p.id, Nil)).toSet + + basic <- queries.notesByPersonIdUnnest(selectedIds).to[Set] + // UNNEST over a `Set` input yields the same rows + basicSet <- queries.notesByPersonIdUnnestSet(selectedPeople.map(_.id).toSet).to[Set] + + // the unnested column decodes + is selectable: (personId, note) + withId <- queries.personIdAndNoteUnnest(selectedIds).to[Set] + + // empty collection -> UNNEST('{}') -> zero rows + empty <- queries.notesByPersonIdUnnest(Seq.empty[java.util.UUID]).to[Set] + + // larger array (all real ids + 1000 random misses), still one bound param + randomIds <- Random.nextUUID.replicateZIO(1000) + bigIds = people.map(_.id).toSeq ++ randomIds + big <- queries.notesByPersonIdUnnest(bigIds).to[Set] + + // composition: UNNEST array input + an extra scalar input in a WHERE + targetNote = notes.head + composed <- queries.notesByPersonIdUnnestFilteredByNote((people.map(_.id).toSeq, targetNote.note)).to[Set] + } yield assertTrue( + basic == expectedNotes, + basicSet == expectedNotes, + withId == expectedNotes.map(n => (n.personId, n)), + empty.isEmpty, + big == notes.toSet, + // only notes matching both a person-id in the array AND the exact note text + composed == notes.filter(_.note == targetNote.note).toSet, + composed.nonEmpty, + ) + }, test("ltree") { def make(labels: String*): UIO[LTreeEx] = Random.nextUUID.map(LTreeEx(_, oxygen.sql.model.LTree(ArraySeq.from(labels)))) diff --git a/modules/sql/it-test/src/test/scala/oxygen/sql/queries.scala b/modules/sql/it-test/src/test/scala/oxygen/sql/queries.scala index bdd542fb..02203cf2 100644 --- a/modules/sql/it-test/src/test/scala/oxygen/sql/queries.scala +++ b/modules/sql/it-test/src/test/scala/oxygen/sql/queries.scala @@ -221,6 +221,83 @@ object queries { _ <- where if p1.groupId == i } yield p1 + @compile + val selectByIdArray: QueryIO[Seq[UUID], Person] = + for { + ids <- input.array[UUID] + p <- select[Person] + _ <- where if ids.contains(p.id) + } yield p + + // same as `selectByIdArray`, but the input is a `Set` (`input.set`) instead of a `Seq` + @compile + val selectByIdSet: QueryIO[Set[UUID], Person] = + for { + ids <- input.set[UUID] + p <- select[Person] + _ <- where if ids.contains(p.id) + } yield p + + @compile + val selectByIdArrayAndGroup: QueryIO[(Seq[UUID], UUID), Person] = + for { + ids <- input.array[UUID] + groupId <- input[UUID] + p <- select[Person] + _ <- where if ids.contains(p.id) && p.groupId == groupId + } yield p + + @compile + val personJoinNotesByIdArray: QueryIO[Seq[UUID], (Person, Note)] = + for { + ids <- input.array[UUID] + p <- select[Person] + n <- join[Note] if n.personId == p.id + _ <- where if ids.contains(p.id) + } yield (p, n) + + ////////////////////////////////////////////////////////////////////////////////////////////////////// + // UNNEST as an input JOIN table source + ////////////////////////////////////////////////////////////////////////////////////////////////////// + + // SELECT ... FROM UNNEST(?::uuid[]) id JOIN note n ON n.person_id = id + @compile + val notesByPersonIdUnnest: QueryIO[Seq[UUID], Note] = + for { + ids <- input.array[UUID] + id <- select.unnest(ids) + n <- join[Note] if n.personId == id + } yield n + + // UNNEST over a `Set` input (`input.set`) rather than a `Seq` + @compile + val notesByPersonIdUnnestSet: QueryIO[Set[UUID], Note] = + for { + ids <- input.set[UUID] + id <- select.unnest(ids) + n <- join[Note] if n.personId == id + } yield n + + // also selects the unnested column itself, to prove it decodes + can be referenced in the SELECT + @compile + val personIdAndNoteUnnest: QueryIO[Seq[UUID], (UUID, Note)] = + for { + ids <- input.array[UUID] + id <- select.unnest(ids) + n <- join[Note] if n.personId == id + } yield (id, n) + + // composition: UNNEST array input + an additional scalar input used in a WHERE + @compile + val notesByPersonIdUnnestFilteredByNote: QueryIO[(Seq[UUID], String), Note] = + for { + ids <- input.array[UUID] + noteText <- input[String] + id <- select.unnest(ids) + n <- join[Note] if n.personId == id + _ <- where if n.note == noteText + } yield n + @compile val othersWithSameLastNameAsId: QueryIO[UUID, Person] = for {