Skip to content

Support temporary functions in stored queries#4291

Open
sergei-pustovykh wants to merge 57 commits into
FoundationDB:mainfrom
sergei-pustovykh:stored-queries-with-temp-funcs
Open

Support temporary functions in stored queries#4291
sergei-pustovykh wants to merge 57 commits into
FoundationDB:mainfrom
sergei-pustovykh:stored-queries-with-temp-funcs

Conversation

@sergei-pustovykh

@sergei-pustovykh sergei-pustovykh commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

This PR extends the stored-queries concept introduced previously: a stored query can now declare the temporary functions that its SELECT body depends on. The DDL syntax, the metadata shape, and the offline plan generation path all grow accordingly.

New DDL clause

CREATE QUERY now accepts zero or more WITH CREATE TEMPORARY FUNCTION ... clauses between the query name and the AS <select> body. Each clause uses the existing CREATE TEMPORARY FUNCTION syntax — same shape as a standalone temp-function DDL, including [OR REPLACE]? and the ON COMMIT DROP FUNCTION keywords.

CREATE QUERY top_orders
    WITH CREATE TEMPORARY FUNCTION half(in x bigint) ON COMMIT DROP FUNCTION
        AS SELECT * FROM orders WHERE col1 < x / 2
    WITH CREATE OR REPLACE TEMPORARY FUNCTION big(in x bigint) ON COMMIT DROP FUNCTION
        AS SELECT * FROM half(x) WHERE col2 > 100
AS
    SELECT id FROM big(200)

Each temp-function source is captured verbatim (whitespace and casing preserved) and persisted alongside the SELECT body.

Metadata shape change

The previous PR stored Map<String, String> (name -> SQL). This PR replaces that with Map<String, StoredQuery> where StoredQuery is a small immutable struct holding the SELECT body and an ordered List of CREATE TEMPORARY FUNCTION ... source strings. There are two parallel StoredQuery types - one in fdb-record-layer-core (RecordMetaData.StoredQuery) and one in fdb-relational-api - for the same module-layering reason as View, Index, and other metadata pairs. The relational-core serializer translates between them.
On the wire, temp functions are added as repeated string temp_function to PStoredQuery message:

message PStoredQuery {
  optional string name = 1;
  optional string query = 2;
  repeated string temp_function = 3;
}

Offline plan generation for stored queries with temp functions

OfflineStoredQueriesProcessor iterates through each stored query's WITH ... CREATE TEMPORARY FUNCTION ... clauses and generates a plan for each one using a custom MetadataOperationsFactory. The factory returns an empty (no-op) ConstantAction. Instead, it rebuilds local schema template from the RecordLayerInvokedRoutine that DdlVisitor hands it during plan generation. We do not execute the resulting procedure plan. All template modifications happen at plan-generation time.
Once all temp functions have been processed this way, the processor plans the stored query's SELECT body against the amended template and inserts the resulting plan into RelationalPlanCache.

@sergei-pustovykh sergei-pustovykh added the enhancement New feature or request label Jun 19, 2026
@sergei-pustovykh sergei-pustovykh marked this pull request as ready for review June 23, 2026 15:25
@sergei-pustovykh sergei-pustovykh requested a review from hatyo June 23, 2026 15:40
…emp-funcs

# Conflicts:
#	fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/RecordMetaData.java
#	fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/RecordMetaDataBuilder.java
#	fdb-record-layer-core/src/main/proto/record_metadata.proto
#	fdb-relational-api/src/main/java/com/apple/foundationdb/relational/api/metadata/SchemaTemplate.java
#	fdb-relational-core/src/main/java/com/apple/foundationdb/relational/recordlayer/metadata/NoOpSchemaTemplate.java
#	fdb-relational-core/src/main/java/com/apple/foundationdb/relational/recordlayer/query/OfflineStoredQueriesProcessor.java
#	fdb-relational-core/src/main/java/com/apple/foundationdb/relational/recordlayer/query/PlanGenerator.java
#	yaml-tests/src/test/resources/schema-template-stored-queries.yamsql
…emp-funcs

# Conflicts:
#	fdb-relational-core/src/main/java/com/apple/foundationdb/relational/recordlayer/query/PlanGenerator.java

@hatyo hatyo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two minor comments, LGTM.

@hatyo hatyo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retracting accidental approval, need to review first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants