matching stats as join table by awildturtok · Pull Request #3835 · ingef/conquery

awildturtok · 2026-01-22T16:50:07Z

Eine Beispielhafte join-table könnte folgende Einträge enthalten:

                     resolved_id                      | col_val
------------------------------------------------------+----------
 database.icd.s00-t98.s00-s09.s00.s00_0.s00_00 | T612
 database.icd.s00-t98.s00-s09.s00.s00_0.s00_00 | T611
 database.icd.s00-t98.s00-s09.s00.s00_0.s00_00 | T610
 database.icd.j00-j99.j30-j39.j38.j38_0.j38_00 | J951

  select
    diag.id as "pid",
    least(
      'infinity'::date,
      lower("datum"),
      upper("datum")
    ) as "lower_bound",
    greatest(
      '-infinity'::date,
      lower("datum"),
      upper("datum")
    ) as "upper_bound",
    "resolved_id"
  from "diag"
    left outer join "icd_ids"
      on "diag"."icd_code" = "col_val"

im Falle von ColumnEquals wären dann noch weitere Spalten dabei. Bei PRESENT/EMPTY wären stattdessen true/false drin, die dann im join abgebildet werden.

# Conflicts: # backend/src/main/java/com/bakdata/conquery/mode/local/LocalStorageListener.java

… SqlMatchingStats easier

Also fixes a case, where missing entries were not registered to the root.

…ostgres

…tats-as-sql-function # Conflicts: # backend/src/main/java/com/bakdata/conquery/models/messages/namespaces/specific/UpdateMatchingStatsMessage.java

(experimental)

(this is a bit absurd tbh)

awildturtok · 2026-04-09T12:16:02Z

+@Data
+public abstract class StorageListener<T extends Namespace>{


Ich hatte glaube ich gehofft mehr logik zu vereinheitlichen, bin auch nicht deiner Meinung. Aber hier ist es definitiv die klasse wert.

thoniTUB

Ich muss nochmal weiter machen.

thoniTUB · 2026-04-10T07:30:12Z

Würde ich behalten

…able' into feature/matching-stats-as-join-table

thoniTUB · 2026-04-21T12:38:01Z

@@ -1,7 +1,7 @@
 {
  "type": "QUERY_TEST",
  "sqlSpec": {
-    "isEnabled": true
+    "isEnabled": false


wieder enablen?

thoniTUB · 2026-04-21T12:39:20Z

@@ -1,7 +1,7 @@
 {
  "type": "QUERY_TEST",
  "sqlSpec": {
-    "isEnabled": true
+    "isEnabled": false


awildturtok · 2026-06-02T12:58:24Z

 Taking the short example from above, we want to explain how we do this using the
 [AnsiSqlIntervalPacker](./AnsiSqlIntervalPacker.java). To outline, we need to
-create 3 consecutive common table expressions (CTE):
+create 3 consecutive common table conceptConditions (CTE):


Suggested change

create 3 consecutive common table conceptConditions (CTE):

create 3 consecutive common table expressions (CTE):

# Conflicts: # backend/src/main/java/com/bakdata/conquery/sql/conversion/cqelement/CQExternalConverter.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/cqelement/CQYesConverter.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/cqelement/concept/CQConceptConverter.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/dialect/SqlFunctionProvider.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/dialect/hana/HanaSqlFunctionProvider.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/dialect/pg/PostgreSqlFunctionProvider.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/forms/AbsoluteStratification.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/model/ColumnDateRange.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/model/SqlIdColumns.java # backend/src/main/java/com/bakdata/conquery/sql/conversion/model/StratificationSqlIdColumns.java # backend/src/main/java/com/bakdata/conquery/util/TablePrimaryColumnUtil.java # backend/src/test/java/com/bakdata/conquery/integration/json/SqlTestDataImporter.java

…able' into feature/matching-stats-as-join-table

awildturtok and others added 30 commits November 12, 2025 15:24

First draft of function to resolve concepts

74d9da4

# Conflicts: # backend/src/main/java/com/bakdata/conquery/mode/local/LocalStorageListener.java

remove qualification from CTConditionContext to make integration with…

e028841

… SqlMatchingStats easier

Collect Auxiliary columns to then generate proper signature

4cccd65

properly insert function (only postgres atm)

209d881

first draft towards applying matching stats

8620c65

Reworks registration of MatchingStats.

51f1aea

Also fixes a case, where missing entries were not registered to the root.

fixes usage in MatchingStatsTests.java

98b69c4

wip

f09cdaa

implements conversion and extraction of matching stats for Hana and P…

107407d

…ostgres

Merge remote-tracking branch 'origin/develop' into feature/matching-s…

1b4949f

…tats-as-sql-function # Conflicts: # backend/src/main/java/com/bakdata/conquery/models/messages/namespaces/specific/UpdateMatchingStatsMessage.java

disable daterange and money type compat check

3f0b1bb

adds missing coalesce for primaryColumn

ff18d93

don't remove connector column

b86af08

(experimental)

removes very verbose logging

7735740

use cursor to iterate results

ed98bd2

adds timing to SQL matching stats fetching

368b4f5

use transaction to disable autocommit

5ffdade

log select statement for debugging

0cb22e8

add PARALLEL SAFE marker to created functions

8e64d47

reworks SqlMatchingStats function as flattened table

8d5cd0a

(this is a bit absurd tbh)

adds some logging

44899ee

adds missing concept Id

e2df161

adds missing error handling in outer loop

750a03b

adds grouping by params to map into most specific child

152fc72

fixes wrong usage of immutable datasctructure

37009b0

try to insert the join-tables

67a0547

adds typing to fields

34b8bb5

remove primary key (nullability issue) and add index

4ff4776

fix index creation

11bd235

fix index creation #2

8e56eda

Fix quoting for hana

6aea1c9

awildturtok requested a review from thoniTUB April 9, 2026 10:00

awildturtok disabled auto-merge April 9, 2026 10:02

awildturtok commented Apr 9, 2026

View reviewed changes

Cleanup

bfaaba1

thoniTUB reviewed Apr 10, 2026

View reviewed changes

thoniTUB reviewed Apr 14, 2026

View reviewed changes

Comment thread backend/src/main/java/com/bakdata/conquery/sql/conversion/dialect/PostgreDialectBundle.java Outdated

restructures CTCondition.Expression

ec0a4cb

awildturtok force-pushed the feature/matching-stats-as-join-table branch from 37845a2 to ec0a4cb Compare April 20, 2026 15:08

awildturtok and others added 3 commits April 20, 2026 17:37

Merge branch 'develop' into feature/matching-stats-as-join-table

b4dbe1a

fixes Hana isNull checks

ed0c95d

Merge remote-tracking branch 'origin/feature/matching-stats-as-join-t…

8561957

…able' into feature/matching-stats-as-join-table

awildturtok requested a review from thoniTUB April 21, 2026 08:18

thoniTUB reviewed Apr 21, 2026

View reviewed changes

awildturtok commented Jun 2, 2026

View reviewed changes

awildturtok and others added 2 commits June 2, 2026 16:24

fixes test importer and some broken tests

5f7fdee

awildturtok force-pushed the feature/matching-stats-as-join-table branch from 402d93d to 5f7fdee Compare June 4, 2026 12:50

awildturtok and others added 4 commits June 4, 2026 16:09

Merge branch 'develop' into feature/matching-stats-as-join-table

c38aff0

cleanup

bd6dcb1

Merge remote-tracking branch 'origin/feature/matching-stats-as-join-t…

5524680

…able' into feature/matching-stats-as-join-table

cleanup of unused files

43c95e8

awildturtok commented Jun 8, 2026

View reviewed changes

Comment thread backend/src/main/java/com/bakdata/conquery/sql/conquery/SqlMatchingStats.java Outdated

awildturtok commented Jun 8, 2026

View reviewed changes

Comment thread backend/src/main/java/com/bakdata/conquery/sql/conquery/SqlMatchingStats.java Outdated

awildturtok commented Jun 8, 2026

View reviewed changes

Comment thread .../src/main/java/com/bakdata/conquery/sql/conversion/dialect/hana/HanaSqlFunctionProvider.java Outdated

awildturtok commented Jun 8, 2026

View reviewed changes

Comment thread backend/src/main/java/com/bakdata/conquery/sql/conversion/dialect/SqlFunctionProvider.java Outdated

awildturtok and others added 2 commits June 9, 2026 09:35

Merge branch 'develop' into feature/matching-stats-as-join-table

dae7276

Cleanup

a53830a

		@Data
		public abstract class StorageListener<T extends Namespace>{

	create 3 consecutive common table conceptConditions (CTE):
	create 3 consecutive common table expressions (CTE):

Conversation

awildturtok commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

awildturtok Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thoniTUB left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thoniTUB Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thoniTUB Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

thoniTUB Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

awildturtok Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

awildturtok commented Jan 22, 2026 •

edited

Loading