Skip to content

DBSQL for Dimension ETL#127

Open
shyamraodb wants to merge 43 commits into
databricks-demos:mainfrom
shyamraodb:main
Open

DBSQL for Dimension ETL#127
shyamraodb wants to merge 43 commits into
databricks-demos:mainfrom
shyamraodb:main

Conversation

@shyamraodb

@shyamraodb shyamraodb commented Feb 27, 2025

Copy link
Copy Markdown

Demo of SCD Type2 ETL using DBSQL:
Added folder dbsql-for-dim-etl under product_demos/DBSQL-Datawarehousing.
Included notebooks, images, csv files corresponding to demo.

@shyamraodb shyamraodb changed the title created fork; added dbsql-for-dim-etl DBSQL for Dimension ETL Feb 27, 2025
@QuentinAmbard

QuentinAmbard commented Mar 1, 2025

Copy link
Copy Markdown
Collaborator

Thanks, can you rename the folders/files as in the other demo, we usually lowercase everything (only keep AI or ETL uppercase) and add dash - in the file name (no white space)
Here is an example:
image

We'll have to improve the dbdemos toolkit because we don't support py files (it wasn't available through API when we did it)

Can you also add a _resource folder with a bundle fine, and add fill the content for this demo?

@QuentinAmbard QuentinAmbard left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hey, I did a few style / sql comment, I'll do another pass on the flow/logic after, let's catchup next week to review it? Thanks!

Comment thread product_demos/DBSQL-Datawarehousing/dbsql-for-dim-etl/_util/initialize-staging.py Outdated
Comment thread product_demos/DBSQL-Datawarehousing/dbsql-for-dim-etl/00-Setup/Initialize.sql Outdated
Comment thread product_demos/DBSQL-Datawarehousing/dbsql-for-dim-etl/00-Setup/Initialize.sql Outdated
Comment thread product_demos/DBSQL-Datawarehousing/dbsql-for-dim-etl/00-Setup/Setup.sql Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need to review this and use the python SDK instead if it's part of the demo, or instead we move it within the bundle file and the workflow got created automatically and we don't expose this file to the user - let's talk about it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants