From 26b31502b1bb7af0c9ccf22f676e0fb768aadaa0 Mon Sep 17 00:00:00 2001 From: devtmaz Date: Tue, 23 Jun 2026 12:18:55 +0200 Subject: [PATCH 1/5] =?UTF-8?q?=F0=9F=93=9D=20add=20java=20installation=20?= =?UTF-8?q?steps,=20unit=20tests=20setup=20and=20refine=20layout=20of=20pr?= =?UTF-8?q?oject=20readme?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 147 +++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 134 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 31fefbe..f58c3ca 100644 --- a/README.md +++ b/README.md @@ -4,24 +4,52 @@ This is a repository created during Udemy course `CI/CD with Databricks Asset Bu Primary repository URL with Wiki owned by course instructor: https://github.com/pathfinder-analytics-uk/dab_project. +## Table of Contents + +- [Project structure](#project-structure) +- [Getting started](#getting-started) + - [Prerequisites](#prerequisites) + - [Setup Instructions](#setup-instructions) + - [Virtual Environment Setup](#virtual-environment-setup) + - [Java Setup for Local PySpark Development](#java-setup-for-local-pyspark-development) + - [Unit Tests with Pytest](#unit-tests-with-pytest) + - [Databricks CLI, Set-Up and Bundle Commands](#databricks-cli-set-up-and-bundle-commands) + ## Project structure -* `demos`: Demo scripts, yaml configurations which demonstrate core concepts and functionalities used to develop project. -* `docs`: Folder with project documentation (diagrams, tables structure) -* `src/`: Python source code with utilities used for this project. -* `resources/`: Resource configurations (jobs, pipelines, etc.) -* `tests/`: Unit tests for the shared Python code. -* `fixtures/`: Fixtures for data sets (primarily used for testing). -* `notebooks`: Notebooks used to test project setup. +``` +dab_project/ +├── demos/ Demo scripts and YAML configurations demonstrating core concepts +├── docs/ Project documentation (diagrams, data models) +├── src/ Python source code and utilities +├── resources/ Resource configurations (jobs, pipelines) +├── tests/ Unit tests for shared Python code +├── fixtures/ Test data sets +├── notebooks/ Notebooks for testing project setup +├── citibike_etl/ ETL workflows and scripts +├── build/ Built project artifacts +├── databricks.yml Databricks Asset Bundle configuration +├── requirements-dbc.txt Databricks Connect dependencies +├── requirements-pyspark.txt PySpark dependencies +└── README.md Project documentation +``` ## Getting started -Before you run or deploy this project, you'll have to follow along with the Udemy course to set up your environment (Databricks Workspaces, Service Principals, GitHub Repository etc.). +### Prerequisites + +Before you proceed, ensure you have: + +- [ ] Completed the Udemy course setup (Databricks Workspaces, Service Principals, GitHub Repository) +- [ ] Updated the `databricks.yml` configuration file with your Workspace URLs and Service Principal details +- [ ] Python 3.11 installed on your system +- [ ] Java Development Kit (JDK) 8, 11, or newer (required for local PySpark development) +- [ ] Databricks CLI installed -You will also need to update the `databricks.yml` configuration file with your Workspace URLs and Service Principal details. +### Setup Instructions -You'll also want to set up local Python environments for Databricks Connect and local PySpark development. Follow the instructions for your platform below. +Follow the instructions for your platform below to set up local Python environments for Databricks Connect and local PySpark development. ### Virtual Environment Setup @@ -78,7 +106,7 @@ You'll also want to set up local Python environments for Databricks Connect and 4. **Create and activate the local PySpark environment** ```powershell - python -m venv .venv_pyspark + py -3.11 -m venv .venv_pyspark .\.venv_pyspark\Scripts\Activate.ps1 ``` 5. **Install PySpark dependencies** @@ -92,9 +120,102 @@ You'll also want to set up local Python environments for Databricks Connect and ``` --- -### Databricks CLI, Set-Up and Bundle Commands -TODO: add instructions for Windows users. +### Java Setup for Local PySpark Development + +This project uses `pyspark` as its data processing and transformation technology. To develop and test pyspark code on a local environment, you need to install Java Development Kit (JDK). + +#### macOS / Linux + +1. Download Java JDK 8, 11, or newer from [Oracle Java Download Website](https://www.oracle.com/java/technologies/downloads/). Recommended version is JDK 11. + ```bash + # Example: Download and install JDK 11 + # On macOS with Homebrew: + brew install java11 + ``` + +2. Set the `JAVA_HOME` environment variable: + ```bash + # Find your Java installation path + /usr/libexec/java_home -v 11 + + # Add to your shell profile (~/.zprofile, ~/.bash_profile, or ~/.bashrc) + export JAVA_HOME=$(/usr/libexec/java_home -v 11) + ``` + +3. Verify Java installation: + ```bash + java --version + ``` + +#### Windows + +1. Download Java JDK 8, 11, or newer from [Oracle Java Download Website](https://www.oracle.com/java/technologies/downloads/). Recommended version is JDK 11. +2. Add new `JAVA_HOME` system environment variable with the path to the folder with Java installation: + + **Windows** + + | Variable | Value | + |----------|-------| + | `JAVA_HOME` | `C:\Program Files\Java\jdk-11.0.31` | + | `Path` (add entry) | `C:\Program Files\Java\jdk-11.0.31\bin` | + + +3. Verify Java installation: + ```pwsh + java --version + ``` + Output should look like this: + ``` + java 11.0.31 2026-04-21 LTS + Java(TM) SE Runtime Environment 18.9 (build 11.0.31+9-LTS-165) + Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.31+9-LTS-165, mixed mode) + ``` + +--- + +### Unit Tests with Pytest + +This project uses the `Pytest` framework for unit tests. + +#### macOS / Linux + +1. Activate the virtual environment with installed `pyspark` and `pytest` packages: + ```bash + source .venv_pyspark/bin/activate + ``` + +2. Run unit tests: + ```bash + pytest tests + ``` + +#### Windows + +1. Activate the virtual environment with installed `pyspark` and `pytest` packages: + ```powershell + .\.venv_pyspark\Scripts\Activate.ps1 + ``` + +2. Run unit tests: + ```powershell + pytest tests + ``` + +#### Configure Unit Tests in VS Code + +Alternatively, you can configure unit tests in Visual Studio Code: + +1. Install the `Python extension for Visual Studio Code` from VS Code Extensions. +2. Navigate to the **Testing** tab in VS Code. +3. Configure unit tests: + - Select `Pytest` as the testing framework + - Select `tests` folder as the unit tests folder +4. Unit tests should now be visible in the Testing window. + + + +### Databricks CLI, Set-Up and Bundle Commands 1. Install the Databricks CLI ```bash From ea444d5aa99d5c3df846a067a4a0af07c6f023b3 Mon Sep 17 00:00:00 2001 From: devtmaz Date: Tue, 23 Jun 2026 12:28:36 +0200 Subject: [PATCH 2/5] =?UTF-8?q?=F0=9F=93=9D=20update=20documentation=20of?= =?UTF-8?q?=20setup.py?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- setup.py | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/setup.py b/setup.py index 56a5f24..1d75816 100644 --- a/setup.py +++ b/setup.py @@ -1,3 +1,10 @@ +""" +Python packaging configuration for dab_project. This file defines how to build and +distribute the project as a Python wheel package. When deploying a wheel with DAB, +the command `python setup.py bdist_wheel` is executed by default to create a +distributable .whl file that can be installed in other Python environments. +""" + from setuptools import setup, find_packages setup( From fe8e524c63b91e04d39031936bd44d90522cf56c Mon Sep 17 00:00:00 2001 From: devtmaz Date: Tue, 23 Jun 2026 12:41:02 +0200 Subject: [PATCH 3/5] =?UTF-8?q?=F0=9F=93=9D=20add=20readme=20documentation?= =?UTF-8?q?=20for=20CI-CD=20workflows?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .github/workflows/README.md | 64 +++++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 .github/workflows/README.md diff --git a/.github/workflows/README.md b/.github/workflows/README.md new file mode 100644 index 0000000..28e9b92 --- /dev/null +++ b/.github/workflows/README.md @@ -0,0 +1,64 @@ +# GitHub Workflows + +This directory contains GitHub Actions workflows for continuous integration and continuous deployment (CI/CD) of the dab_project. + +## Workflows Overview + +### CI Workflow (ci-workflow.yml) + +**Purpose:** Validates code quality and runs unit tests on every pull request. + +**Trigger:** Automatically runs when a pull request is opened or updated against the `main` branch. + +**Steps:** +1. Checks out the code +2. Sets up Python 3.11 +3. Installs PySpark and testing dependencies +4. Runs unit tests with code coverage using pytest +5. Generates HTML coverage report +6. Uploads coverage report as an artifact + +**Artifacts:** HTML coverage report (available for download in the workflow run) + +--- + +### CD Workflow (cd-workflow.yml) + +**Purpose:** Automatically deploys the project to Databricks workspaces (TEST and PROD environments). + +**Trigger:** Automatically runs when code is pushed to the `main` branch. + +**Steps:** + +#### Deploy to TEST +1. Checks out the code +2. Sets up Python 3.11 +3. Installs Databricks CLI +4. Installs build dependencies (setuptools, wheel) +5. Configures Databricks credentials using service principal secrets +6. Deploys bundle to TEST environment using `databricks bundle deploy --target test` + +#### Deploy to PROD +1. Runs only after TEST deployment succeeds (dependency) +2. Follows the same steps as TEST deployment +3. Deploys bundle to PROD environment using `databricks bundle deploy --target prod` + +**Environment Protection:** Both TEST and PROD deployments use GitHub Environments for credential management and approval workflows. + +--- + +## Required Secrets + +For CD workflows to function, configure the following secrets in your GitHub repository settings: + +- `SP_CLIENT_ID`: Service Principal Client ID for Databricks authentication +- `SP_CLIENT_SECRET`: Service Principal Client Secret for Databricks authentication + +## Workspace Configuration + +Update the Databricks workspace URLs in the CD workflow file: +- Line 32: TEST workspace URL +- Line 65: PROD workspace URL + +---------- +[Go back to main README](../../README.md) \ No newline at end of file From be6eae766bef0b8d2c4eedc8d40bd895ee4ce230 Mon Sep 17 00:00:00 2001 From: devtmaz Date: Tue, 23 Jun 2026 12:41:36 +0200 Subject: [PATCH 4/5] =?UTF-8?q?=F0=9F=93=9D=20update=20project=20structure?= =?UTF-8?q?=20section=20in=20main=20README?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index f58c3ca..9f21630 100644 --- a/README.md +++ b/README.md @@ -19,19 +19,21 @@ Primary repository URL with Wiki owned by course instructor: https://github.com/ ``` dab_project/ -├── demos/ Demo scripts and YAML configurations demonstrating core concepts -├── docs/ Project documentation (diagrams, data models) -├── src/ Python source code and utilities -├── resources/ Resource configurations (jobs, pipelines) -├── tests/ Unit tests for shared Python code -├── fixtures/ Test data sets -├── notebooks/ Notebooks for testing project setup -├── citibike_etl/ ETL workflows and scripts -├── build/ Built project artifacts -├── databricks.yml Databricks Asset Bundle configuration -├── requirements-dbc.txt Databricks Connect dependencies -├── requirements-pyspark.txt PySpark dependencies -└── README.md Project documentation +├── .github/ +│ └── workflows/ GitHub Actions CI/CD workflows +├── demos/ Demo scripts and YAML configurations demonstrating core concepts +├── docs/ Project documentation (diagrams, data models) +├── src/ Python source code and utilities +├── resources/ Resource configurations (jobs, pipelines) +├── tests/ Unit tests for shared Python code +├── fixtures/ Test data sets +├── notebooks/ Notebooks for testing project setup +├── citibike_etl/ ETL workflows and scripts +├── databricks.yml Databricks Asset Bundle configuration +├── setup.py Python packaging configuration for wheel distribution +├── requirements-dbc.txt Databricks Connect dependencies +├── requirements-pyspark.txt PySpark dependencies +└── README.md Project documentation ``` @@ -265,3 +267,7 @@ Alternatively, you can configure unit tests in Visual Studio Code: for this project, and for CI/CD configuration, see https://docs.databricks.com/dev-tools/bundles/index.html. + +## CI-CD Setup + +Refer to [Worflows README](.github/workflows/README.md) for more details about CI-CD setup for this project From 54cc5151e315baa7a6d456d4deb8ee1ea5543502 Mon Sep 17 00:00:00 2001 From: devtmaz Date: Tue, 23 Jun 2026 12:46:00 +0200 Subject: [PATCH 5/5] =?UTF-8?q?=F0=9F=93=9D=20update=20table=20of=20conten?= =?UTF-8?q?ts?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 9f21630..35f1ddf 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,7 @@ Primary repository URL with Wiki owned by course instructor: https://github.com/ - [Java Setup for Local PySpark Development](#java-setup-for-local-pyspark-development) - [Unit Tests with Pytest](#unit-tests-with-pytest) - [Databricks CLI, Set-Up and Bundle Commands](#databricks-cli-set-up-and-bundle-commands) +- [CI-CD Setup](#ci-cd-setup) ## Project structure