📝 Docs | Update project readme

.github/workflows/README.md

@@ -0,0 +1,64 @@
+# GitHub Workflows
+
+This directory contains GitHub Actions workflows for continuous integration and continuous deployment (CI/CD) of the dab_project.
+
+## Workflows Overview
+
+### CI Workflow (ci-workflow.yml)
+
+**Purpose:** Validates code quality and runs unit tests on every pull request.
+
+**Trigger:** Automatically runs when a pull request is opened or updated against the `main` branch.
+
+**Steps:**
+1. Checks out the code
+2. Sets up Python 3.11
+3. Installs PySpark and testing dependencies
+4. Runs unit tests with code coverage using pytest
+5. Generates HTML coverage report
+6. Uploads coverage report as an artifact
+
+**Artifacts:** HTML coverage report (available for download in the workflow run)
+
+---
+
+### CD Workflow (cd-workflow.yml)
+
+**Purpose:** Automatically deploys the project to Databricks workspaces (TEST and PROD environments).
+
+**Trigger:** Automatically runs when code is pushed to the `main` branch.
+
+**Steps:**
+
+#### Deploy to TEST
+1. Checks out the code
+2. Sets up Python 3.11
+3. Installs Databricks CLI
+4. Installs build dependencies (setuptools, wheel)
+5. Configures Databricks credentials using service principal secrets
+6. Deploys bundle to TEST environment using `databricks bundle deploy --target test`
+
+#### Deploy to PROD
+1. Runs only after TEST deployment succeeds (dependency)
+2. Follows the same steps as TEST deployment
+3. Deploys bundle to PROD environment using `databricks bundle deploy --target prod`
+
+**Environment Protection:** Both TEST and PROD deployments use GitHub Environments for credential management and approval workflows.
+
+---
+
+## Required Secrets
+
+For CD workflows to function, configure the following secrets in your GitHub repository settings:
+
+- `SP_CLIENT_ID`: Service Principal Client ID for Databricks authentication
+- `SP_CLIENT_SECRET`: Service Principal Client Secret for Databricks authentication
+
+## Workspace Configuration
+
+Update the Databricks workspace URLs in the CD workflow file:
+- Line 32: TEST workspace URL
+- Line 65: PROD workspace URL
+
+----------
+[Go back to main README](../../README.md)

README.md

@@ -4,24 +4,55 @@ This is a repository created during Udemy course `CI/CD with Databricks Asset Bu
 
 Primary repository URL with Wiki owned by course instructor: https://github.com/pathfinder-analytics-uk/dab_project.
 
+## Table of Contents
+
+- [Project structure](#project-structure)
+- [Getting started](#getting-started)
+  - [Prerequisites](#prerequisites)
+  - [Setup Instructions](#setup-instructions)
+  - [Virtual Environment Setup](#virtual-environment-setup)
+  - [Java Setup for Local PySpark Development](#java-setup-for-local-pyspark-development)
+  - [Unit Tests with Pytest](#unit-tests-with-pytest)
+  - [Databricks CLI, Set-Up and Bundle Commands](#databricks-cli-set-up-and-bundle-commands)
+- [CI-CD Setup](#ci-cd-setup)
+
 ## Project structure
 
-* `demos`: Demo scripts, yaml configurations which demonstrate core concepts and functionalities used to develop project.
-* `docs`: Folder with project documentation (diagrams, tables structure)
-* `src/`: Python source code with utilities used for this project.
-* `resources/`:  Resource configurations (jobs, pipelines, etc.)
-* `tests/`: Unit tests for the shared Python code.
-* `fixtures/`: Fixtures for data sets (primarily used for testing).
-* `notebooks`: Notebooks used to test project setup.
+```
+dab_project/
+├── .github/
+│   └── workflows/              GitHub Actions CI/CD workflows
+├── demos/                      Demo scripts and YAML configurations demonstrating core concepts
+├── docs/                       Project documentation (diagrams, data models)
+├── src/                        Python source code and utilities
+├── resources/                  Resource configurations (jobs, pipelines)
+├── tests/                      Unit tests for shared Python code
+├── fixtures/                   Test data sets
+├── notebooks/                  Notebooks for testing project setup
+├── citibike_etl/               ETL workflows and scripts
+├── databricks.yml              Databricks Asset Bundle configuration
+├── setup.py                    Python packaging configuration for wheel distribution
+├── requirements-dbc.txt        Databricks Connect dependencies
+├── requirements-pyspark.txt    PySpark dependencies
+└── README.md                   Project documentation
+```
 
 
 ## Getting started
 
-Before you run or deploy this project, you'll have to follow along with the Udemy course to set up your environment (Databricks Workspaces, Service Principals, GitHub Repository etc.).
+### Prerequisites
+
+Before you proceed, ensure you have:
+
+- [ ] Completed the Udemy course setup (Databricks Workspaces, Service Principals, GitHub Repository)
+- [ ] Updated the `databricks.yml` configuration file with your Workspace URLs and Service Principal details
+- [ ] Python 3.11 installed on your system
+- [ ] Java Development Kit (JDK) 8, 11, or newer (required for local PySpark development)
+- [ ] Databricks CLI installed
 
-You will also need to update the `databricks.yml` configuration file with your Workspace URLs and Service Principal details.
+### Setup Instructions
 
-You'll also want to set up local Python environments for Databricks Connect and local PySpark development. Follow the instructions for your platform below.
+Follow the instructions for your platform below to set up local Python environments for Databricks Connect and local PySpark development.
 
 ### Virtual Environment Setup
 
@@ -78,7 +109,7 @@ You'll also want to set up local Python environments for Databricks Connect and
 
 4. **Create and activate the local PySpark environment**
    ```powershell
-   python -m venv .venv_pyspark
+   py -3.11 -m venv .venv_pyspark
    .\.venv_pyspark\Scripts\Activate.ps1
    ```
 5. **Install PySpark dependencies**
@@ -92,9 +123,102 @@ You'll also want to set up local Python environments for Databricks Connect and
    ```
 
 ---
-### Databricks CLI, Set-Up and Bundle Commands
 
-TODO: add instructions for Windows users.
+### Java Setup for Local PySpark Development
+
+This project uses `pyspark` as its data processing and transformation technology. To develop and test pyspark code on a local environment, you need to install Java Development Kit (JDK).
+
+#### macOS / Linux
+
+1. Download Java JDK 8, 11, or newer from [Oracle Java Download Website](https://www.oracle.com/java/technologies/downloads/). Recommended version is JDK 11.
+   ```bash
+   # Example: Download and install JDK 11
+   # On macOS with Homebrew:
+   brew install java11
+   ```
+
+2. Set the `JAVA_HOME` environment variable:
+   ```bash
+   # Find your Java installation path
+   /usr/libexec/java_home -v 11
+
+   # Add to your shell profile (~/.zprofile, ~/.bash_profile, or ~/.bashrc)
+   export JAVA_HOME=$(/usr/libexec/java_home -v 11)
+   ```
+
+3. Verify Java installation:
+   ```bash
+   java --version
+   ```
+
+#### Windows
+
+1. Download Java JDK 8, 11, or newer from [Oracle Java Download Website](https://www.oracle.com/java/technologies/downloads/). Recommended version is JDK 11.
+2. Add new `JAVA_HOME` system environment variable with the path to the folder with Java installation:
+
+   **Windows**
+
+   | Variable | Value |
+   |----------|-------|
+   | `JAVA_HOME` | `C:\Program Files\Java\jdk-11.0.31` |
+   | `Path` (add entry) | `C:\Program Files\Java\jdk-11.0.31\bin` |
+
+
+3. Verify Java installation:
+   ```pwsh
+   java --version
+   ```
+   Output should look like this:
+   ```
+   java 11.0.31 2026-04-21 LTS
+   Java(TM) SE Runtime Environment 18.9 (build 11.0.31+9-LTS-165)
+   Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.31+9-LTS-165, mixed mode)
+   ```
+
+---
+
+### Unit Tests with Pytest
+
+This project uses the `Pytest` framework for unit tests.
+
+#### macOS / Linux
+
+1. Activate the virtual environment with installed `pyspark` and `pytest` packages:
+   ```bash
+   source .venv_pyspark/bin/activate
+   ```
+
+2. Run unit tests:
+   ```bash
+   pytest tests
+   ```
+
+#### Windows
+
+1. Activate the virtual environment with installed `pyspark` and `pytest` packages:
+   ```powershell
+   .\.venv_pyspark\Scripts\Activate.ps1
+   ```
+
+2. Run unit tests:
+   ```powershell
+   pytest tests
+   ```
+
+#### Configure Unit Tests in VS Code
+
+Alternatively, you can configure unit tests in Visual Studio Code:
+
+1. Install the `Python extension for Visual Studio Code` from VS Code Extensions.
+2. Navigate to the **Testing** tab in VS Code.
+3. Configure unit tests:
+   - Select `Pytest` as the testing framework
+   - Select `tests` folder as the unit tests folder
+4. Unit tests should now be visible in the Testing window.
+
+
+
+### Databricks CLI, Set-Up and Bundle Commands
 
 1. Install the Databricks CLI
    ```bash
@@ -144,3 +268,7 @@ TODO: add instructions for Windows users.
    for this project, and for CI/CD configuration, see
    https://docs.databricks.com/dev-tools/bundles/index.html.
 
+
+## CI-CD Setup
+
+Refer to [Worflows README](.github/workflows/README.md) for more details about CI-CD setup for this project

setup.py

@@ -1,3 +1,10 @@
+"""
+Python packaging configuration for dab_project. This file defines how to build and
+distribute the project as a Python wheel package. When deploying a wheel with DAB,
+the command `python setup.py bdist_wheel` is executed by default to create a
+distributable .whl file that can be installed in other Python environments.
+"""
+
 from setuptools import setup, find_packages
 
 setup(