From 4c5909911398270b20b4fd4f83ad3552e785c99b Mon Sep 17 00:00:00 2001 From: yu1005 <109812036+jeromychu23@users.noreply.github.com> Date: Fri, 6 Mar 2026 09:31:08 +0800 Subject: [PATCH 1/4] docs: add polished README describing project purpose and use case --- README.md | 140 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 140 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..853dd74 --- /dev/null +++ b/README.md @@ -0,0 +1,140 @@ +# πŸš€ hello-rust: Accelerate Hierarchical DataFrame Processing with Rust + PyO3 + +[![Rust](https://img.shields.io/badge/Rust-stable-orange.svg)](https://www.rust-lang.org/) +[![Python](https://img.shields.io/badge/Python-3.10%2B-blue.svg)](https://www.python.org/) +[![PyO3](https://img.shields.io/badge/PyO3-enabled-purple.svg)](https://pyo3.rs/) +[![Polars](https://img.shields.io/badge/Polars-DataFrame-0ea5e9.svg)](https://pola.rs/) +[![Conda](https://img.shields.io/badge/Conda-ready-44A833.svg)](https://docs.conda.io/) + +> A practical template for writing performance-critical Rust functions and exposing them to Python with **PyO3**, optimized for **Polars DataFrame** workflows and publishable to **Conda**. + +--- + +## 🎯 Repository Purpose + +This repository demonstrates how to: + +- Build Rust functions for complex data transformations. +- Expose Rust logic to Python via PyO3. +- Use these functions in Python/Polars pipelines for better performance. +- Package and distribute the project through Conda. + +The main goal is to speed up heavy hierarchical processing that can become slow and hard to maintain in pure Python. + +--- + +## πŸ˜΅β€πŸ’« Pain Point This Repo Solves + +A common real-world scenario is processing hierarchical maintenance history in tabular form, such as: + +- Aircraft component install/remove records +- Parent-child relationship chains +- Unknown hierarchy depth + +For each component event, you may need to determine: + +> β€œIs its parent (or ancestor) currently installed on the aircraft at that point in time?” + +Doing this with Python loops over large datasets often leads to: + +- High runtime cost +- Complex DFS traversal logic +- Difficult-to-maintain code paths + +This repo tackles that by moving DFS-style traversal and matching logic into Rust. + +--- + +## 🧠 Core Approach + +- **DataFrame layer:** Polars (Python side) +- **Compute layer:** Rust +- **Bridge layer:** PyO3 +- **Distribution:** Conda recipe (`recipe/meta.yaml`) + +Typical flow: + +1. Load/prepare a Polars DataFrame in Python. +2. Call Rust-powered functions for hierarchical traversal and matching. +3. Receive processed results back as DataFrame-ready outputs. + +--- + +## πŸ“Š Example: Before vs After Hierarchical Processing + +### Input (Before) + +A simplified event history: + +| ts | component | parent | event | +|---:|:---------:|:------:|:-----:| +| 1 | A320 | AIRCRAFT | Install | +| 2 | ENG-1 | A320 | Install | +| 3 | FAN-9 | ENG-1 | Install | +| 4 | ENG-1 | A320 | Remove | +| 5 | FAN-9 | ENG-1 | Inspect | + +Challenge: at `ts=5`, `FAN-9` still points to parent `ENG-1`, but parent status depends on timeline and ancestor chain. + +### Output (After) + +After DFS-like hierarchical evaluation: + +| ts | component | parent | event | ancestor_on_aircraft | +|---:|:---------:|:------:|:-----:|:--------------------:| +| 1 | A320 | AIRCRAFT | Install | true | +| 2 | ENG-1 | A320 | Install | true | +| 3 | FAN-9 | ENG-1 | Install | true | +| 4 | ENG-1 | A320 | Remove | false | +| 5 | FAN-9 | ENG-1 | Inspect | false | + +This extra result column is what downstream analytics need, but computing it efficiently is where Rust helps. + +--- + +## βœ… Why Rust + PyO3 Here? + +- Faster traversal for deep/irregular hierarchies +- Better control over memory and algorithmic behavior +- Reusable from Python with minimal API friction +- Easier scaling to larger maintenance/event datasets + +--- + +## πŸ“¦ Conda-Friendly Packaging + +This repo includes a Conda recipe so the extension can be distributed and installed in data-science environments. + +- Build from Rust + Python packaging metadata +- Ship as Conda artifact for team-wide usage + +--- + +## πŸ› οΈ Tech Stack + +- Rust +- PyO3 +- Python +- Polars +- Conda + +--- + +## 🧭 Project Direction + +Potential next steps: + +- Add benchmark comparisons (pure Python vs Rust extension) +- Add end-to-end Polars examples in `examples/` +- Add test fixtures for deep hierarchy and edge cases +- Publish package to internal/public Conda channels + +--- + +## πŸ‘€ Who This Is For + +- Data engineers dealing with hierarchical tabular data +- MRO / aviation analytics teams +- Python users who need selective Rust acceleration + +If your bottleneck is DFS-like logic over unknown-depth parent-child structures in DataFrames, this repo is designed for you. From 92b8254d1b64f7c70c3e90ffb1a59930660890d3 Mon Sep 17 00:00:00 2001 From: yu1005 <109812036+jeromychu23@users.noreply.github.com> Date: Fri, 6 Mar 2026 09:55:28 +0800 Subject: [PATCH 2/4] Enhance README with transaction dates and clarifications Updated the input and output tables in the README to include transaction dates and clarified the challenge description. --- README.md | 31 +++++++++++++------------------ 1 file changed, 13 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 853dd74..ad1d940 100644 --- a/README.md +++ b/README.md @@ -66,28 +66,25 @@ Typical flow: A simplified event history: -| ts | component | parent | event | -|---:|:---------:|:------:|:-----:| -| 1 | A320 | AIRCRAFT | Install | -| 2 | ENG-1 | A320 | Install | -| 3 | FAN-9 | ENG-1 | Install | -| 4 | ENG-1 | A320 | Remove | -| 5 | FAN-9 | ENG-1 | Inspect | +| ts | component | parent | event | transaction date | +|---:|:---------:|:------:|:-----:|:----------------:| +| 1 | A320 | AIRCRAFT | Install | 2026-01-01 | +| 2 | ENG-1 | A320 | Install | 2025-12-31 | +| 3 | FAN-9 | ENG-1 | Install | 2025-12-30 | -Challenge: at `ts=5`, `FAN-9` still points to parent `ENG-1`, but parent status depends on timeline and ancestor chain. +Challenge: at `2025-12-30`, `FAN-9` installed to parent `ENG-1`, but parent haven't been installed to `AIRCRAFT`. ### Output (After) After DFS-like hierarchical evaluation: -| ts | component | parent | event | ancestor_on_aircraft | -|---:|:---------:|:------:|:-----:|:--------------------:| -| 1 | A320 | AIRCRAFT | Install | true | -| 2 | ENG-1 | A320 | Install | true | -| 3 | FAN-9 | ENG-1 | Install | true | -| 4 | ENG-1 | A320 | Remove | false | -| 5 | FAN-9 | ENG-1 | Inspect | false | +| ts | component | parent | event | transaction date | new transaction date | +|---:|:---------:|:------:|:-----:|:----------------:|:--------------------:| +| 1 | A320 | AIRCRAFT | Install | 2026-01-01 | 2026-01-01 | +| 2 | ENG-1 | A320 | Install | 2025-12-31 | 2026-01-01 | +| 3 | FAN-9 | ENG-1 | Install | 2025-12-30 | 2026-01-01 | +Explain: All child components should point to the date when it's top parent be installed to `AIRCRAFT`. This extra result column is what downstream analytics need, but computing it efficiently is where Rust helps. --- @@ -134,7 +131,5 @@ Potential next steps: ## πŸ‘€ Who This Is For - Data engineers dealing with hierarchical tabular data -- MRO / aviation analytics teams +- Aviation analytics teams (Tracking component usage) - Python users who need selective Rust acceleration - -If your bottleneck is DFS-like logic over unknown-depth parent-child structures in DataFrames, this repo is designed for you. From a79e00e96dcf532eed7069b86316d21d025ce392 Mon Sep 17 00:00:00 2001 From: yu1005 <109812036+jeromychu23@users.noreply.github.com> Date: Fri, 6 Mar 2026 10:30:48 +0800 Subject: [PATCH 3/4] Revise README for clarity and additional examples Clarify explanations and improve examples in README. --- README.md | 40 +++++++++++++++++++++++++++++++++++----- 1 file changed, 35 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index ad1d940..85b5fcd 100644 --- a/README.md +++ b/README.md @@ -33,13 +33,14 @@ A common real-world scenario is processing hierarchical maintenance history in t For each component event, you may need to determine: -> β€œIs its parent (or ancestor) currently installed on the aircraft at that point in time?” +> β€œIs its parent currently installed on the aircraft at that point in time?” -Doing this with Python loops over large datasets often leads to: +Doing this with multi-level join over large datasets often leads to: - High runtime cost -- Complex DFS traversal logic +- Complex traversal logic - Difficult-to-maintain code paths +- Lack of Edge-Case Handling This repo tackles that by moving DFS-style traversal and matching logic into Rust. @@ -60,7 +61,7 @@ Typical flow: --- -## πŸ“Š Example: Before vs After Hierarchical Processing +## πŸ“Š Example 1: Before vs After Hierarchical Processing ### Input (Before) @@ -84,9 +85,38 @@ After DFS-like hierarchical evaluation: | 2 | ENG-1 | A320 | Install | 2025-12-31 | 2026-01-01 | | 3 | FAN-9 | ENG-1 | Install | 2025-12-30 | 2026-01-01 | -Explain: All child components should point to the date when it's top parent be installed to `AIRCRAFT`. +Explain: All child components should point to the date when its top parent be installed to `AIRCRAFT`. This extra result column is what downstream analytics need, but computing it efficiently is where Rust helps. +## πŸ“Š Example 2: Before vs After Hierarchical Processing (Include block key) + +### Input (Before) + +A simplified event history: + +| ts | component | parent | event | transaction date | +|---:|:---------:|:------:|:-----:|:----------------:| +| 1 | A320 | AIRCRAFT | Install | 2026-01-01 | +| 2 | ENG-1 | A320 | Install | 2025-12-31 | +| 3 | FAN-9 | ENG-1 | Remove | 2025-12-31 | +| 4 | FAN-9 | ENG-1 | Install | 2025-12-30 | + +Challenge: Removal involved in the dataframe, without `block key` the install date will be set to `2026-01-01` which is unreasonable. + +### Output (After, with `block ley`) + +After DFS-like hierarchical evaluation: + +| ts | component | parent | event | transaction date | new transaction date | +|---:|:---------:|:------:|:-----:|:----------------:|:--------------------:| +| 1 | A320 | AIRCRAFT | Install | 2026-01-01 | 2026-01-01 | +| 2 | ENG-1 | A320 | Install | 2025-12-31 | 2026-01-01 | +| 3 | FAN-9 | ENG-1 | Remove | 2025-12-31 | 2025-12-31 | +| 4 | FAN-9 | ENG-1 | Install | 2025-12-30 | 2025-12-30 | + +Explain: Set the `block key` to prevent unreasonable transaction. +Because `FAN-9` has been removed from its parent before installed to the `AIRCRAFT`, the transaction date should remain the same. + --- ## βœ… Why Rust + PyO3 Here? From 9aef21625720ddb0167803bd83954eccc0e70c04 Mon Sep 17 00:00:00 2001 From: yu1005 <109812036+jeromychu23@users.noreply.github.com> Date: Fri, 6 Mar 2026 10:31:35 +0800 Subject: [PATCH 4/4] Fix typo in 'block key' section of README --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 85b5fcd..89458aa 100644 --- a/README.md +++ b/README.md @@ -103,7 +103,7 @@ A simplified event history: Challenge: Removal involved in the dataframe, without `block key` the install date will be set to `2026-01-01` which is unreasonable. -### Output (After, with `block ley`) +### Output (After, with `block key`) After DFS-like hierarchical evaluation: