Feature/api rework

[Mg30]

This pull request updates the Flowrun example DAGs and documentation to use the new Pipeline API instead of the previous engine-based approach. It also introduces a comprehensive authoring guide for Flowrun DAGs. The most significant changes include API migration in all examples, improved code clarity, and the addition of detailed instructions for best practices and patterns.

API Migration and Code Refactoring:

• All example files (demo.py, micro_batch_demo.py, polars_workflow_demo.py) have been refactored to use the Pipeline class for DAG creation and task registration, replacing the older build_default_engine and .dag() methods. Task decorators and execution helpers have been updated accordingly. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17]

Documentation and Authoring Guidance:

• A new, in-depth authoring guide for Flowrun DAGs has been added at .github/instructions/flowrun-dag-authoring.instructions.md. This guide covers the philosophy behind Flowrun, best practices for DAG and task authoring, API usage, dependency management, retries, async timeouts, hooks, and example workflow patterns. It also provides clear rules and anti-patterns to avoid.

Improvements to Example Code Patterns:

• Example code now follows the recommended structure: plain helpers for business logic, typed models for IO, thin task wrappers, and a concise main() for orchestration and reporting. This improves readability and maintainability across all examples. [1] [2] [3]

Consistency and Best Practices:

• All examples now consistently use typed interfaces (dataclass, TypedDict), explicit context passing, and structured return values, aligning with the new documentation’s recommendations. [1] [2] [3]

Error Handling and Validation:

• The demo.py and micro_batch_demo.py examples now include improved error handling for subgraph runs and ensure that validation and reporting are performed using the new API. [1] [2]

These changes modernize the Flowrun examples and documentation, making them easier to follow, more idiomatic, and aligned with the intended usage of the Flowrun library.

[safedep]

SafeDep Report Summary

Package Details

Package	Malware	Vulnerability	Risky License	Report
flowrun-dag @ 1.0.0 uv.lock				🔗

View complete scan results →

_{This report is generated by SafeDep Github App}

[Copilot]

Pull request overview

This PR migrates Flowrun’s public-facing examples and documentation to a new Pipeline-first authoring/execution API while keeping the internal Engine as a compatibility layer. It also expands guidance around DAG authoring patterns (typed RunContext, dependency inference, retries/timeouts, resume/subgraph, hooks).

Changes:

• Introduce flowrun.Pipeline as the primary API and remove Engine/helpers from top-level exports.
• Rework task registration/execution internals to support DAG namespaces (duplicate task names across DAGs) and context injection by annotated parameter name (incl. postponed annotations).
• Update examples/tests/docs to use Pipeline, and add an in-repo authoring guide + agent guidance.

Reviewed changes

Copilot reviewed 22 out of 24 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
uv.lock	Updates locked editable project version (currently mismatched vs pyproject.toml).
pyproject.toml	Bumps project version; ensures py.typed is included in built distributions.
README.md	Rewrites docs around the new Pipeline API, patterns, and public exports.
src/flowrun/init.py	Exports Pipeline and removes top-level Engine helpers from __all__.
src/flowrun/dag.py	Tightens DAG validation (empty DAGs, unknown descendants) and updates DAG building for DAG namespaces.
src/flowrun/engine.py	Adds build() and internal helpers for running built DAGs with isolated registries.
src/flowrun/executor.py	Injects RunContext via the detected annotated parameter name (positional/keyword-only aware).
src/flowrun/hooks.py	Updates docs/examples to reference Pipeline as the primary entrypoint.
src/flowrun/pipeline.py	Adds the new Pipeline implementation (task registration, run helpers, subgraphs, resume, overrides).
src/flowrun/scheduler.py	Supports per-run registries via clone_with_registry() and DAG-aware spec lookup.
src/flowrun/task.py	Adds DAG-aware task registry keys and context parameter detection (incl. postponed annotations).
examples/demo.py	Migrates demo to Pipeline and updates resume/subgraph flows.
examples/micro_batch_demo.py	Migrates micro-batch example to Pipeline.run_many().
examples/polars_workflow_demo.py	Migrates Polars/Pandera example to Pipeline and updated orchestration pattern.
tests/test_hooks.py	Updates hook propagation test to use Pipeline.
tests/test_logging.py	Adds Pipeline logging propagation coverage; keeps internal engine logging test.
tests/test_new_features.py	Adds validation tests for empty DAGs, duplicate names across DAGs, and context injection with deps.
tests/test_partial_dag.py	Adds validation for unknown resume/subgraph seeds.
tests/test_pipeline.py	New test coverage for Pipeline overrides, introspection, reporting, and resume behavior.
tests/test_task_decorator.py	Adds coverage for postponed annotations and DAG-scoped duplicate task names.
tests/test_task_executor.py	Adds coverage for named-deps + context injection by parameter name.
.github/instructions/flowrun-dag-authoring.instructions.md	New authoring guide covering best practices and anti-patterns for Flowrun DAGs.
.github/agents/flowrun-etl.agent.md	New agent guidance for building ETL DAGs with Flowrun conventions.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.