Skip to content

Feature/api rework#1

Merged
Mg30 merged 3 commits into
mainfrom
feature/api-rework
Apr 2, 2026
Merged

Feature/api rework#1
Mg30 merged 3 commits into
mainfrom
feature/api-rework

Conversation

@Mg30
Copy link
Copy Markdown
Owner

@Mg30 Mg30 commented Apr 2, 2026

This pull request significantly improves the documentation and examples for flowrun, clarifying its positioning, strengths, and tradeoffs, while also simplifying and updating the usage patterns. It introduces a new micro-batch ETL example and removes deprecated or confusing features from both the documentation and code examples. The overall goal is to make the framework's intended use cases, limitations, and API surface much clearer for users.

Documentation and Positioning Updates:

  • Expanded the README.md to clarify flowrun's intended use cases, strengths, and tradeoffs, positioning it as a lightweight, in-process DAG engine for small to medium ETL jobs, not as a distributed scheduler or orchestration platform.
  • Added sections on micro-batch and layered workflow patterns, and improved explanations for dependency injection, run metadata, and task registration. [1] [2] [3]

API and Usage Simplification:

  • Removed deprecated retain_result, task_template, and global @task decorator usage from both documentation and examples, focusing on the preferred, simpler API. [1] [2] Faaeb5a6L225R276, [3] [4] [5] [6] [7]
  • Clarified and simplified task registration: task names now default to the function name unless an explicit alias is needed, and dependencies can be inferred from parameter names. [1] Faaeb5a6L194R215, Faaeb5a6L225R276, [2]

New and Improved Examples:

  • Added a new examples/micro_batch_demo.py illustrating the micro-batch pattern, where the DAG is run once per chunk from an external async source, with clear separation between chunk fetching and DAG execution.
  • Updated other examples to use the simplified task registration and dependency injection patterns. [1] [2] [3] [4]

Clarification of Execution Semantics:

  • Clarified timeout semantics: only async tasks support framework-level timeouts; sync tasks should handle timeouts in user code. (Faaeb5a6L194R215, Faaeb5a6L340R352)
  • Improved documentation of dependency validation, result injection, and run reporting. [1] [2]

Removal of Deprecated Features:

  • Removed documentation and references to the SQLite state store and custom result serializers, focusing on the default in-memory store for simplicity. [1] [2]

These changes make the documentation and examples more focused, easier to follow, and better aligned with the intended use cases for flowrun.

@Mg30 Mg30 merged commit 41dfe34 into main Apr 2, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant