Discuss workflow macro design and lifecycle

[carloea2]

Feature Summary

Workflow macros are an important feature for Texera, but the current discussion in #5179 mixes several design choices with implementation review. Based on the feedback from @Yicong-Huang and @chenlica, we should move the macro design discussion into an issue, hold a focused live discussion, and record the decisions here before continuing.

References:

• feat: add workflow macro UI support #5179: frontend-oriented workflow macro implementation
• Add workflow macros for importing, grouping, and collapsing reusable sub-workflows #5178: original feature proposal for importing/grouping/collapsing reusable sub-workflows
• [Hackathon] feat: AI-augmented macro operators #5115: hackathon macro-operator prototype with a different UX
• Added operator grouping feature #754: previous operator grouping/collapse behavior

Proposed Solution or Design

Use this issue to align on the macro design before reviewing code in detail. The discussion should decide:

1. Macro creation UX

   • Should users create a macro by selecting existing operators and choosing Create Macro from the context menu?
   • Should users create a macro by importing an existing workflow?
   • Should both flows be supported?

2. Macro representation and reuse

   • Should a macro look and behave like a normal logical operator?
   • Should macro input/output ports be derived from the internal sub-DAG?
   • Should reusable macros appear directly in the operator menu, for example under My Macros?
   • Should the long-term design also be able to represent operators such as Hash Join as macros?

3. Macro internal view and editing

   • Should users edit macro internals in-place on the current canvas, or jump into a separate/drill-down workflow view?
   • Are edits local to one workflow snapshot, or shared across all uses of the macro?
   • Should macro internals expose explicit input/output ports instead of dummy source/sink operators?

4. Runtime and statistics behavior

   • Should macro expansion remain frontend-only, with the backend executing the flattened operator graph?
   • Should the backend be unaware of macros?
   • When collapsed, should a macro show aggregated statistics, internal leaf-operator statistics, or allow users to choose?

5. Macro lifecycle and persistence

   • How should macro definitions and macro instances be stored?
   • How should users remove, rename, disable, search, share, or version macros?
   • What documentation, tutorials, and tests are needed before enabling the feature by default?

Impact / Priority

?

Affected Area

• Workflow UI
• Workflow Engine (Amber)
• Storage / Metadata
• Documentation / Tutorials

[carloea2]

@chenlica @Yicong-Huang @Xiao-zhen-Liu @aglinxinyuan

[Yicong-Huang]

Let me convert this to a Discussion post. sorry if that's not clear. for uncertain topics, we use disucussions. once a consensus is reached, we open an issue (or feel free to use umbrella issue + multiple sub-issues) to track decided todos. Discussions can have back and forth ideas. issues should be clear on what exactly to do.

[Yicong-Huang]

I shared my preferences in #5179 already. If you are having an offline discussion, please report back here.

[carloea2]

In that case, we can just wait for @chenlica and an internal meeting to decide and report the decision here.