diff --git a/VALUE_OPERATIONS_AUDIT.md b/VALUE_OPERATIONS_AUDIT.md new file mode 100644 index 00000000..22078c04 --- /dev/null +++ b/VALUE_OPERATIONS_AUDIT.md @@ -0,0 +1,193 @@ +# Value Operations Discovery Audit Report + +## A. Capability Inventory + +This inventory lists all value manipulation capabilities exposed in FlexiRule, categorized by their entry point. + +### 1. Structured Value Resolvers (via `ValueResolverControl`) +| Kind | Name | Description | Sub-Operations | +| :--- | :--- | :--- | :--- | +| `date_formula` | Date Formula | Calculate a date by adding or subtracting days, months, or years from a base field or today. | Add/Subtract Days, Months, Years | +| `date_diff` | Date Difference | Calculate the time difference between two dates in days, months, or years. | Days, Months, Years | +| `math_formula` | Math Formula | Perform basic arithmetic between two fields or a field and a constant value. | `+`, `-`, `*`, `/` | +| `child_aggregation` | Child Table Aggregation | Aggregate numeric values from a child table using Sum, Average, or Count. | Sum, Average, Count | +| `string_formula` | String Manipulation | Combine text fields, change casing, or format currency strings. | Concatenate, Uppercase, Lowercase, Format Currency | +| `normalization` | Normalization | Clean up text data by trimming whitespace, changing case, or converting to slug/snake case. | Pipeline of 25+ operations (Trim, Slug, Masking, etc.) | +| `format` | Format | Presentation-level formatting for dates, currency, and templates. | Date/Time, Currency, String Template | +| `fetch` | Fetch From Link | Fetch a value from a linked document. | Database lookup across DocTypes | +| `system_context` | System Context | Resolve global values like current user or role status. | Current User, Role Check | + +### 2. Formula Registry & Slash Commands (`formula_registry.js`) +These are exposed as autocomplete suggestions or slash commands in the `FlexValueControl`. + +**Slash Commands (Value-Related):** +- `/formula`: Opens logic selection. +- `/resolver`: Opens structured resolver. +- `/formatter`: Opens formatting UI. +- `/normalize`: Opens normalization UI. +- `/fetch`: Opens link fetching UI. +- `/link`: Opens link picker. +- `/boolean`: Opens toggle UI. + +**Formula Registry Items (by Category):** +- **TEXT:** `concat`, `normalize`, `format`, `replace`, `trim`, `upper`, `lower`, `slug`. +- **NUMERIC:** `sum`, `round`, `avg`, `min`, `max`, `percentage`, `abs`. +- **DATE:** `today`, `now`, `add_days`, `add_months`, `date_diff`, `format_date`, `start_of`, `end_of`. +- **BOOLEAN:** `if`, `equals`, `not`. +- **LINK:** `lookup`. +- **TABLE:** `sum`, `count`, `map`, `filter`, `reduce`. + +### 3. Normalization Operations (`normalization.py`) +Exposed primarily through the `NormalizationResolver` pipeline. + +- **Cleaning:** `trim`, `remove_spaces`, `remove_extra_spaces`, `remove_punctuation`, `remove_numbers`. +- **Casing:** `lowercase`, `uppercase`, `casefold`, `title_case`, `name_normalize`. +- **Transformations:** `slug`, `snake_case`, `unicode_normalize`, `remove_diacritics`, `translate_chars` (Arabic/Persian digits & characters). +- **Specialized:** `phone_normalize`, `email_normalize`, `currency_to_number`, `tax_id_clean`, `standard_date`. +- **Masking:** `mask_email`, `mask_phone`, `mask_credit_card`, `mask_partial`. + +--- + +## B. Runtime Mapping + +This table traces frontend capabilities to their backend execution logic. + +| Frontend Identifier | Backend Implementation Path | Actual Execution Logic | +| :--- | :--- | :--- | +| `date_formula` | `ValueResolver.compile` → `DateFormulaResolver.resolve` | `frappe.utils.add_days` or `frappe.utils.add_to_date` | +| `math_formula` | `ValueResolver.compile` → `MathFormulaResolver.resolve` | `frappe.utils.flt(a op b, precision)` | +| `date_diff` | `ValueResolver.compile` → `DateDiffResolver.resolve` | `frappe.utils.date_diff` or `frappe.utils.month_diff` | +| `child_aggregation`| `ValueResolver.compile` → `ChildAggregationResolver.resolve` | List comprehension over `doc.get(table)` with `sum()`/`len()` | +| `string_formula` | `ValueResolver.compile` → `StringFormulaResolver.resolve` | Python `str.upper()`, `str.lower()`, or `frappe.utils.fmt_money` | +| `normalization` | `ValueResolver.compile` → `NormalizationResolver.resolve` | `flexirule.ruleflow.utils.normalization.execute_normalization_pipeline` | +| `format` | `ValueResolver.compile` → `FormatResolver.resolve` | `frappe.utils.format_date`, `frappe.utils.fmt_money`, or `"".format()` | +| `fetch` | `ValueResolver.compile` → `FetchResolver.resolve` | `frappe.db.get_value` | +| `system_context` | `ValueResolver.compile` → `SystemContextResolver.resolve` | `frappe.session.user` or `frappe.get_roles` | +| `today()` (Formula) | `RuleEngine._build_eval_locals` | `frappe.utils.nowdate` mapped as `nowdate` in context | +| `add_days()` (Formula)| `RuleEngine._build_eval_locals` | `frappe.utils.add_days` mapped as `add_days` in context | +| `sum()` (Formula) | `ActionHandler._build_template_context` | Python built-in `sum()` | +| `normalize()` (Jinja)| `ActionHandler._build_template_context` | `flexirule.ruleflow.process.normalization.normalization.apply_transformations` | + +--- + +## C. Duplicate List (Exact Duplicates) + +Operations that use identical underlying logic but are exposed under different names or categories: + +1. **Currency Formatting:** + - `String Formula` resolver → `fmt_money` + - `Format` resolver → `fmt_money` + - *Both call `frappe.utils.fmt_money`.* +2. **Basic Casing:** + - `String Formula` resolver → `uppercase`/`lowercase` + - `Normalization` resolver → `uppercase`/`lowercase` pipeline steps + - *Both call Python `.upper()` / `.lower()`.* +3. **Date Difference:** + - `Date Difference` resolver + - `/formula` suggestion for `date_diff()` + - *Both call `frappe.utils.date_diff`.* + +--- + +## D. Similarity List (Functional Overlaps) + +Operations that produce substantially similar results through different mechanisms: + +1. **Concatenation:** + - `String Formula` resolver: 2-field visual builder. + - `concat()` Formula: Variadic text function. + - `String Template` (Format resolver): Uses `"{field_a}{field_b}".format()`. +2. **Date Math:** + - `Date Formula` resolver: Step-based UI (Add 5 days). + - `add_days()` / `add_months()` Formulas: Functional approach. +3. **Normalization vs Formulas:** + - `trim()`, `upper()`, `lower()`, `slug()` exist both as standalone formulas and as steps in the `Normalization` pipeline. +4. **Aggregation:** + - `Child Table Aggregation` resolver: Visual UI for Sum/Avg/Count. + - `sum()`, `count()`, `len()` formulas: Direct access to Python aggregations. + +--- + +## E. Observations + +### 1. Registry vs. Runtime Gaps +The `formula_registry.js` contains many items (like `map`, `reduce`, `slug`) that are primarily UI "hints." There is no single centralized "Formula Library" on the backend; instead, the `RuleEngine` and `ActionHandler` manually inject various helpers into different evaluation contexts (Python Eval vs. Jinja), leading to inconsistent availability of functions depending on where they are used. + +### 2. Architectural Redundancy +The `String Formula` resolver is largely a subset of the `Format` and `Normalization` resolvers. It appears to be a legacy entry point that has been superseded by more specialized strategies. + +### 3. Hidden Backend Power +The `normalization.py` utility contains sophisticated logic (Arabic character unification, diacritic removal, various masking strategies) that is only fully accessible through the `Normalization` resolver's "Custom Pipeline" mode. These capabilities are not exposed in the simpler `/formula` or `/formatter` registries. + +### 4. Implementation Leakage +Several frontend strategies (e.g., `Aggregation`) "leak" implementation details by compiling complex Python list comprehensions in the frontend `compileToCode` function. This makes it difficult to change the aggregation logic (e.g., adding null checks) without updating the frontend registry. + +### 5. Inconsistent Scoping +The `Fetch` resolver defaults to `doc.` scoping if no scope is provided, while the `Formula` system requires explicit scoping or relies on the evaluator's local resolution logic. + +### 6. Special Case Context +The `Assignment` action handler injects a special `value` variable into the evaluation context specifically for Normalization and Formatting steps. This "current value" awareness is not consistently available across other action types or resolvers. + +--- + +## F. Capability Coverage Matrix + +Mapping of unique value operations to their availability across different entry points. + +| Capability | Formula | String | Normalize | Format | Date | Math | Aggreg | Fetch | System | +| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | +| **TEXT** | | | | | | | | | | +| trim | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | +| lowercase | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | +| uppercase | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | +| title_case | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | +| concat | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | +| slug | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | +| snake_case | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | +| template format | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | +| remove_spaces | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | +| remove_punctuation | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | +| **NUMERIC** | | | | | | | | | | +| sum | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | +| average | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | +| count | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | +| arithmetic (+,-,*,/) | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | +| round | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | +| currency format | ✗ | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | +| **DATE/TIME** | | | | | | | | | | +| today / now | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | +| add_days/months | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | +| date_diff | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | +| format_date | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | +| **SPECIALIZED** | | | | | | | | | | +| phone_normalize | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | +| email_normalize | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | +| mask_email/phone | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | +| Arabic unification | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | +| lookup / fetch | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | +| current_user | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | +| role_check | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | + +--- + +## G. Matrix Analysis + +### 1. Multi-Source Capabilities +Capabilities like **lowercase**, **uppercase**, **sum**, and **average** are heavily distributed, appearing in 3 or more places. This suggests high demand but inconsistent implementation across the UI. + +### 2. Single-Source Capabilities +- **Masking** and **Arabic unification** are exclusive to the Normalization Resolver. +- **Arithmetic operators** are exclusive to the Math Resolver (when using the builder) or raw Formulas. +- **Role checks** are exclusive to the System Resolver. +- **Fetch** logic is centralized in the Fetch Resolver. + +### 3. Resolvers with No Unique Capability +The **String Formula Resolver** contributes no unique capabilities. Everything it does (concat, upper, lower, fmt_money) is available through either the Normalization Resolver, Format Resolver, or direct Formulas. + +### 4. UX Wrappers +- The **Math Resolver** is essentially a UX wrapper for Python's `flt()` and arithmetic operators. +- The **Date Resolver** is a UX wrapper for `frappe.utils.add_days` and `add_months`. +- The **Child Aggregation Resolver** is a UX wrapper for list comprehensions. + +### 5. Hidden Capabilities +The **Normalization Resolver** contains many specialized operations (Phone/Email normalization, Punctuation removal, Masking, Arabic unification) that are powerful but hidden from the general "Format" or "Formula" user journeys. These are currently "locked" behind a specific resolver kind, making them inaccessible for users who don't think to look under "Normalization." diff --git a/VALUE_OPERATIONS_REFACTOR.md b/VALUE_OPERATIONS_REFACTOR.md new file mode 100644 index 00000000..852f1d4e --- /dev/null +++ b/VALUE_OPERATIONS_REFACTOR.md @@ -0,0 +1,92 @@ +# FlexiRule Value Operations Architecture Refactor + +This document outlines the proposed target architecture for Value Operations in FlexiRule, moving from a fragmented "Resolver-Kind" system to a unified, capability-driven "Operation Registry." + +--- + +## 1. Resolver Evaluation Matrix + +Challenge the status quo of existing resolver boundaries. + +| Resolver Type | Purpose | Unique Capabilities | Overlap | Recommendation | Justification | +| :--- | :--- | :--- | :--- | :--- | :--- | +| **String Formula** | Simple text/currency ops | None | Normalization, Format, Formulas | **Remove / Merge** | Redundant. Its logic exists in more specialized or functional entry points. | +| **Math Formula** | Simple arithmetic | None | Formulas, Python `flt()` | **Remove / Merge** | Pure UX wrapper for basic Python operators. Should be atomic operations in a registry. | +| **Date Formula** | Date arithmetic | None | Formulas, `frappe.utils` | **Remove / Merge** | UX wrapper. "Add Days" is an operation, not a standalone architecture category. | +| **Normalization** | Data cleaning/standardization | Arabic unification, specialized cleaning | Casing, slugging | **Keep** | Highly specialized domain with deep logic that shouldn't be diluted into general transformation. | +| **Format** | Presentation logic | String Templates, Locale-aware formatting | Currency formatting | **Keep** | Solves the specific problem of *display* vs *value*, which is conceptually distinct. | +| **Child Aggregation** | List math | Table-to-Scalar reduction | `sum`, `count` formulas | **Keep** | Essential for handling list context which requires specific configuration UI. | +| **Fetch** | Remote data | Cross-DocType retrieval | Lookup formula | **Keep** | Distinct capability requiring unique UI for DocType/Field selection. | +| **System Context** | Global state | Role checking, session data | None | **Keep** | Unique source of truth (the environment, not the document). | + +--- + +## 2. Proposed Capability Taxonomy + +Operations should be grouped by **author intent**, maintaining distinct domains for specialized data handling. + +1. **Text Operations:** Trimming, casing, slugging, snake_case, find/replace, concatenation. +2. **Numeric & Math:** Arithmetic, rounding, absolute value, percentages. +3. **Date & Time:** Adding/subtracting time units, date differences, relative dates (today, now). +4. **Data Normalization:** Cleaning phone numbers, emails, tax IDs, Arabic/Persian unification. +5. **Masking & Privacy:** Masking emails, credit cards, or partial strings (security-centric). +6. **Formatting & Localization:** Currency formatting, date/time formatting, string templates, locale overrides. +7. **Collection & Aggregation:** Sum, average, count, min/max across child tables. +8. **Data Retrieval:** Fetching values from linked documents or remote DocTypes. +9. **Environment & Context:** Current user, role checks, rule metadata. + +--- + +## 3. Proposed Backend Architecture + +**Core Principle:** Registry-bound operations with flexible input signatures. + +- **`flexirule.ruleflow.operations.registry`**: A centralized singleton registry that maps `operation_id` to an implementation class. +- **`ValueOperation` (Base Class)**: + - `execute(context, config)`: Standard non-unary interface. Configuration handles all specific inputs (including target values if applicable). + - `get_meta()`: Returns required parameters, return type, and description. +- **Module Structure:** + - `operations/text.py` + - `operations/math.py` + - `operations/date.py` + - `operations/normalization.py` + - `operations/masking.py` +- **Compiler**: `ValueResolver.compile` simply looks up the `operation_id` in the registry and builds a standard execution call passing the context and pre-parsed config. + +--- + +## 4. Proposed Frontend Architecture + +**Core Principle:** Hybrid configuration for optimal UX. + +- **`ValueRegistry`**: A shared registry between `formula_registry.js` and `ValueResolverControl`. +- **`OperationBrowser.vue`**: A "Command Palette" style component used in `FlexValueControl` that allows searching all categories at once. +- **Hybrid Configuration System**: + - **Metadata-Driven Renderer**: For simple operations (e.g., Round, Trim, Add Days), the UI is automatically generated from the operation's metadata/schema. + - **Dedicated Vue Components**: For complex operations where a generic form is insufficient (e.g., `Aggregation`, `Fetch`, `System Context`, `Template Formatting`), the registry allows mapping an `operation_id` to a specialized Vue configuration component. +- **Orchestrator**: A unified modal that determines whether to show the generic renderer or the dedicated component based on the selected operation. + +--- + +## 5. Recommended End-State + +The new architecture eliminates the need for users to understand "Resolver Kinds" before they start searching for functionality. + +### The Author Experience: +1. User clicks into a value field. +2. User types `/` to open the **Operation Palette**. +3. User searches for what they want to *do* (e.g., "mask", "sum", "days"). +4. FlexiRule displays a unified list of matches categorized by domain: + - **Add Days** (Date & Time) + - **Mask Email** (Masking & Privacy) + - **Normalize Phone** (Data Normalization) +5. Selection opens the configuration flyout: + - *Simple ops:* Use the generic renderer. + - *Complex ops:* Use the dedicated Vue component. +6. The token in the editor is an **Operation Token** (e.g., `[Add 5 Days]`). + +### Why this is better: +- **No "Type-First" Friction:** Users don't need to know if "Slug" is a "Normalization" or a "String Formula." +- **Flexible API:** Supporting `execute(context, config)` allows for N-ary operations (Concat), null-input operations (System Context), and complex retrieval (Fetch). +- **Maintainability:** Most new operations can be added via metadata alone. Dedicated components are only written when the UX requires a specialized interface. +- **Discoverability:** Surfaces powerful hidden capabilities (Arabic Unification, Masking) alongside standard ones.