Value Operations Discovery Audit Report

VALUE_OPERATIONS_AUDIT.md

@@ -0,0 +1,193 @@
+# Value Operations Discovery Audit Report
+
+## A. Capability Inventory
+
+This inventory lists all value manipulation capabilities exposed in FlexiRule, categorized by their entry point.
+
+### 1. Structured Value Resolvers (via `ValueResolverControl`)
+| Kind | Name | Description | Sub-Operations |
+| :--- | :--- | :--- | :--- |
+| `date_formula` | Date Formula | Calculate a date by adding or subtracting days, months, or years from a base field or today. | Add/Subtract Days, Months, Years |
+| `date_diff` | Date Difference | Calculate the time difference between two dates in days, months, or years. | Days, Months, Years |
+| `math_formula` | Math Formula | Perform basic arithmetic between two fields or a field and a constant value. | `+`, `-`, `*`, `/` |
+| `child_aggregation` | Child Table Aggregation | Aggregate numeric values from a child table using Sum, Average, or Count. | Sum, Average, Count |
+| `string_formula` | String Manipulation | Combine text fields, change casing, or format currency strings. | Concatenate, Uppercase, Lowercase, Format Currency |
+| `normalization` | Normalization | Clean up text data by trimming whitespace, changing case, or converting to slug/snake case. | Pipeline of 25+ operations (Trim, Slug, Masking, etc.) |
+| `format` | Format | Presentation-level formatting for dates, currency, and templates. | Date/Time, Currency, String Template |
+| `fetch` | Fetch From Link | Fetch a value from a linked document. | Database lookup across DocTypes |
+| `system_context` | System Context | Resolve global values like current user or role status. | Current User, Role Check |
+
+### 2. Formula Registry & Slash Commands (`formula_registry.js`)
+These are exposed as autocomplete suggestions or slash commands in the `FlexValueControl`.
+
+**Slash Commands (Value-Related):**
+- `/formula`: Opens logic selection.
+- `/resolver`: Opens structured resolver.
+- `/formatter`: Opens formatting UI.
+- `/normalize`: Opens normalization UI.
+- `/fetch`: Opens link fetching UI.
+- `/link`: Opens link picker.
+- `/boolean`: Opens toggle UI.
+
+**Formula Registry Items (by Category):**
+- **TEXT:** `concat`, `normalize`, `format`, `replace`, `trim`, `upper`, `lower`, `slug`.
+- **NUMERIC:** `sum`, `round`, `avg`, `min`, `max`, `percentage`, `abs`.
+- **DATE:** `today`, `now`, `add_days`, `add_months`, `date_diff`, `format_date`, `start_of`, `end_of`.
+- **BOOLEAN:** `if`, `equals`, `not`.
+- **LINK:** `lookup`.
+- **TABLE:** `sum`, `count`, `map`, `filter`, `reduce`.
+
+### 3. Normalization Operations (`normalization.py`)
+Exposed primarily through the `NormalizationResolver` pipeline.
+
+- **Cleaning:** `trim`, `remove_spaces`, `remove_extra_spaces`, `remove_punctuation`, `remove_numbers`.
+- **Casing:** `lowercase`, `uppercase`, `casefold`, `title_case`, `name_normalize`.
+- **Transformations:** `slug`, `snake_case`, `unicode_normalize`, `remove_diacritics`, `translate_chars` (Arabic/Persian digits & characters).
+- **Specialized:** `phone_normalize`, `email_normalize`, `currency_to_number`, `tax_id_clean`, `standard_date`.
+- **Masking:** `mask_email`, `mask_phone`, `mask_credit_card`, `mask_partial`.
+
+---
+
+## B. Runtime Mapping
+
+This table traces frontend capabilities to their backend execution logic.
+
+| Frontend Identifier | Backend Implementation Path | Actual Execution Logic |
+| :--- | :--- | :--- |
+| `date_formula` | `ValueResolver.compile` → `DateFormulaResolver.resolve` | `frappe.utils.add_days` or `frappe.utils.add_to_date` |
+| `math_formula` | `ValueResolver.compile` → `MathFormulaResolver.resolve` | `frappe.utils.flt(a op b, precision)` |
+| `date_diff` | `ValueResolver.compile` → `DateDiffResolver.resolve` | `frappe.utils.date_diff` or `frappe.utils.month_diff` |
+| `child_aggregation`| `ValueResolver.compile` → `ChildAggregationResolver.resolve` | List comprehension over `doc.get(table)` with `sum()`/`len()` |
+| `string_formula` | `ValueResolver.compile` → `StringFormulaResolver.resolve` | Python `str.upper()`, `str.lower()`, or `frappe.utils.fmt_money` |
+| `normalization` | `ValueResolver.compile` → `NormalizationResolver.resolve` | `flexirule.ruleflow.utils.normalization.execute_normalization_pipeline` |
+| `format` | `ValueResolver.compile` → `FormatResolver.resolve` | `frappe.utils.format_date`, `frappe.utils.fmt_money`, or `"".format()` |
+| `fetch` | `ValueResolver.compile` → `FetchResolver.resolve` | `frappe.db.get_value` |
+| `system_context` | `ValueResolver.compile` → `SystemContextResolver.resolve` | `frappe.session.user` or `frappe.get_roles` |
+| `today()` (Formula) | `RuleEngine._build_eval_locals` | `frappe.utils.nowdate` mapped as `nowdate` in context |
+| `add_days()` (Formula)| `RuleEngine._build_eval_locals` | `frappe.utils.add_days` mapped as `add_days` in context |
+| `sum()` (Formula) | `ActionHandler._build_template_context` | Python built-in `sum()` |
+| `normalize()` (Jinja)| `ActionHandler._build_template_context` | `flexirule.ruleflow.process.normalization.normalization.apply_transformations` |
+
+---
+
+## C. Duplicate List (Exact Duplicates)
+
+Operations that use identical underlying logic but are exposed under different names or categories:
+
+1.  **Currency Formatting:**
+    - `String Formula` resolver → `fmt_money`
+    - `Format` resolver → `fmt_money`
+    - *Both call `frappe.utils.fmt_money`.*
+2.  **Basic Casing:**
+    - `String Formula` resolver → `uppercase`/`lowercase`
+    - `Normalization` resolver → `uppercase`/`lowercase` pipeline steps
+    - *Both call Python `.upper()` / `.lower()`.*
+3.  **Date Difference:**
+    - `Date Difference` resolver
+    - `/formula` suggestion for `date_diff()`
+    - *Both call `frappe.utils.date_diff`.*
+
+---
+
+## D. Similarity List (Functional Overlaps)
+
+Operations that produce substantially similar results through different mechanisms:
+
+1.  **Concatenation:**
+    - `String Formula` resolver: 2-field visual builder.
+    - `concat()` Formula: Variadic text function.
+    - `String Template` (Format resolver): Uses `"{field_a}{field_b}".format()`.
+2.  **Date Math:**
+    - `Date Formula` resolver: Step-based UI (Add 5 days).
+    - `add_days()` / `add_months()` Formulas: Functional approach.
+3.  **Normalization vs Formulas:**
+    - `trim()`, `upper()`, `lower()`, `slug()` exist both as standalone formulas and as steps in the `Normalization` pipeline.
+4.  **Aggregation:**
+    - `Child Table Aggregation` resolver: Visual UI for Sum/Avg/Count.
+    - `sum()`, `count()`, `len()` formulas: Direct access to Python aggregations.
+
+---
+
+## E. Observations
+
+### 1. Registry vs. Runtime Gaps
+The `formula_registry.js` contains many items (like `map`, `reduce`, `slug`) that are primarily UI "hints." There is no single centralized "Formula Library" on the backend; instead, the `RuleEngine` and `ActionHandler` manually inject various helpers into different evaluation contexts (Python Eval vs. Jinja), leading to inconsistent availability of functions depending on where they are used.
+
+### 2. Architectural Redundancy
+The `String Formula` resolver is largely a subset of the `Format` and `Normalization` resolvers. It appears to be a legacy entry point that has been superseded by more specialized strategies.
+
+### 3. Hidden Backend Power
+The `normalization.py` utility contains sophisticated logic (Arabic character unification, diacritic removal, various masking strategies) that is only fully accessible through the `Normalization` resolver's "Custom Pipeline" mode. These capabilities are not exposed in the simpler `/formula` or `/formatter` registries.
+
+### 4. Implementation Leakage
+Several frontend strategies (e.g., `Aggregation`) "leak" implementation details by compiling complex Python list comprehensions in the frontend `compileToCode` function. This makes it difficult to change the aggregation logic (e.g., adding null checks) without updating the frontend registry.
+
+### 5. Inconsistent Scoping
+The `Fetch` resolver defaults to `doc.` scoping if no scope is provided, while the `Formula` system requires explicit scoping or relies on the evaluator's local resolution logic.
+
+### 6. Special Case Context
+The `Assignment` action handler injects a special `value` variable into the evaluation context specifically for Normalization and Formatting steps. This "current value" awareness is not consistently available across other action types or resolvers.
+
+---
+
+## F. Capability Coverage Matrix
+
+Mapping of unique value operations to their availability across different entry points.
+
+| Capability | Formula | String | Normalize | Format | Date | Math | Aggreg | Fetch | System |
+| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
+| **TEXT** | | | | | | | | | |
+| trim | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
+| lowercase | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
+| uppercase | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
+| title_case | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
+| concat | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
+| slug | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
+| snake_case | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
+| template format | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
+| remove_spaces | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
+| remove_punctuation | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
+| **NUMERIC** | | | | | | | | | |
+| sum | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ |
+| average | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ |
+| count | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ |
+| arithmetic (+,-,*,/) | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ |
+| round | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ |
+| currency format | ✗ | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
+| **DATE/TIME** | | | | | | | | | |
+| today / now | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ |
+| add_days/months | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ |
+| date_diff | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
+| format_date | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
+| **SPECIALIZED** | | | | | | | | | |
+| phone_normalize | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
+| email_normalize | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
+| mask_email/phone | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
+| Arabic unification | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
+| lookup / fetch | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ |
+| current_user | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ |
+| role_check | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ |
+
+---
+
+## G. Matrix Analysis
+
+### 1. Multi-Source Capabilities
+Capabilities like **lowercase**, **uppercase**, **sum**, and **average** are heavily distributed, appearing in 3 or more places. This suggests high demand but inconsistent implementation across the UI.
+
+### 2. Single-Source Capabilities
+- **Masking** and **Arabic unification** are exclusive to the Normalization Resolver.
+- **Arithmetic operators** are exclusive to the Math Resolver (when using the builder) or raw Formulas.
+- **Role checks** are exclusive to the System Resolver.
+- **Fetch** logic is centralized in the Fetch Resolver.
+
+### 3. Resolvers with No Unique Capability
+The **String Formula Resolver** contributes no unique capabilities. Everything it does (concat, upper, lower, fmt_money) is available through either the Normalization Resolver, Format Resolver, or direct Formulas.
+
+### 4. UX Wrappers
+- The **Math Resolver** is essentially a UX wrapper for Python's `flt()` and arithmetic operators.
+- The **Date Resolver** is a UX wrapper for `frappe.utils.add_days` and `add_months`.
+- The **Child Aggregation Resolver** is a UX wrapper for list comprehensions.
+
+### 5. Hidden Capabilities
+The **Normalization Resolver** contains many specialized operations (Phone/Email normalization, Punctuation removal, Masking, Arabic unification) that are powerful but hidden from the general "Format" or "Formula" user journeys. These are currently "locked" behind a specific resolver kind, making them inaccessible for users who don't think to look under "Normalization."

VALUE_OPERATIONS_REFACTOR.md

@@ -0,0 +1,92 @@
+# FlexiRule Value Operations Architecture Refactor
+
+This document outlines the proposed target architecture for Value Operations in FlexiRule, moving from a fragmented "Resolver-Kind" system to a unified, capability-driven "Operation Registry."
+
+---
+
+## 1. Resolver Evaluation Matrix
+
+Challenge the status quo of existing resolver boundaries.
+
+| Resolver Type | Purpose | Unique Capabilities | Overlap | Recommendation | Justification |
+| :--- | :--- | :--- | :--- | :--- | :--- |
+| **String Formula** | Simple text/currency ops | None | Normalization, Format, Formulas | **Remove / Merge** | Redundant. Its logic exists in more specialized or functional entry points. |
+| **Math Formula** | Simple arithmetic | None | Formulas, Python `flt()` | **Remove / Merge** | Pure UX wrapper for basic Python operators. Should be atomic operations in a registry. |
+| **Date Formula** | Date arithmetic | None | Formulas, `frappe.utils` | **Remove / Merge** | UX wrapper. "Add Days" is an operation, not a standalone architecture category. |
+| **Normalization** | Data cleaning/standardization | Arabic unification, specialized cleaning | Casing, slugging | **Keep** | Highly specialized domain with deep logic that shouldn't be diluted into general transformation. |
+| **Format** | Presentation logic | String Templates, Locale-aware formatting | Currency formatting | **Keep** | Solves the specific problem of *display* vs *value*, which is conceptually distinct. |
+| **Child Aggregation** | List math | Table-to-Scalar reduction | `sum`, `count` formulas | **Keep** | Essential for handling list context which requires specific configuration UI. |
+| **Fetch** | Remote data | Cross-DocType retrieval | Lookup formula | **Keep** | Distinct capability requiring unique UI for DocType/Field selection. |
+| **System Context** | Global state | Role checking, session data | None | **Keep** | Unique source of truth (the environment, not the document). |
+
+---
+
+## 2. Proposed Capability Taxonomy
+
+Operations should be grouped by **author intent**, maintaining distinct domains for specialized data handling.
+
+1.  **Text Operations:** Trimming, casing, slugging, snake_case, find/replace, concatenation.
+2.  **Numeric & Math:** Arithmetic, rounding, absolute value, percentages.
+3.  **Date & Time:** Adding/subtracting time units, date differences, relative dates (today, now).
+4.  **Data Normalization:** Cleaning phone numbers, emails, tax IDs, Arabic/Persian unification.
+5.  **Masking & Privacy:** Masking emails, credit cards, or partial strings (security-centric).
+6.  **Formatting & Localization:** Currency formatting, date/time formatting, string templates, locale overrides.
+7.  **Collection & Aggregation:** Sum, average, count, min/max across child tables.
+8.  **Data Retrieval:** Fetching values from linked documents or remote DocTypes.
+9.  **Environment & Context:** Current user, role checks, rule metadata.
+
+---
+
+## 3. Proposed Backend Architecture
+
+**Core Principle:** Registry-bound operations with flexible input signatures.
+
+-   **`flexirule.ruleflow.operations.registry`**: A centralized singleton registry that maps `operation_id` to an implementation class.
+-   **`ValueOperation` (Base Class)**:
+    -   `execute(context, config)`: Standard non-unary interface. Configuration handles all specific inputs (including target values if applicable).
+    -   `get_meta()`: Returns required parameters, return type, and description.
+-   **Module Structure:**
+    -   `operations/text.py`
+    -   `operations/math.py`
+    -   `operations/date.py`
+    -   `operations/normalization.py`
+    -   `operations/masking.py`
+-   **Compiler**: `ValueResolver.compile` simply looks up the `operation_id` in the registry and builds a standard execution call passing the context and pre-parsed config.
+
+---
+
+## 4. Proposed Frontend Architecture
+
+**Core Principle:** Hybrid configuration for optimal UX.
+
+-   **`ValueRegistry`**: A shared registry between `formula_registry.js` and `ValueResolverControl`.
+-   **`OperationBrowser.vue`**: A "Command Palette" style component used in `FlexValueControl` that allows searching all categories at once.
+-   **Hybrid Configuration System**:
+    -   **Metadata-Driven Renderer**: For simple operations (e.g., Round, Trim, Add Days), the UI is automatically generated from the operation's metadata/schema.
+    -   **Dedicated Vue Components**: For complex operations where a generic form is insufficient (e.g., `Aggregation`, `Fetch`, `System Context`, `Template Formatting`), the registry allows mapping an `operation_id` to a specialized Vue configuration component.
+-   **Orchestrator**: A unified modal that determines whether to show the generic renderer or the dedicated component based on the selected operation.
+
+---
+
+## 5. Recommended End-State
+
+The new architecture eliminates the need for users to understand "Resolver Kinds" before they start searching for functionality.
+
+### The Author Experience:
+1.  User clicks into a value field.
+2.  User types `/` to open the **Operation Palette**.
+3.  User searches for what they want to *do* (e.g., "mask", "sum", "days").
+4.  FlexiRule displays a unified list of matches categorized by domain:
+    -   **Add Days** (Date & Time)
+    -   **Mask Email** (Masking & Privacy)
+    -   **Normalize Phone** (Data Normalization)
+5.  Selection opens the configuration flyout:
+    -   *Simple ops:* Use the generic renderer.
+    -   *Complex ops:* Use the dedicated Vue component.
+6.  The token in the editor is an **Operation Token** (e.g., `[Add 5 Days]`).
+
+### Why this is better:
+-   **No "Type-First" Friction:** Users don't need to know if "Slug" is a "Normalization" or a "String Formula."
+-   **Flexible API:** Supporting `execute(context, config)` allows for N-ary operations (Concat), null-input operations (System Context), and complex retrieval (Fetch).
+-   **Maintainability:** Most new operations can be added via metadata alone. Dedicated components are only written when the UX requires a specialized interface.
+-   **Discoverability:** Surfaces powerful hidden capabilities (Arabic Unification, Masking) alongside standard ones.