Fallback Parser Contains Too Much Parsing Logic

[jac18281828]

Problem Description

The current fallback.rs module has grown into a complex secondary parsing system that duplicates and sometimes contradicts the main Pest grammar. This creates several issues:

Issues with Current Approach

1. Duplicate Parsing Logic: The fallback parser reimplements parsing for:

   • Template patterns (${variable}, $variable, {content})
   • Boolean expressions with AND/OR operators
   • Field selectors and path parsing
   • Filter expressions with comparison operators
   • Manual string splitting and tokenization

2. Inconsistent Behavior: Having two parsing systems can lead to:

   • Different interpretations of the same syntax
   • Edge cases that behave differently between main parser and fallback
   • Maintenance burden of keeping both systems in sync

3. Complex State Management: The fallback parser includes intricate logic for:

   • Quote tracking and nested bracket/brace handling
   • Manual tokenization and operator detection
   • Complex regex-like pattern matching for template detection

4. Overly Permissive: The fallback parser accepts many patterns that may not be intended:

   • Manual parsing of filter+template combinations
   • Heuristic-based detection of expression types
   • Multiple fallback strategies that may mask grammar issues

Current Fallback Functionality

The fallback parser currently handles:

• Simple Template Patterns: $name, ${name}, {content with ${vars}}
• Field Selectors: user.email, "field with spaces"
• Boolean Expressions: field1? && field2?, !suspended
• Manual Parsing: Complex filter+template combinations like age > 25 {Hello ${name}}
• Operator Detection: Parsing comparison operators manually
• Quote and Bracket Tracking: Manual lexical analysis

Proposed Solutions

Option 1: Push Functionality into Pest Grammar (Recommended)

Move legitimate parsing cases into the main Pest grammar:

// Extend the main grammar to handle currently unsupported cases
template_simple = { "${" ~ field_name ~ "}" | "$" ~ field_name }
boolean_expr = { truthy_field ~ ("&&" | "||") ~ truthy_field }
truthy_field = { field_name ~ "?" | "!" ~ field_name }

Benefits:

• Single source of truth for parsing rules
• Consistent error messages and behavior
• Better performance (compiled parser vs runtime parsing)
• Easier to test and maintain

Option 2: Eliminate Unsupported Syntax

Remove fallback parsing entirely and require users to use only grammar-supported syntax:

Benefits:

• Simpler codebase
• Clear boundaries of what's supported
• Forces users to learn the "correct" syntax
• Eliminates maintenance burden

Drawbacks:

• May break existing user workflows
• Less user-friendly for edge cases

Option 3: Minimal Fallback (Compromise)

Keep only the most essential fallback cases:

• Simple field selectors (name, user.email)
• Basic variable substitution ($name)
• Remove complex manual parsing logic

Recommended Approach

1. Audit Current Usage: Identify which fallback patterns are actually used by analyzing logs/metrics
2. Extend Pest Grammar: Move commonly used patterns into the main grammar
3. Remove Complex Fallback Logic: Eliminate manual parsing, operator detection, and quote tracking
4. Keep Minimal Safety Net: Only handle truly simple cases like bare field names

Files Affected

• src/dsl/fallback.rs - Main fallback parsing logic (482 lines)
• src/dsl/grammar.rs - Main Pest parser integration
• pest/parsm.pest - Pest grammar file
• Tests in tests/dsl_integration_test.rs

Implementation Steps

1. Create comprehensive test suite for current fallback behavior
2. Identify which patterns should be moved to grammar vs eliminated
3. Extend Pest grammar for approved patterns
4. Gradually remove fallback logic
5. Update error messages to guide users to supported syntax
6. Update documentation with clear syntax examples

Priority

High - This architectural issue affects maintainability, performance, and user experience. The dual parsing system creates technical debt and potential inconsistencies.