Snapshot Fallback & Refs for DOM based interactions

[braden-seaborn]

Snapshot Reference System Enhancement

Problem Statement

While we previously removed the snapshot reference system due to its complexity and various technical challenges, we've identified a critical limitation in our current approach: when multiple components share the same role and name, our agent cannot distinguish between them, completely breaking interaction capabilities.

Proposed Solution

1. Custom DOM Snapshot System

• Approach: Build a ground-up solution that walks the entire DOM tree

• Process:

  • Traverse DOM nodes systematically

  • Filter elements based on configurable validation rules

  • Inject unique ref attributes into qualifying elements

  • Generate comprehensive JSON snapshot

• Benefit: Provides deterministic element identification regardless of duplicate selectors

2. Configuration-Driven Element Selection

• Flexibility: Configurable rules determine which elements are captured

• Precision: Avoid noise by only tracking interaction-relevant elements

• Adaptability: Easy to modify selection criteria for different website types

Technical Requirements

Core Features

• Implement DOM walker with configurable element filtering

• Inject unique reference attributes into valid elements

• Generate structured JSON snapshot output

• Maintain low token usage through selective element capture

• Support dynamic DOM updates and re-snapshotting

Snapshot Structure

json
{
  "timestamp": "2025-09-02T02:01:00Z",
  "elements": [
    {
      "ref": "elem_001",
      "tag": "button",
      "selector": ".submit-btn",
      "text": "Submit Application",
      "attributes": {...},
      "position": {...}
    }
  ]
}

Configuration Schema

• Define element selection rules (tags, classes, roles, etc.)

• Set filtering criteria for interactive vs. static elements

• Configure snapshot depth and scope limits

Expected Outcomes

Metric | Before | After -- | -- | -- Component Disambiguation | ❌ Fails with duplicates | ✅ Unique refs solve this Dynamic Site Support | ⚠️ Limited | ✅ Re-snapshot capability Token Efficiency | ✅ Optimized | ✅ Config-driven selection Reliability | ⚠️ Moderate | ✅ Deterministic refs

Implementation Priority

High Priority: This enhancement is essential for agent reliability and should be prioritized to resolve current interaction limitations.

Architecture Benefit: Custom-built solution gives us complete control over the snapshot process, allowing optimization for our specific use cases.

---

This ground-up approach provides the robustness we need while maintaining full control over performance and token usage optimization.