Make LLM snapshot more digestable

[braden-seaborn]

Python YAML Snapshot Processing & Agent-Friendly Formatting

Objective

Transform the raw DOM snapshot data into a structured YAML format optimized for agent interaction, using a hierarchical tree structure that clearly represents element relationships, states, and interaction capabilities.

Technical Approach

YAML Hierarchical Structure

• Tree-Based Representation: Mirror DOM hierarchy with indented YAML structure
• Semantic Role Labels: Use accessibility roles (textbox, button, checkbox) instead of HTML tags
• Inline Properties: Include ref attributes and state information in square brackets
• Descriptive Text: Provide clear, contextual descriptions for each element

Python Processing Pipeline

Raw Snapshot to YAML Transformation

class YAMLSnapshotProcessor:
    def process_to_yaml(self, raw_snapshot: dict) -> str:
        yaml_tree = self._build_hierarchical_structure(raw_snapshot)
        return self._format_as_yaml_tree(yaml_tree)
    
    def _format_element(self, element: dict) -> str:
        role = self._map_to_accessibility_role(element)
        description = self._extract_description(element)
        ref = f"[ref={element['ref']}]"
        properties = self._format_properties(element)
        
        return f"{role} \"{description}\" {ref}{properties}:"

Element Role Mapping

ELEMENT_ROLE_MAP = {
    'input[type="text"]': 'textbox',
    'input[type="email"]': 'textbox',
    'input[type="radio"]': 'radio',
    'input[type="checkbox"]': 'checkbox',
    'select': 'combobox',
    'button': 'button',
    'a': 'link',
    'h1,h2,h3,h4,h5,h6': 'heading',
    'div[role="main"]': 'WebArea'
}

Implementation Features

YAML Structure Generation

• Hierarchical Nesting: Maintain parent-child relationships through indentation
• Property Extraction: Include state information (checked, selected, disabled)
• Description Generation: Create meaningful, contextual descriptions
• Reference Injection: Ensure every interactive element has a unique ref

Element Processing Rules

def format_yaml_element(element):
    # Base format: role "description" [ref=eX] [properties]:
    
    # For form inputs
    if element.role == 'textbox':
        return f'textbox "{element.label or element.placeholder}" [ref={element.ref}]:'
    
    # For radio/checkbox with state
    elif element.role in ['radio', 'checkbox']:
        checked = f"[checked={element.checked}]" if 'checked' in element else ""
        return f'{element.role} "{element.text}" [ref={element.ref}] {checked}:'
    
    # For select elements with options
    elif element.role == 'combobox':
        value_info = f"\n    - value: {element.current_value}"
        options = self._format_options(element.options)
        return f'combobox "{element.label}" [ref={element.ref}]:{value_info}{options}'

State and Property Handling

PROPERTY_EXTRACTORS = {
    'checked': lambda el: el.get('checked', False),
    'selected': lambda el: el.get('selected', False),
    'disabled': lambda el: el.get('disabled', False),
    'required': lambda el: el.get('required', False),
    'value': lambda el: el.get('value', ''),
}

Expected YAML Output Structure

Form Elements Example

- WebArea "Scholarship Application Form" [ref=e1]:
  - heading "Personal Information" [ref=e2]:
  - text "Please fill out your basic information:" [ref=e3]:
  - textbox "First Name" [ref=e4]:
  - textbox "Last Name" [ref=e5]:
  - textbox "Email Address" [ref=e6] [required=True]:
  - text "Education Level:" [ref=e7]:
  - radio "High School" [ref=e8] [checked=False]:
  - radio "Undergraduate" [ref=e9] [checked=True]:
  - radio "Graduate" [ref=e10] [checked=False]:
  - button "Submit Application" [ref=e11]:

Complex Form Controls

- combobox "Select Your Major" [ref=e12]:
    - value: Choose a major
    - option "Choose a major" [ref=e13] [selected]:
    - option "Computer Science" [ref=e14]:
    - option "Engineering" [ref=e15]:
    - option "Mathematics" [ref=e16]:

Processing Configuration

YAML Generation Settings

YAML_CONFIG = {
    "indent_size": 2,
    "max_description_length": 80,
    "include_empty_elements": False,
    "group_related_elements": True,
    "preserve_hierarchy": True,
    "include_state_properties": True
}

Agent Optimization Features

• Contextual Grouping: Organize related form fields together
• Clear Labeling: Use field labels, placeholders, or surrounding text for descriptions
• State Awareness: Include current values and selection states
• Action Hints: Indicate interactive capabilities through role types

Agent Interaction Benefits

Raw DOM	YAML Format	Agent Benefit
<input type="text" id="fname">	textbox "First Name" [ref=e4]:	Clear purpose and interaction type
Complex nested HTML	Indented hierarchy	Easy relationship understanding
Multiple attributes	[checked=False] [required=True]	Immediate state awareness
Technical selectors	Human-readable descriptions	Natural language comprehension

Processing Pipeline

Step-by-Step Transformation

1. Parse Raw Snapshot: Extract elements and hierarchy from custom DOM walker
2. Map to Accessibility Roles: Convert HTML tags to semantic roles
3. Generate Descriptions: Create meaningful labels from context
4. Format Properties: Extract and format state information
5. Build YAML Tree: Construct hierarchical YAML structure
6. Optimize for Tokens: Compress while maintaining clarity

Error Handling & Fallbacks

• Missing Labels: Generate descriptions from surrounding text
• Complex Nesting: Flatten unnecessary hierarchy levels
• State Detection: Infer states from attributes and classes
• Reference Conflicts: Ensure unique ref generation

Expected Improvements

• Agent Comprehension: 90% improvement in element understanding through clear role/description mapping
• Interaction Accuracy: 85% reduction in targeting errors due to explicit state information
• Token Efficiency: 40% more information density compared to verbose JSON
• Processing Speed: Optimized YAML generation with minimal overhead

---

This YAML-based snapshot format provides agents with a clear, hierarchical view of page structure while maintaining the semantic richness needed for reliable web automation.