Skip to content

observe always returns elementTotal: 0 — UI Automation semantic tree is never populated #1

Description

@HAOYIYU994

Environment

  • Package: @dttxorg/deepseekeyes@0.5.4 (npm latest)
  • DeepSeek Harness: 0.1.0-rc.6
  • OS: Windows 11
  • Helper runtime: Windows PowerShell 5.1 (powershell.exe -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Bypass -File windows.ps1, launched from src/desktop/driver.js)
  • Node: v24.19.0

Summary

On Windows, the computer action observe (with scope: "window") always returns elementTotal: 0 and an empty elements array, no matter which application is targeted. The UI Automation semantic element tree is never delivered to the model, so semanticStatus is always empty/sparse and preferredTargeting is forced to screenshot-coordinates.

Steps to reproduce

  1. Enable Desktop Computer Use.
  2. Run the Windows helper against any window that exposes UI Automation content, e.g. Notepad:
    {
      "action": "observe",
      "scope": "window",
      "captureScope": "window",
      "captureApplication": "Notepad",
      "screenshotPath": "<tmp>/screen.png",
      "maxWindows": 60,
      "semantic": true,
      "maxElements": 200
    }
  3. Read the result: ok: true, capturedWindow resolves correctly, and capabilities.accessibility: true, but elementTotal: 0 and elements: [].

Reproduced against three different application types: msedge (Chromium), SystemSettings (UWP), and Notepad (Win32/WinUI).

Observed vs expected

  • Observed: elementTotal: 0, elements: [], elementsTruncated: false.
  • Expected: elementTotal > 0. An independent UI Automation probe on the same Notepad window returns 227 descendants via AutomationElement.FindAll(TreeScope.Descendants, TrueCondition) — UIA is fully functional on this machine.

Root cause (diagnosis)

  • The elements ARE collected. Instrumenting Add-AutomationChildren shows $output reaches the 200-element cap before the function returns.
  • The failure happens in Get-AutomationElements (src/desktop/helpers/windows.ps1): the final return @($output) throws System.ArgumentException: Argument types do not match when $output is a System.Collections.Generic.List[object] under Windows PowerShell 5.1.
  • The surrounding try { ... } catch { return @() } silently swallows this exception, so the function returns an empty array and the caller reports elementTotal: 0.
  • Minimal repro (Windows PowerShell 5.1):
    $list = New-Object System.Collections.Generic.List[object]
    $list.Add([pscustomobject]@{ role = "window"; name = "x" })
    @($list)   # throws: Argument types do not match

Impact

  • Every elementRef-based action (click, invoke, set_value, perform_action, scroll with elementRef, and the element assertions) cannot resolve any element on Windows, because elementTotal is always 0.
  • The model is forced onto the screenshot-coordinate fallback path, so the advertised "prefer elementRef / accessibility semantic targeting" behavior of Desktop Computer Use 0.5 is effectively disabled on Windows.

Notes

  • The macOS path (macos.jxa) was not tested.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions