Environment
- Package:
@dttxorg/deepseekeyes@0.5.4 (npm latest)
- DeepSeek Harness:
0.1.0-rc.6
- OS: Windows 11
- Helper runtime: Windows PowerShell 5.1 (
powershell.exe -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Bypass -File windows.ps1, launched from src/desktop/driver.js)
- Node:
v24.19.0
Summary
On Windows, the computer action observe (with scope: "window") always returns elementTotal: 0 and an empty elements array, no matter which application is targeted. The UI Automation semantic element tree is never delivered to the model, so semanticStatus is always empty/sparse and preferredTargeting is forced to screenshot-coordinates.
Steps to reproduce
- Enable Desktop Computer Use.
- Run the Windows helper against any window that exposes UI Automation content, e.g. Notepad:
{
"action": "observe",
"scope": "window",
"captureScope": "window",
"captureApplication": "Notepad",
"screenshotPath": "<tmp>/screen.png",
"maxWindows": 60,
"semantic": true,
"maxElements": 200
}
- Read the result:
ok: true, capturedWindow resolves correctly, and capabilities.accessibility: true, but elementTotal: 0 and elements: [].
Reproduced against three different application types: msedge (Chromium), SystemSettings (UWP), and Notepad (Win32/WinUI).
Observed vs expected
- Observed:
elementTotal: 0, elements: [], elementsTruncated: false.
- Expected:
elementTotal > 0. An independent UI Automation probe on the same Notepad window returns 227 descendants via AutomationElement.FindAll(TreeScope.Descendants, TrueCondition) — UIA is fully functional on this machine.
Root cause (diagnosis)
- The elements ARE collected. Instrumenting
Add-AutomationChildren shows $output reaches the 200-element cap before the function returns.
- The failure happens in
Get-AutomationElements (src/desktop/helpers/windows.ps1): the final return @($output) throws System.ArgumentException: Argument types do not match when $output is a System.Collections.Generic.List[object] under Windows PowerShell 5.1.
- The surrounding
try { ... } catch { return @() } silently swallows this exception, so the function returns an empty array and the caller reports elementTotal: 0.
- Minimal repro (Windows PowerShell 5.1):
$list = New-Object System.Collections.Generic.List[object]
$list.Add([pscustomobject]@{ role = "window"; name = "x" })
@($list) # throws: Argument types do not match
Impact
- Every
elementRef-based action (click, invoke, set_value, perform_action, scroll with elementRef, and the element assertions) cannot resolve any element on Windows, because elementTotal is always 0.
- The model is forced onto the screenshot-coordinate fallback path, so the advertised "prefer elementRef / accessibility semantic targeting" behavior of Desktop Computer Use 0.5 is effectively disabled on Windows.
Notes
- The macOS path (
macos.jxa) was not tested.
Environment
@dttxorg/deepseekeyes@0.5.4(npmlatest)0.1.0-rc.6powershell.exe -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Bypass -File windows.ps1, launched fromsrc/desktop/driver.js)v24.19.0Summary
On Windows, the
computeractionobserve(withscope: "window") always returnselementTotal: 0and an emptyelementsarray, no matter which application is targeted. The UI Automation semantic element tree is never delivered to the model, sosemanticStatusis alwaysempty/sparseandpreferredTargetingis forced toscreenshot-coordinates.Steps to reproduce
{ "action": "observe", "scope": "window", "captureScope": "window", "captureApplication": "Notepad", "screenshotPath": "<tmp>/screen.png", "maxWindows": 60, "semantic": true, "maxElements": 200 }ok: true,capturedWindowresolves correctly, andcapabilities.accessibility: true, butelementTotal: 0andelements: [].Reproduced against three different application types:
msedge(Chromium),SystemSettings(UWP), andNotepad(Win32/WinUI).Observed vs expected
elementTotal: 0,elements: [],elementsTruncated: false.elementTotal> 0. An independent UI Automation probe on the same Notepad window returns 227 descendants viaAutomationElement.FindAll(TreeScope.Descendants, TrueCondition)— UIA is fully functional on this machine.Root cause (diagnosis)
Add-AutomationChildrenshows$outputreaches the 200-element cap before the function returns.Get-AutomationElements(src/desktop/helpers/windows.ps1): the finalreturn @($output)throwsSystem.ArgumentException: Argument types do not matchwhen$outputis aSystem.Collections.Generic.List[object]under Windows PowerShell 5.1.try { ... } catch { return @() }silently swallows this exception, so the function returns an empty array and the caller reportselementTotal: 0.Impact
elementRef-based action (click,invoke,set_value,perform_action,scrollwithelementRef, and the element assertions) cannot resolve any element on Windows, becauseelementTotalis always 0.Notes
macos.jxa) was not tested.