From 6faaf7764488d7000bfda925d6a37c987c3b8768 Mon Sep 17 00:00:00 2001 From: John Lambert Date: Thu, 21 May 2026 13:41:24 -0400 Subject: [PATCH 1/2] LT-22324: add FieldWorks UI automation support --- .github/skills/fieldworks-winapp/SKILL.md | 114 ++++++++++++++ .../navigation/launch-or-attach.md | 66 ++++++++ .../navigation/project-loading.md | 40 +++++ .../navigation/screenshot-evidence.md | 72 +++++++++ .../navigation/styles-font-tab.md | 52 +++++++ .../navigation/writing-system-font-options.md | 65 ++++++++ .../references/how-to-update.md | 84 +++++++++++ .../references/mcp-selection.md | 67 +++++++++ .../fieldworks-winapp/references/research.md | 74 +++++++++ .../skills/smart-screenshot-capture/SKILL.md | 141 ++++++++++++++++++ .vscode/mcp.json | 13 ++ Docs/mcp.md | 46 +++++- 12 files changed, 828 insertions(+), 6 deletions(-) create mode 100644 .github/skills/fieldworks-winapp/SKILL.md create mode 100644 .github/skills/fieldworks-winapp/navigation/launch-or-attach.md create mode 100644 .github/skills/fieldworks-winapp/navigation/project-loading.md create mode 100644 .github/skills/fieldworks-winapp/navigation/screenshot-evidence.md create mode 100644 .github/skills/fieldworks-winapp/navigation/styles-font-tab.md create mode 100644 .github/skills/fieldworks-winapp/navigation/writing-system-font-options.md create mode 100644 .github/skills/fieldworks-winapp/references/how-to-update.md create mode 100644 .github/skills/fieldworks-winapp/references/mcp-selection.md create mode 100644 .github/skills/fieldworks-winapp/references/research.md create mode 100644 .github/skills/smart-screenshot-capture/SKILL.md diff --git a/.github/skills/fieldworks-winapp/SKILL.md b/.github/skills/fieldworks-winapp/SKILL.md new file mode 100644 index 0000000000..6fb159bc6b --- /dev/null +++ b/.github/skills/fieldworks-winapp/SKILL.md @@ -0,0 +1,114 @@ +--- +name: fieldworks-winapp +description: > + Control and document the FieldWorks desktop application with WinForms MCP or WinApp MCP. + Use this skill whenever a task requires launching FieldWorks, restoring or + opening a FieldWorks project, walking WinForms UI flows, collecting manual + screenshots, reproducing UI bugs, or verifying a fix inside the live FLEx + application. +license: MIT +compatibility: > + Requires a Windows/x64 FieldWorks workspace. Prefer WinForms MCP + for hidden-desktop UIA2 automation when available; use WinApp MCP as the UIA3 + fallback and for visible desktop/window diagnostics. +metadata: + author: FieldWorks team + version: "1.0" +--- + +# FieldWorks WinForms/WinApp Automation + +Use this skill to launch, inspect, navigate, and capture evidence from the +FieldWorks Language Explorer desktop app through WinForms MCP or WinApp MCP. +Keep the workflow grounded in the live UI tree: inspect first, interact second, +capture evidence after the target state is visible. + +This skill is intentionally organized as a small index plus route-specific +navigation files. Read only the route files needed for the task. + +## Core Rules + +- Prefer WinForms MCP UIA2 tools for fresh FieldWorks runtime automation because + FieldWorks is currently WinForms and the workspace config runs WinForms MCP in + hidden-desktop mode. +- Use WinApp MCP UIA3 tools when WinForms MCP is unavailable, when a task needs + visible desktop/window diagnostics, or when UIA2 cannot see the target + surface. +- Prefer UI Automation IDs and names over coordinates. Use coordinates only + after snapshots and element searches fail. +- Always inspect the current tree with `winforms_get_element_tree`, + `mcp_winapp_get_snapshot`, or targeted element search before assuming a + dialog structure. +- Treat FieldWorks UI automation as stateful. After opening menus, dialogs, or + popups, re-query the snapshot or element list. +- Capture screenshots into a task-specific evidence folder before closing the + app or dialog. +- Close evidence-only dialogs with `Cancel` or `Escape` unless the user asked + to change project data. +- Do not globally register COM or add registry hacks. FieldWorks uses reg-free + COM from the build output. + +## MCP Selection + +- Default to WinForms MCP for launch/test/screenshot flows. Use + `winforms_launch_app`, `winforms_wait_for_element`, + `winforms_get_element_tree`, `winforms_find_element`, + `winforms_click_element`, `winforms_type_text`, `winforms_set_value`, + `winforms_select_item`, `winforms_click_menu_item`, and + `winforms_take_screenshot`. +- Use WinForms MCP headless-safe operations only. Avoid `winforms_send_keys`, + drag/drop, and double-clicks on hidden-desktop processes. +- Use WinApp MCP for visible desktop/window diagnostics, UIA3 comparison, + fallback screenshots, or controls that WinForms MCP cannot operate. +- Read `references/mcp-selection.md` before changing a route from one MCP driver + to the other. + +## How This Skill Is Organized + +- `SKILL.md`: trigger metadata, global safety rules, and route index. +- `navigation/*.md`: one goal-oriented navigation path per file. These are the + FieldWorks equivalent of Page Objects: each file owns the menu path, + automation IDs, verification cues, and safe exit path for one user-facing + destination or workflow. +- `references/how-to-update.md`: rules for adding or revising navigation paths. +- `references/mcp-selection.md`: rules for choosing WinForms MCP or WinApp MCP. +- `references/research.md`: source-backed rationale for this structure. + +When a task names a destination, read the matching navigation file. When a task +discovers a better route or a fragile selector, read and update +`references/how-to-update.md` before changing the route file. + +## Navigation Path Index + +- Launch or attach to FieldWorks: `navigation/launch-or-attach.md` +- Confirm or restore a sample project: `navigation/project-loading.md` +- Writing System Properties > Font tab: `navigation/writing-system-font-options.md` +- Styles dialog > Font tab: `navigation/styles-font-tab.md` +- Manual screenshot evidence: `navigation/screenshot-evidence.md` +- MCP driver selection: `references/mcp-selection.md` + +## Screenshot Evidence + +For evidence tasks, read `navigation/screenshot-evidence.md`. Prefer committed +OpenSpec evidence only when the screenshot should be part of review history; +otherwise use `Output/ManualEvidence//`. + +For target detection, descriptive filenames, inline review, annotation, and +retake quality gates, also use `../smart-screenshot-capture/SKILL.md`. + +## How To Update This Skill + +Keep improving this skill as you explore FieldWorks and discover the most +efficient ways to do things. When you find a reliable automation ID, keyboard +route, menu path, restore flow, modal-dialog workaround, or screenshot trick, +update the relevant route file in the same change set or propose the update to +the user. Prefer short, verified notes over broad guesses. + +Use `references/how-to-update.md` for the exact update checklist. In short: + +- add one navigation file per distinct destination or workflow; +- keep each file task-focused and action-oriented; +- record stable automation IDs, entry state, verification cues, and exit path; +- move shared rules back to this index only when at least two route files need + the same guidance; +- remove or revise stale routes when WinApp snapshots prove they changed. diff --git a/.github/skills/fieldworks-winapp/navigation/launch-or-attach.md b/.github/skills/fieldworks-winapp/navigation/launch-or-attach.md new file mode 100644 index 0000000000..230512bc23 --- /dev/null +++ b/.github/skills/fieldworks-winapp/navigation/launch-or-attach.md @@ -0,0 +1,66 @@ +# Launch Or Attach To FieldWorks + +Use this path when a task needs the live FieldWorks Language Explorer app. +Choose WinForms MCP first for fresh FieldWorks launches, and use WinApp MCP for +visible desktop diagnostics or fallback. + +## Entry State + +- Workspace is a Windows/x64 FieldWorks checkout. +- Prefer an existing debug build at `Output/Debug/FieldWorks.exe`. + +## Preferred Steps: WinForms MCP UIA2 + +1. Launch debug FieldWorks with `winforms_launch_app`: + - `path`: `\\Output\\Debug\\FieldWorks.exe` +2. Keep the returned `pid`; pass it to later process-scoped calls when the MCP + client exposes a process parameter. +3. Wait for a recognizable app window or menu with `winforms_wait_for_element`. +4. Inspect the tree with `winforms_get_element_tree` before interacting. +5. Use `winforms_click_menu_item`, `winforms_find_element`, + `winforms_click_element`, `winforms_type_text`, `winforms_set_value`, and + `winforms_select_item` for headless-safe interaction. +6. Capture screenshots with `winforms_take_screenshot`. Prefer passing the + FieldWorks `pid` so the server can use the hidden-desktop window directly. + +## Fallback Steps: WinApp MCP UIA3 + +1. If FieldWorks is already running, attach to the existing process with + `mcp_winapp_attach_to_app` or `mcp_winapp_attach_to_pid`. +2. Otherwise launch debug FieldWorks with `mcp_winapp_launch_app`: + - `exePath`: `\\Output\\Debug\\FieldWorks.exe` +3. If `mcp_winapp_wait_for_input_idle` is not implemented or fails, continue + with direct window discovery. +4. Use `mcp_winapp_list_windows` and `mcp_winapp_list_desktop_windows` to verify + the app is visible. +5. Use `mcp_winapp_get_snapshot` to confirm FieldWorks has a main window and a + recognizable UI tree. +6. If screenshots capture VS Code or another foreground app, press `ALT+ESCAPE` + with `mcp_winapp_press_key_combo` until FieldWorks is first in desktop window + order or a FieldWorks element has focus. + +## Expected Signals + +- WinForms MCP launch returns a process ID and can operate on a hidden desktop + because `.vscode/mcp.json` sets `HEADLESS=true`. +- Main window title may initially appear as `The Window` in UI Automation. +- A loaded project window title may include a project name such as `Sena 3`. +- The side pane often contains `Lexicon`, `Texts & Words`, `Grammar`, + `Notebook`, and `Lists` groups. + +## Known Workarounds + +- In WinForms MCP headless mode, avoid `winforms_send_keys`, drag/drop, and + double-click. Use UIA pattern operations such as `winforms_type_text`, + `winforms_set_value`, `winforms_select_item`, and single + `winforms_click_element` calls. +- `mcp_winapp_wait_for_input_idle` may return `not implemented` for + FieldWorks. Use snapshots and desktop-window order instead. +- `mcp_winapp_take_screenshot` captures the foreground surface. Bring + FieldWorks forward before relying on screenshots. + +## Exit + +Close only the FieldWorks instance you launched for the task, using +`winforms_close_app` or `mcp_winapp_close_app`. Do not close a user's +pre-existing app instance unless the user asked for cleanup. diff --git a/.github/skills/fieldworks-winapp/navigation/project-loading.md b/.github/skills/fieldworks-winapp/navigation/project-loading.md new file mode 100644 index 0000000000..f435243ca5 --- /dev/null +++ b/.github/skills/fieldworks-winapp/navigation/project-loading.md @@ -0,0 +1,40 @@ +# Confirm Or Restore A Sample Project + +Use this path after launching FieldWorks when a task needs a repeatable project. + +## Entry State + +- FieldWorks is launched or attached. +- You have an app ID from WinApp MCP. + +## Steps + +1. Get a snapshot with `mcp_winapp_get_snapshot`. +2. Treat the project as loaded if the snapshot contains a root document such as + `Root - Sena 3`, or if the desktop title contains the expected project name. +3. For repeatable manual evidence, use the current project if `Sena 3` is + already loaded. +4. If no project is loaded, restore from the sample backup at the repository + root: + - `Sena 3 2018-09-11 1145.fwbackup` +5. Use the File menu and inspect the menu tree for restore/project-management + commands before clicking. +6. If a file chooser opens, select the backup path and proceed through restore. +7. Re-inspect the app and capture a loaded-project screenshot. + +## Expected Signals + +- Project loaded: `Root - Sena 3` document in the snapshot. +- Common loaded side pane: `Lexicon Edit`, `Browse`, `Dictionary`, + `Collect Words`, `Classified Dictionary`, `Bulk Edit Entries`. + +## Safety + +- Do not restore over an existing loaded project unless the task requires it. +- Prefer evidence-only screenshots over changing project data. + +## Exit + +Continue to the requested navigation path. If project restore was only a setup +step and the user did not ask to keep the app open, close the launched app at +the end of the task. diff --git a/.github/skills/fieldworks-winapp/navigation/screenshot-evidence.md b/.github/skills/fieldworks-winapp/navigation/screenshot-evidence.md new file mode 100644 index 0000000000..ce8ea59202 --- /dev/null +++ b/.github/skills/fieldworks-winapp/navigation/screenshot-evidence.md @@ -0,0 +1,72 @@ +# Capture Manual Screenshot Evidence + +Use this path when the task needs before/after images, Jira evidence, OpenSpec +evidence, or visual proof that a FieldWorks UI flow works. + +For general screenshot quality rules, target inference, sorted filenames, +annotations, and retake criteria, read `../../smart-screenshot-capture/SKILL.md` +alongside this route. + +## Entry State + +- FieldWorks is launched and foregrounded. +- The target UI state is visible or reachable through another navigation path. + +## Steps + +1. Create a deterministic evidence folder: + - transient: `Output/ManualEvidence//` + - committed OpenSpec evidence: + `openspec/changes//evidence/manual-winapp/` +2. Confirm the target FieldWorks process, window, dialog, or tab before capture. +3. Capture the initial loaded-project state when it helps reviewers orient + themselves. +4. Capture the broken/reproduced state from an unfixed build when available. +5. Capture the fixed state from the current build. +6. Inspect each saved image and retake if it is blank, wrong-window, covered, + unreadable, or missing the intended state. +7. Capture related canonical surfaces when the issue spans more than one UI + path. +8. Use an image sequence or GIF when the issue involves timing, redraw, focus, + modal transitions, or steps that a single screenshot cannot communicate. + +## MCP Choice + +- Prefer `winforms_take_screenshot` for fresh FieldWorks launches; hidden + desktop screenshots should not steal focus or require foregrounding. +- Prefer passing the FieldWorks `pid` to WinForms MCP screenshots when the + client exposes a process parameter. +- Prefer `mcp_winapp_take_screenshot_optimized` for visible-desktop fallback + captures when image size or token budget matters. +- Use `mcp_winapp_take_screenshot` when WinForms MCP is unavailable, when the + task is explicitly about the visible desktop, or when comparing UIA3 behavior. +- Use `mcp_winapp_annotate_screenshot` when reviewers need specific controls or + changed regions called out. +- Use `mcp_winapp_screenshot_diff` for before/after comparisons when pixel + changes are the evidence. + +## Naming + +Use names that tell the story in sorted order: + +- `01-initial-.png` +- `02-before-.png` +- `03-after-.png` +- `04-after-.png` +- `sequence--001.png`, `sequence--002.png`, ... + +## Expected Signals + +- Screenshots should show FieldWorks, not VS Code or another foreground app. +- WinForms MCP screenshots can capture headless FieldWorks windows through + `PrintWindow`; foreground-window order should not matter for that path. +- If `mcp_winapp_take_screenshot` captures the wrong surface, use the launch + route's `ALT+ESCAPE` foregrounding workaround and capture again. +- If a true before-state is not available in the current worktree, document why + and describe how to capture it from an unfixed build. + +## Exit + +Copy only review-worthy screenshots into committed evidence folders. Leave +scratch captures under `Output/ManualEvidence` unless the user asks to commit +them. diff --git a/.github/skills/fieldworks-winapp/navigation/styles-font-tab.md b/.github/skills/fieldworks-winapp/navigation/styles-font-tab.md new file mode 100644 index 0000000000..f8300fba4b --- /dev/null +++ b/.github/skills/fieldworks-winapp/navigation/styles-font-tab.md @@ -0,0 +1,52 @@ +# Navigate To Styles Font Tab + +Use this path to inspect or document the shared Font Features control in the +Styles dialog. + +## Entry State + +- FieldWorks is launched and a project is loaded. +- If screenshots are needed, FieldWorks is foregrounded. + +## Steps + +1. Click the `Format` menu item. +2. Click `Styles...`. +3. In `FwStylesDlg`, click the `Font` tab. +4. Select the target style in `m_lstStyles` if the task requires a specific + style. For general evidence, `Normal` is usually sufficient. +5. Capture or inspect the Font Features state. + +## Stable Elements Observed + +- Dialog: `FwStylesDlg` +- Styles list: `m_lstStyles` +- List filter combo: `m_cboTypes` +- Tab control: `m_tabControl` +- Font tab pane: `m_tbFont` +- Font attributes pane: `m_FontAttributes` +- Writing systems list: `m_lstWritingSystems` +- Font combo: `m_cboFontNames` +- Font Features button: `m_btnFontFeatures` +- OK button: `m_btnOk` +- Cancel button: `m_btnCancel` + +## Expected Signals + +- The Font tab shows `Font features` in the Attributes area. +- The writing-system list includes `` and project writing + systems such as `English`, `Portuguese`, `Sena`, and `Sena (Phonetic)` for + the Sena 3 sample project. + +## Known Workarounds + +- The `Styles...` menu item appears only after the `Format` menu is open; if a + first click does not expose submenu items, click `Format` again and re-query + menu items. +- Use `mcp_winapp_invoke_element` for `m_btnCancel` when a normal click does not + close the dialog. + +## Exit + +Use `Cancel` for evidence-only sessions. Use `OK` only when the task requires a +style change. diff --git a/.github/skills/fieldworks-winapp/navigation/writing-system-font-options.md b/.github/skills/fieldworks-winapp/navigation/writing-system-font-options.md new file mode 100644 index 0000000000..7e8bcbc83a --- /dev/null +++ b/.github/skills/fieldworks-winapp/navigation/writing-system-font-options.md @@ -0,0 +1,65 @@ +# Navigate To Writing System Font Options + +Use this path to inspect or document Writing System Properties > Font behavior, +including Font Options and Font Features controls. + +## Entry State + +- FieldWorks is launched and a project is loaded. +- If screenshots are needed, FieldWorks is foregrounded. + +## Steps + +1. With WinForms MCP, restore the main FieldWorks window if it is minimized or + offscreen. +2. Open `Format` > `Set up Vernacular Writing Systems...` or + `Set up Analysis Writing Systems...`. If WinForms MCP cannot see the dynamic + submenu command, use the visible-desktop keyboard fallback + `%o{DOWN}{DOWN}{DOWN}{ENTER}` after the main window is focused. +3. In `Writing System Properties`, inspect the UIA2 tree and click the `Font` + tab item under `_tabControl`. +4. Capture or inspect the target state. +5. If a temporary evidence font is needed, select it in `m_defaultFontComboBox` + and cancel the dialog afterward. + +## Stable Elements Observed + +- Dialog: `FwWritingSystemSetupDlg` +- Writing-system list: `_writingSystemList` +- Tab control: `_tabControl` +- Font tab: `_fontTab` +- Default font control: `_defaultFontControl` +- Default font combo: `m_defaultFontComboBox` +- Font options group: `m_graphiteGroupBox` +- Graphite checkbox: `m_enableGraphiteCheckBox` +- Font Features button: `m_defaultFontFeaturesButton` +- OK button: `_okBtn` +- Cancel button: `_cancelBtn` + +## Expected Signals + +- The Font tab exposes the font options group, Graphite checkbox, and Font + Features button when supported by the current build and selected font. +- Record the exact visible labels and enabled state from the UI tree or + screenshot instead of assuming they match a previous evidence run. + +## Known Workarounds + +- For WinForms MCP, `winforms_raise_event` with `eventName: "invoke"` is often + more reliable than a click for dialog buttons and popup-opening buttons. +- For WinApp MCP fallback, `mcp_winapp_invoke_element` is often more reliable + than a click. +- WinForms MCP global `winforms_find_elements` may miss modal dialog children. + Use `winforms_get_element_tree` on the process, then reuse cached element IDs + from the returned dialog tree. +- The FieldWorks font combo is owner-drawn. If `winforms_select_item` cannot + select an offscreen font, open the combo and use keyboard navigation from a + verified sorted font-family list. +- Invoking `m_defaultFontFeaturesButton` through WinForms MCP can expose a + top-level `Menu` window. Re-query the process tree after invoking; menu items + may become the UIA2 roots while the menu is open. + +## Exit + +Use `Cancel` for evidence-only sessions. Use `OK` only when the task requires a +project data change. diff --git a/.github/skills/fieldworks-winapp/references/how-to-update.md b/.github/skills/fieldworks-winapp/references/how-to-update.md new file mode 100644 index 0000000000..afc1a32711 --- /dev/null +++ b/.github/skills/fieldworks-winapp/references/how-to-update.md @@ -0,0 +1,84 @@ +# How To Update This Skill + +Update this skill as FieldWorks navigation improves or as WinForms MCP and +WinApp MCP reveal more stable automation patterns. + +## Organization Rules + +1. Keep `SKILL.md` small. It is the trigger, safety, and table-of-contents + layer. +2. Put each distinct navigation destination or workflow in its own file under + `navigation/`. +3. Use one route file when the user goal is one destination, even if it crosses + several menus. +4. Split a route file when a path has a different entry state, different target + dialog, different safety profile, or different verification signal. +5. Keep source rationale and maintenance rules in `references/`, not in every + route file. + +## Route File Template + +Use this structure for new navigation files: + +```markdown +# Navigate To + +Use this path when ... + +## Entry State + +- ... + +## Steps + +1. ... + +## Stable Elements Observed + +- Dialog: `...` +- Button: `...` + +## Expected Signals + +- ... + +## Known Workarounds + +- ... + +## Exit + +- ... +``` + +## Update Checklist + +- Start from a fresh `winforms_get_element_tree` or `mcp_winapp_get_snapshot`. +- Prefer automation IDs and element names over coordinates. +- Record menu path, dialog title, stable automation IDs, and the successful + tool call pattern. +- Record which MCP driver was verified: WinForms MCP UIA2/headless or WinApp + MCP UIA3/visible-desktop. +- Record failed tool calls only when the workaround is useful later. +- Include the safe exit path and whether the route mutates project data. +- Keep verification cues separate from route mechanics; route files can say how + to reach and recognize the state, while tests or Jira/OpenSpec notes decide + whether the state passes. +- If a route changes, update or remove stale selectors in the same edit. +- If a route becomes long, move repeated control groups into a second route file + only when another path needs them. +- If a route works with both MCP drivers, document the preferred driver first + and keep fallback steps shorter. + +## When To Add A New Navigation File + +Add a new file when you discover a repeatable user goal such as: + +- opening a dialog; +- restoring a project; +- reaching a major side-pane area; +- setting up export/import options; +- collecting a standard evidence sequence. + +Do not add a file for a one-off button unless it is a stable reusable component +used by multiple paths. diff --git a/.github/skills/fieldworks-winapp/references/mcp-selection.md b/.github/skills/fieldworks-winapp/references/mcp-selection.md new file mode 100644 index 0000000000..68185bb7a2 --- /dev/null +++ b/.github/skills/fieldworks-winapp/references/mcp-selection.md @@ -0,0 +1,67 @@ +# MCP Driver Selection + +FieldWorks is currently a WinForms application. Use WinForms MCP as the default +driver for fresh live-app automation, and keep WinApp MCP as the UIA3 fallback +and visible-desktop diagnostic path. + +## Default: WinForms MCP UIA2 + +Use WinForms MCP when: + +- launching FieldWorks yourself for verification or screenshot evidence; +- the task can run in the background on a hidden desktop; +- the target is a WinForms menu, dialog, button, tab, text box, combo box, or + standard control; +- you need `render_form` to preview a `.Designer.cs` surface; +- foreground-window stealing would disrupt the developer. + +Preferred tools: + +- process: `winforms_launch_app`, `winforms_attach_to_process`, + `winforms_close_app`, `winforms_get_process_status`; +- discovery: `winforms_get_element_tree`, `winforms_find_element`, + `winforms_wait_for_element`, `winforms_element_exists`; +- interaction: `winforms_click_menu_item`, `winforms_click_element`, + `winforms_type_text`, `winforms_set_value`, `winforms_select_item`; +- evidence: `winforms_take_screenshot`, `winforms_render_form`. + +Headless-safe rule: prefer UIA pattern operations. Avoid `winforms_send_keys`, +drag/drop, and double-click against headless processes because those rely on +input simulation on the visible desktop. + +## Fallback: WinApp MCP UIA3 + +Use WinApp MCP when: + +- WinForms MCP is not configured in the current agent client; +- the task is about foreground, focus, taskbar, window ordering, or another + visible desktop behavior; +- a control, popup, or non-WinForms surface is missing or unreliable through + WinForms MCP; +- you need UIA3 behavior for comparison or troubleshooting. + +Useful tools include `mcp_winapp_launch_app`, `mcp_winapp_attach_to_app`, +`mcp_winapp_get_snapshot`, `mcp_winapp_find_elements`, +`mcp_winapp_click_element`, `mcp_winapp_invoke_element`, +`mcp_winapp_select_option`, and `mcp_winapp_take_screenshot`. + +## Dynamic Choice Procedure + +1. Decide whether the task requires visible desktop behavior. +2. If not, choose WinForms MCP and launch FieldWorks headless. +3. Keep the returned process ID for screenshots and cleanup. +4. Inspect the tree before interacting. +5. If an element cannot be found or invoked through WinForms MCP, switch to + WinApp MCP for that route and record the fallback in the navigation file. +6. If the task needs human-visible screenshots or focus behavior, start with + WinApp MCP. + +## Recording Findings + +When updating a navigation path, record: + +- which driver was verified; +- whether the process was launched headless or attached visibly; +- stable automation IDs/names; +- the successful tool sequence; +- any driver-specific failure and the fallback that worked. \ No newline at end of file diff --git a/.github/skills/fieldworks-winapp/references/research.md b/.github/skills/fieldworks-winapp/references/research.md new file mode 100644 index 0000000000..6ba6990cf0 --- /dev/null +++ b/.github/skills/fieldworks-winapp/references/research.md @@ -0,0 +1,74 @@ +# Research Notes + +These sources informed the FieldWorks WinApp skill structure. + +## Agent Skills and progressive disclosure + +- Anthropic Engineering, "Equipping agents for the real world with Agent + Skills" explains that a skill is a directory with `SKILL.md`, and that large + or scenario-specific content should be split into referenced files so the + agent loads only the context it needs. + https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills +- Agent Skills overview describes progressive disclosure as discovery from + metadata, activation through `SKILL.md`, and execution through optional + referenced files, scripts, and resources. + https://agentskills.io/ + +Applied here: `SKILL.md` stays as a compact trigger and index, while each +navigation path lives in a separate markdown file. + +## Page Object / navigation encapsulation + +- Playwright's Page Object Model docs recommend encapsulating common operations + and selectors in one place to make UI automation easier to author and + maintain. + https://playwright.dev/docs/pom +- Selenium's Page Object Model docs emphasize clean separation between tests and + page-specific locators/layout, plus a single repository for operations a page + offers. + https://www.selenium.dev/documentation/test_practices/encouraged/page_object_models/ +- Martin Fowler's Page Object article says a page object should wrap a page or + fragment with an application-specific API, hide widget details, model + significant user-facing elements rather than every HTML/control container, and + generally leave assertions to tests. + https://martinfowler.com/bliki/PageObject.html + +Applied here: a navigation file owns menu paths, stable automation IDs, +workarounds, expected loaded-state cues, and safe exits for one user-facing +destination. Jira/OpenSpec/test notes own pass/fail assertions. + +## How-to documentation shape + +- Diataxis how-to guide guidance says practical guides should be task-oriented, + goal-focused, action-based, named clearly, and not overloaded with reference + detail. + https://diataxis.fr/how-to-guides/ + +Applied here: navigation files are named by user goal and contain just enough +sequence, cues, and safety notes to perform that route. + +## Resulting convention + +- One reusable navigation destination or workflow per `navigation/*.md` file. +- Shared safety and triggering rules stay in `SKILL.md`. +- Skill maintenance rules stay in `references/how-to-update.md`. +- Research rationale stays in this file so future edits can revisit the design + without bloating the active route instructions. + +## WinForms MCP headless automation + +- `fnrhombus/winforms-mcp` supports `npx -y @fnrhombus/winforms-mcp` MCP setup, + `HEADLESS=true` hidden desktop launches, `TFM=net48`, and + `TELEMETRY_OPTOUT=true`. + https://github.com/fnrhombus/winforms-mcp +- Its headless mode creates a hidden Windows desktop, launches processes there, + routes UIA operations per process, and captures screenshots via `PrintWindow`. + https://github.com/fnrhombus/winforms-mcp/wiki/Headless-Mode +- Hidden desktop supports UIA-pattern tools such as element discovery, tree + inspection, invoke/toggle/selection actions, value setting, and screenshots. + Input-simulation tools such as `send_keys`, drag/drop, and double-click need + the visible desktop. + +Applied here: FieldWorks is currently WinForms, so WinForms MCP is the default +for new live-app automation. WinApp MCP remains the UIA3 fallback and the right +tool for visible desktop/window diagnostics. diff --git a/.github/skills/smart-screenshot-capture/SKILL.md b/.github/skills/smart-screenshot-capture/SKILL.md new file mode 100644 index 0000000000..3786b87b60 --- /dev/null +++ b/.github/skills/smart-screenshot-capture/SKILL.md @@ -0,0 +1,141 @@ +--- +name: smart-screenshot-capture +description: > + Capture high-quality screenshots with MCP tools. Use when the user asks to + take a screenshot, show the screen, capture a window, document UI evidence, + collect before/after images, inspect browser output, or verify a desktop app + visually with WinForms MCP, WinApp MCP, browser screenshots, image previews, + annotations, or screenshot diffs. +license: MIT +compatibility: > + Windows-first adaptation for VS Code agents with WinForms MCP, WinApp MCP, + browser page tools, and image preview tools. Inspired by brennacodes' + screenshotr skill, but adapted away from macOS screencapture/sips commands. +metadata: + author: FieldWorks team + version: "1.0" + upstream: https://mcpmarket.com/tools/skills/smart-screenshot-capture +--- + +# Smart Screenshot Capture + +Use this skill to capture clear, reviewable screenshots without making the user +manually crop, upload, or explain the screen. Prefer focused captures of the +target app, window, browser page, or element over whole-desktop images. + +## Core Rules + +- Infer the target from the conversation when it is obvious, such as the active + FieldWorks dialog, a named app, a browser page, or a before/after UI state. +- Ask one short question only when the target is ambiguous enough that a wrong + capture is likely. +- Save screenshots to a deterministic path before reporting them. +- Generate concise kebab-case filenames from the target and state. +- Display or inspect the saved image after capture with `view_image` or the + appropriate image preview tool. +- Retake the screenshot when it is blank, the wrong app, overlapped by another + window, too small to read, or missing the state the user asked to document. +- Prefer non-mutating inspection and capture tools. Do not change project data + just to make a screenshot easier. + +## Target Selection + +Choose the narrowest useful target: + +1. FieldWorks or another WinForms desktop app: capture by process id or window + handle when the MCP tool supports it. +2. A visible Windows app where WinForms MCP cannot attach: use WinApp MCP and a + specific `appId` or `windowHandle`. +3. Browser content controlled by the integrated browser: use `screenshot_page` + for the whole viewport or an element-specific capture. +4. A UI element that needs visual callouts: capture the app window, then use + annotation tools when available. +5. Whole desktop: use only when the user asks for desktop context or no focused + capture path exists. + +When capturing a desktop app, inspect the UI tree or window list first if there +is any risk of targeting the wrong window. + +## Output Paths + +For this repository, default to: + +- transient evidence: `Output/ManualEvidence//` +- OpenSpec review evidence: `openspec/changes//evidence/manual-winapp/` +- ad hoc screenshots: `Output/ManualEvidence/screenshots/` + +Create the folder if needed. Do not put scratch screenshots in committed +evidence folders unless the user asks for review-ready evidence. + +## Naming + +Use sorted, descriptive names: + +- single capture: `-.png` +- before/after: `01-before-.png`, `02-after-.png` +- sequence: `step-01-.png`, `step-02-.png` +- app tour: `-.png` +- temporary fallback: `screenshot-YYYY-MM-DD-HHMMSS.png` + +Keep generated names short, lowercase, and kebab-case. Prefer 2-5 meaningful +words over timestamps unless ordering or uniqueness requires a timestamp. + +## Tool Preference + +Use the best available tool for the current target: + +- FieldWorks/WinForms hidden desktop: `mcp_winforms-mcp_winforms_take_screenshot`. +- WinApp visible desktop: `mcp_winapp_take_screenshot_optimized` when image size + matters, otherwise `mcp_winapp_take_screenshot`. +- UI callouts: `mcp_winapp_annotate_screenshot`. +- Visual comparison: `mcp_winapp_screenshot_diff`. +- Browser pages: `screenshot_page`. +- Saved image review: `view_image`. + +Prefer MCP screenshot tools over shell commands. Do not use the upstream +macOS-specific `screencapture`, `sips`, or AppleScript workflow on Windows. + +## Capture Workflow + +1. Identify the target and output path. +2. Inspect the app/page/window when needed to confirm the target is visible. +3. Capture to a deterministic filename. +4. Inspect the saved image. +5. Retake or annotate if the first capture does not communicate the requested + state clearly. +6. Report the saved path and any useful capture metadata, such as dimensions or + diff percentage. + +For before/after work, capture the before state first whenever it is available. +If the before state is unavailable in the current worktree, say so and capture +the fixed or current state with a clear filename. + +## Multi-Screenshot Workflows + +Use multiple captures when one image cannot tell the story: + +- before/after: two captures with matching framing; +- step sequence: one capture per user-visible step; +- app tour: each relevant dialog, pane, or tab; +- redraw/focus/modal timing: a short ordered sequence rather than a single + screenshot; +- comparison: capture both images, then run a screenshot diff when available. + +For sequences, keep the same target, window size, and framing across captures +unless the task is specifically about responsive or layout behavior. + +## Quality Checklist + +Before considering the screenshot done, verify: + +- the image exists at the reported path; +- the target app, page, or element is visible; +- text needed for review is readable; +- modal dialogs, focus state, selected tabs, and highlighted controls match the + requested state; +- no unrelated foreground window covers the target; +- committed evidence contains only review-worthy images. + +If a screenshot tool produces a cropped, blank, or wrong-window capture, switch +drivers in this order: focused WinForms capture, WinApp optimized capture, +browser capture, then whole-window or whole-desktop fallback. \ No newline at end of file diff --git a/.vscode/mcp.json b/.vscode/mcp.json index d87532adc9..5927a80acf 100644 --- a/.vscode/mcp.json +++ b/.vscode/mcp.json @@ -20,6 +20,19 @@ "env": { "SERENA_API_KEY": "${env:SERENA_API_KEY}" } + }, + "winforms-mcp": { + "type": "stdio", + "command": "npx", + "args": [ + "-y", + "@fnrhombus/winforms-mcp" + ], + "env": { + "HEADLESS": "true", + "TFM": "net48", + "TELEMETRY_OPTOUT": "true" + } } } } \ No newline at end of file diff --git a/Docs/mcp.md b/Docs/mcp.md index 8a13b297b3..e0bfc63ec1 100644 --- a/Docs/mcp.md +++ b/Docs/mcp.md @@ -1,10 +1,11 @@ # Model Context Protocol helpers -FieldWorks ships a small workspace `.vscode/mcp.json` so Model Context Protocol clients can spin up two +FieldWorks ships a small workspace `.vscode/mcp.json` so Model Context Protocol clients can spin up three servers automatically: - **GitHub server** via the hosted GitHub MCP endpoint (`https://api.githubcopilot.com/mcp/`). - **Serena server** for accelerated navigation of this large mono-repo. +- **WinForms MCP server** for background UIA2 automation of the FieldWorks WinForms desktop app. ## Prerequisites @@ -13,6 +14,7 @@ servers automatically: | VS Code 1.101+ | Remote MCP + OAuth support | https://code.visualstudio.com | | Serena CLI | Provides Serena search/navigation | `pipx install serena-cli` or `uv tool install serena` | | `uvx` | Launches Serena from workspace `.vscode/mcp.json` | https://github.com/astral-sh/uv | +| Node.js/npm | Launches `@fnrhombus/winforms-mcp` with `npx` | https://nodejs.org | > **Note**: Serena **auto-downloads** its language servers on first use: > - **C# (`csharp`)**: Microsoft.CodeAnalysis.LanguageServer (Roslyn) from Azure NuGet + .NET 9 runtime @@ -24,13 +26,35 @@ Authentication: - GitHub MCP uses OAuth through your normal VS Code GitHub/Copilot sign-in. - `SERENA_API_KEY` remains optional and is only needed when your Serena deployment requires it. +- WinForms MCP is launched locally with `npx -y @fnrhombus/winforms-mcp`; no service login is required. ## How it works -1. `.vscode/mcp.json` defines a hosted GitHub MCP server and a local Serena stdio server. +1. `.vscode/mcp.json` defines a hosted GitHub MCP server plus local Serena and WinForms MCP stdio servers. 2. GitHub MCP uses VS Code OAuth authentication for repository operations. 3. Serena starts from `uvx --from git+https://github.com/oraios/serena serena start-mcp-server --context ide --project ${workspaceFolder}`. -4. In chat, use tool sets / tool picker to keep active tool counts low and focused. +4. WinForms MCP starts from `npx -y @fnrhombus/winforms-mcp` with `HEADLESS=true`, `TFM=net48`, and `TELEMETRY_OPTOUT=true`. +5. In chat, use tool sets / tool picker to keep active tool counts low and focused. + +## WinForms MCP for FieldWorks UI automation + +FieldWorks is currently a WinForms desktop app, so prefer WinForms MCP for most runtime UI walks and screenshots. The workspace config enables headless mode so `winforms_launch_app` starts FieldWorks on a hidden desktop instead of stealing focus from the developer's visible desktop. + +Use WinForms MCP when: + +- launching a fresh FieldWorks instance for manual verification; +- capturing screenshots without bringing FieldWorks to the foreground; +- inspecting standard WinForms controls, menus, and dialogs by AutomationId, name, class, or control type; +- using `render_form` for a `.Designer.cs` preview. + +Use the UIA3 WinApp MCP tools when: + +- WinForms MCP is unavailable in the active client; +- you need to attach to a user-visible process and inspect the desktop/window list; +- a route needs UIA3-specific behavior or a control is not exposed correctly through WinForms MCP; +- troubleshooting focus, foreground-window behavior, or non-WinForms surfaces. + +Headless WinForms MCP limitations: `send_keys`, drag/drop, and double-click paths use input simulation and require the visible desktop. Prefer `winforms_type_text`, `winforms_set_value`, `winforms_select_item`, `winforms_click_element`, and `winforms_click_menu_item` for headless work. ## Running the servers manually @@ -41,9 +65,15 @@ If you want to test outside an MCP-aware editor: # Serena server uvx --from git+https://github.com/oraios/serena serena start-mcp-server --context ide --project . + +# WinForms MCP server +$env:HEADLESS = "true" +$env:TFM = "net48" +$env:TELEMETRY_OPTOUT = "true" +npx -y @fnrhombus/winforms-mcp ``` -The Serena process runs until you press `Ctrl+C`. When invoked through an MCP host, it stops +The local stdio processes run until you press `Ctrl+C`. When invoked through an MCP host, they stop when the client disconnects. ## Multiple Worktrees and Serena Conflicts @@ -90,9 +120,9 @@ project_name: 010-advanced-entry-view ## Best-practice profile for this repo -- Keep repo-level MCP servers minimal: `github` + `serena` only. +- Keep repo-level MCP servers focused: `github`, `serena`, and `winforms-mcp`. - Keep workflow/task conventions in skills/prompt files, not additional MCP servers. -- Enable extra MCP servers only per-task via the tool picker, then disable again. +- Enable WinForms MCP only for UI automation tasks via the tool picker when the client supports per-task tool selection. - If tool list feels noisy, reset cached tools with **MCP: Reset Cached Tools**. ## Worktree best practices @@ -108,6 +138,10 @@ project_name: 010-advanced-entry-view - **GitHub tools fail with auth errors** – sign out/in of GitHub in VS Code and restart MCP servers. - **`uvx` was not found on PATH** – install `uv` and reopen your shell. +- **`npx` was not found on PATH** – install Node.js/npm and reopen VS Code. +- **WinForms MCP package cannot be resolved** – run `npm view @fnrhombus/winforms-mcp version` to check npm registry access. +- **Headless text input does nothing** – use WinForms MCP `type_text` or `set_value`, not `send_keys`. +- **Headless screenshot is blank or stale** – pass the FieldWorks process ID to `take_screenshot` when the client exposes that parameter. - **Unable to start Serena MCP** – ensure `uvx` can reach GitHub and run: `uvx --from git+https://github.com/oraios/serena serena start-mcp-server --help` - **Language server download fails (network error)** – Serena auto-downloads C# (Roslyn) and C++ (clangd) From 45ed34a337a972806871a0c83e4aff2273bebbeb Mon Sep 17 00:00:00 2001 From: John Lambert Date: Thu, 21 May 2026 13:41:24 -0400 Subject: [PATCH 2/2] Updates from Copilot comments --- Docs/mcp.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/Docs/mcp.md b/Docs/mcp.md index e0bfc63ec1..71fae39ed6 100644 --- a/Docs/mcp.md +++ b/Docs/mcp.md @@ -11,7 +11,6 @@ servers automatically: | Component | Purpose | Install guidance | | -------------- | ----------------------------------------- | ----------------------------------------------------- | -| VS Code 1.101+ | Remote MCP + OAuth support | https://code.visualstudio.com | | Serena CLI | Provides Serena search/navigation | `pipx install serena-cli` or `uv tool install serena` | | `uvx` | Launches Serena from workspace `.vscode/mcp.json` | https://github.com/astral-sh/uv | | Node.js/npm | Launches `@fnrhombus/winforms-mcp` with `npx` | https://nodejs.org | @@ -40,12 +39,14 @@ Authentication: FieldWorks is currently a WinForms desktop app, so prefer WinForms MCP for most runtime UI walks and screenshots. The workspace config enables headless mode so `winforms_launch_app` starts FieldWorks on a hidden desktop instead of stealing focus from the developer's visible desktop. +Tool names in this section use the WinForms MCP `winforms_*` namespace as shown in the client tool picker. If a client shortens a name to the underlying action, map it back to the same WinForms MCP command. + Use WinForms MCP when: - launching a fresh FieldWorks instance for manual verification; - capturing screenshots without bringing FieldWorks to the foreground; - inspecting standard WinForms controls, menus, and dialogs by AutomationId, name, class, or control type; -- using `render_form` for a `.Designer.cs` preview. +- using `winforms_render_form` for a `.Designer.cs` preview. Use the UIA3 WinApp MCP tools when: @@ -54,7 +55,7 @@ Use the UIA3 WinApp MCP tools when: - a route needs UIA3-specific behavior or a control is not exposed correctly through WinForms MCP; - troubleshooting focus, foreground-window behavior, or non-WinForms surfaces. -Headless WinForms MCP limitations: `send_keys`, drag/drop, and double-click paths use input simulation and require the visible desktop. Prefer `winforms_type_text`, `winforms_set_value`, `winforms_select_item`, `winforms_click_element`, and `winforms_click_menu_item` for headless work. +Headless WinForms MCP limitations: `winforms_send_keys`, drag/drop, and double-click paths use input simulation and require the visible desktop. Prefer `winforms_type_text`, `winforms_set_value`, `winforms_select_item`, `winforms_click_element`, and `winforms_click_menu_item` for headless work. ## Running the servers manually @@ -140,8 +141,8 @@ project_name: 010-advanced-entry-view - **`uvx` was not found on PATH** – install `uv` and reopen your shell. - **`npx` was not found on PATH** – install Node.js/npm and reopen VS Code. - **WinForms MCP package cannot be resolved** – run `npm view @fnrhombus/winforms-mcp version` to check npm registry access. -- **Headless text input does nothing** – use WinForms MCP `type_text` or `set_value`, not `send_keys`. -- **Headless screenshot is blank or stale** – pass the FieldWorks process ID to `take_screenshot` when the client exposes that parameter. +- **Headless text input does nothing** – use WinForms MCP `winforms_type_text` or `winforms_set_value`, not `winforms_send_keys`. +- **Headless screenshot is blank or stale** – pass the FieldWorks process ID to `winforms_take_screenshot` when the client exposes that parameter. - **Unable to start Serena MCP** – ensure `uvx` can reach GitHub and run: `uvx --from git+https://github.com/oraios/serena serena start-mcp-server --help` - **Language server download fails (network error)** – Serena auto-downloads C# (Roslyn) and C++ (clangd)