An agent can read the accessibility tree. Why might it still click the wrong button?
The collected description says vercel-labs/agent-desktop uses accessibility structure to help agents identify and operate desktop applications. Start with low-impact tasks and check labels, window context and resulting state.
What to consider
Identifying a control is only a start. It must belong to the current task, act on the intended object and produce the expected state. The queue describes agent-desktop as using accessibility structure to identify and operate desktop applications. It does not establish supported systems, permission setup or success rates.
Check whether labels distinguish the intended action
Use a recoverable test document to list the window, controls and intended actions. Multiple Save buttons require document and dialog context. If the structure exposes unnamed icons, record that identification gap instead of guessing a position.
Observe whether a control description remains valid after a document switch, menu expansion or dialog. Read the current interface after state changes before choosing the next action. These are test scenarios, not claims about the repository’s locating algorithm or recovery features.
Verify what changed after each action
A click without an error does not establish a save, export or submission. Define an observable end condition in advance: a file at the expected location, a success state or changes surviving a reopen. Choose evidence that matches the task.
If the result does not change, distinguish processing time, the wrong target and a new confirmation step. Avoid repeating an action merely because the outcome is not yet visible. For tasks that can create duplicate records, keep enough progress information to resume safely.
Prove a reversible task before expanding access
Begin with a copy, control inspection and an edit to a test document. Check originals, outputs and the action record afterward. External messages, real payments and deletion require their own authorization and recovery boundaries. Reading a UI does not grant permission to act.
We did not install the project, verify its license or test it on an operating system, and measured no automation success rate. The description establishes its intended use. Adoption still needs testing against your applications, control visibility and outcomes.
Source links
The links below were supplied with the collected records. This article offers independent analysis of their summaries; we have not individually checked the original sources or tested the products.