What is computer use? How models interact with software interfaces
Screenshot observation, actions and feedback, distinguishing computer use from browser agents and direct APIs, with execution sessions, coordinates and authorization.
Understand the concept
Computer use lets AI carry out tasks through desktop or browser interfaces. Commonly, models observe screens and propose clicks or typing, execution tools perform them and updated state guides the next step. Describing actions does not establish control of a computer: drivers, sessions and permissions are still needed. The term is not exclusive to one product or model family.
How observations, actions and new screens form a loop
For a hypothetical task organizing public webpage material, a model observes a page and chooses clicks or text input. Tools execute actions and return new state. No real action is claimed here; the example separates plans from execution rather than treating generated steps as completed work.
Interface tools need an execution environment, such as a controlled browser or separate desktop. Screenshots and other observations inform continuation. A service API does not automatically gain control of all applications on the user’s computer.
Keep screenshot coordinates aligned with the execution session
Screenshot-based actions require matching visible dimensions and actual coordinates. Resizing observations without adjusting action coordinates can hit the wrong location. Window movement, popups and loading also alter targets. Use new observations rather than a long click sequence based on a stale screen.
Preserve the intended browser or desktop session so actions do not target another window. Visible external text is not fresh user authorization. Login state makes material accessible but does not bring every read or submission within scope.
Browser tasks differ from whole-desktop operation
A browser agent is defined by its web-task scope and may use page structure or screenshots. Computer use describes interface interaction and can be broader. They overlap, but a browser task does not establish full desktop control.
Products package similar capabilities as desktop assistants, web tools or automation components. Inspect drivers, observations and permitted actions rather than infer from branding. WorkBuddy’s local-file tools, for example, need not all use mouse clicks.
Compare interface interaction with available business APIs
A business API can express explicit objects and parameters, while UI tools fit steps available only through interfaces. Both need legitimate access and outcome checks. Interfaces are not universally better, and APIs must not be used without authorization to bypass confirmations.
Limit applications and resources to the task and respect authorization for important writes. Confirm saved files or records after an apparent success; screenshots provide only part of the evidence. This page explains mechanisms without success-rate comparisons or universal application support.
Checked sources
These sources support the concepts and methods described here. Examples and test scope are labeled separately; check the version of the tools you use.