Computer Use and Browser Agents: AI That Clicks and Types for You

How screen-controlling AI works through screenshots, why it is still more fragile than APIs, when to use it, and how to manage the risks.

The HR team at a factory has to key new employee details into an old payroll system every month. The system has no API and no import button, only screens where someone clicks and types field by field. This is exactly the kind of work Computer Use and Browser Agents were built for: using the screen the way a person does.

What are Computer Use and Browser Agents

Computer Use is the ability for AI to see the screen through screenshots and control the mouse and keyboard. A Browser Agent does the same but stays inside a web browser. Both differ from ordinary Tool Calling in that they need no API; the agent operates software designed for humans directly.

Anthropic released computer use as a public beta in October 2024. OpenAI launched ChatGPT agent in July 2025, describing it as building on Operator and deep research, with a visual browser, a text browser and a terminal.

Under the hood: a loop of images and actions

A browser agent uses the same Agent Loop as other agents, but it observes through "images":

1. Observe: take a screenshot of the current screen
2. Plan: the model interprets the image: which page it is on and where the button it needs is
3. Act: move the cursor to coordinates, click, or type text
4. Take a new screenshot to check whether the action worked as expected

Claude's tool use documentation places the computer and browser tools in the group your own code runs, which means you must provide the machine or environment the agent uses. That is an important security point.

Why it is still harder than it looks

The OSWorld benchmark (published in 2024) built 369 tasks on real computers and reported that humans succeeded on more than 72.36%, while the best model at the time managed 12.24%. The main problems were locating elements on the GUI and lacking knowledge of how programs work.

At launch, Anthropic reported that Claude 3.5 Sonnet scored 14.9% on OSWorld in the screenshot-only category, and acknowledged limitations such as scrolling, dragging, zooming and being slower than humans. Scores on this benchmark change quickly, so check the latest results before deciding, but the lesson still holds: screen-based work is more fragile than API-based work.

When to use it, and when not to

Good fit

• Legacy systems with no API that are not worth further development
• Gathering information from many websites with different structures
• Low-volume repetitive work where slower-than-human speed is acceptable

Poor fit

• Work that already has an API, because tool calling is faster, cheaper and more accurate
• High-volume work needing high reliability, such as processing thousands of documents
• Anything involving money or sensitive data without a person watching

Watch out for

• Prompt injection from web pages: OpenAI states plainly that hidden instructions on web pages could make the agent take unintended actions, such as sharing data from connectors, and acknowledges that the overall risk is higher than before
• Permissions of the logged-in account: the agent can do anything that account can do. Use limited-permission accounts and a separate machine or VM for the agent
• Keep people in control at key points: ChatGPT agent lets users take over the browser to log in, asks permission before actions with real consequences, and requires users to watch certain tasks such as sending email. The same principles apply to systems you build. Read more in Guardrails and Human-in-the-loop

If there is an API, use the API. If not, let AI click for you, but never let it click alone.

How to get started

1. Pick one weekly, repetitive screen task that cannot be connected through an API
2. Write down the steps a person follows in detail, with the correct result of each step
3. Always test on a test account or dummy data first
4. Have the agent stop and wait for a person to confirm before saving or sending
5. Time it and count errors against a person doing the task, then decide whether it is worth it

Want to try an agent that works in the browser for you? Start with the course ChatGPT Agent: Let AI Do the Work on Plants.work Learn, then apply it to real work in your team.