How-to
Anthropic Computer Use: How Claude Controls a Screen
Quick Summary
Anthropic computer use is a beta capability that lets Claude view a screen and take actions like a person would, through a loop of screenshots and actions. It works through the Anthropic API, Amazon Bedrock and Google Cloud Vertex AI, and does not need a special integration for each app.
- Your code sends a screenshot and request, Claude returns an action, and your code performs it.
- Setup needs a beta tool definition, a screenshot step, an action handler, and a request loop.
- Run computer use in an isolated virtual machine or container rather than your main work machine.
- Every screenshot is part of the API call, so keep secrets and sensitive data off screen.
- Cap the step limit and define a stop condition so stuck tasks do not keep consuming calls.
Table of Contents
- What Anthropic Computer Use Is and How It Works
- Anthropic Computer Use API: Setup and Required Parameters
- Claude Computer Use Examples in Practice
- Is Anthropic Computer Use Safe? Permissions, Data Handling, and Oversight
- Windows Desktop Automation Tools: Cloud API vs. a Local Windows Agent
- Common Failures and Troubleshooting
- Deciding Whether to Build on the API or Try Able on Your Windows PC
- Frequently Asked Questions
What Anthropic Computer Use Is and How It Works
Anthropic's computer use is a capability that lets Claude view a screen and take actions the way a person would. Anthropic announced it as a beta for developers, available through the Anthropic API, Amazon Bedrock and Google Cloud Vertex AI (Anthropic's computer use announcement).

Anthropic describes the approach as having the model look at the screen and act in the same way a person does (Anthropic's write-up on developing a computer use model). That is the core idea.
The screenshot and action loop
Computer use runs as a loop. Your code sends a screenshot and a request to the API. Claude returns an action, moving the cursor, clicking, or typing.
Anthropic's platform documentation says screenshot images and action requests are processed in real time as part of each API call (Claude platform documentation on computer use). This matters for data handling, which we cover later.
Anthropic Computer Use API: Setup and Required Parameters
To use the API, you need an Anthropic account, an API key, and a sandboxed environment. An isolated virtual machine or container is a sensible choice over your main work machine.
Setup comes down to four concrete pieces. Get each right before you write task logic, because a mistake in any of them shows up later as a confusing misclick or a stalled loop.
1. The beta tool definition
You declare the computer-use tool in the tools array of your request, alongside the model name and the beta header Anthropic requires. The tool definition includes the display width and height in pixels Claude should assume.
A common pattern is to fix one resolution for the whole session and never change it mid-run. Resizing windows or switching monitors mid-task is a frequent cause of coordinate drift.
2. The screenshot step
Your code captures the screen and sends it as an image content block with each request. Anthropic's platform documentation describes screenshots and action requests as being processed in real time as part of each API call (Claude platform documentation on computer use).
Capture the full screen the model is expected to act on, not a cropped region, unless your tool definition says the display is that cropped size. Keep image encoding consistent between calls so the model sees a stable view.
3. The action handler
Claude returns actions such as moving the cursor, clicking, typing text, pressing key combinations, and scrolling. Your handler translates each into real input events on the target machine.
Add a short settle delay after each action before the next screenshot, and verify the screen actually changed before sending the next request. If it did not change, treat it as a failed step rather than looping on the same action.
4. The request and response loop
Two parameters control that loop. A step limit caps how many actions a single task can take, preventing runaway loops and cost. A stop condition, the model signaling completion or your code detecting a target state, ends the loop cleanly.
Watch out: beta parameter names, header values, and tool schema details change between releases, so verify current names in Anthropic's documentation rather than copying an old tutorial.
What the API route costs you in effort
Building on the Anthropic API gives you full control of the loop and lets you embed computer use inside your own software. It also means you write and host the screenshot capture, action handler, sandbox, step limits, and retry logic yourself, and you pay per API call as the loop runs. You can read more about Able on Windows to see how that split works in practice.
The API route needs four pieces in place before any task logic is written.
| Setup piece | What to do | Note |
|---|---|---|
| Beta tool definition | Declare the computer-use tool in the tools array | Display size must match captured screenshots |
| Screenshot step | Send a full-screen image with each request | Every screenshot is part of the API call |
| Action handler | Translate model actions into real input events | Add a settle delay after each action |
| Request and response loop | Cap steps and define a stop condition | Without both, stuck tasks keep consuming calls |
Claude Computer Use Examples in Practice
Typical uses include filling in web forms, moving data between two apps that lack an integration, and running a multi-step research task in a browser. In each case the model reads the screen, decides the next step, and acts.
Start with a small, low-risk task, such as opening a file and renaming it, then build up.
Is Anthropic Computer Use Safe? Permissions, Data Handling, and Oversight
Computer use should be treated as a capability with limits, not a reliable replacement for human oversight. You can limit AI agent permissions on Windows with folder scoping and approval gates so your agent only touches what you allow.
The practical version is a set of permission boundaries you design before the first task runs. Here is how to build them.
Sandboxing: choose the isolation level deliberately
Running computer use in an isolated virtual machine or container rather than your main work machine is a sensible default. The isolation level you pick determines the blast radius of a mistake:
- Full VM. Strongest isolation. The agent can only touch what exists inside the VM, so a destructive action cannot reach your real files. The trade-off is setup effort and provisioning the apps and logins the task needs.
- Container with a virtual display. Lighter than a full VM and good for browser-only tasks. Weaker if the task needs desktop apps that expect a real window manager.
- Dedicated physical machine or spare PC. Useful when the task depends on real hardware or a specific desktop environment. Isolation depends on what else is installed there.
Start every new task type in the strongest isolation you can tolerate, then relax it only after you have watched the task succeed repeatedly.
Credentials: keep secrets out of the screenshot path
Every screenshot you send is part of an API request, so anything visible on screen, passwords, tokens, customer data, private messages, travels with that request. That is the single most important data-handling fact about computer use, and it drives three habits:
- Do not type secrets on screen while the agent is capturing. Log in ahead of time, or use a credential store the agent can invoke without rendering the secret.
- Scope the account the agent uses. Give the agent its own account with only the permissions the task needs, not your admin account.
- Redact where you can. If your capture pipeline can mask known sensitive regions before sending, do it; if not, keep sensitive windows closed during runs.
On a Windows desktop, Able handles this differently: vault logins are typed on your PC so the model never sees the password, and Able acts only after you say yes. Chat and research still go to a cloud model over the internet, so Able is not fully offline, organize, find, and vault work run without a brain, but chat needs internet. You can see how we handle trust and privacy for the details.
Approval gates: decide what needs a yes
A human in the loop is only useful if the loop stops at the right moments. Blanket approval for every click makes the agent unusable; no approval makes it dangerous. Learn how to approve AI actions before they execute on your machine. A workable middle ground is explicit approval for a defined set of high-consequence actions while routine actions run:
- Anything that deletes, overwrites, or moves files outside a scoped folder.
- Anything that sends a message, submits a form, or posts publicly.
- Anything that pays, purchases, or changes account settings.
Everything else, reading a screen, opening a file, navigating a menu, can usually run without a prompt. Write the list down before deployment so the boundary is a decision you made, not one you discover mid-task.
Action scoping: limit what the agent can touch
Isolation controls where the agent runs; scoping controls what it can reach inside that environment. On a Windows desktop, that means scoping folder and app access so the agent sees only the directories and applications the task needs.
Able works this way: it scopes folder and app access and requires your yes before each action, so the agent is able to do the job you approved and not much else.
Logging and review: make failures visible
Log every action and every screenshot during testing and keep the logs for production runs. When something goes wrong, a misclick, a wrong file moved, a form submitted with bad data, the log is how you reconstruct what the agent saw and why it acted.
Watch out: an agent that succeeds ninety-nine times can still fail once in a way that matters, so keep approval gates on destructive and outbound actions even after a task looks reliable.
The bottom line on safety
Treat computer use as a capable assistant with a narrow, explicit permission boundary. Sandbox it, keep secrets off the screen, gate the actions that can hurt, scope what it can reach, and log everything. Do that, and the agent is able to take real work off your plate without taking control of your machine.
Windows Desktop Automation Tools: Cloud API vs. a Local Windows Agent
There are two broad routes for automating a Windows PC with AI.
Cloud API route (such as Anthropic computer use). You build the loop yourself.
Desktop agent route (such as Able). Able is a Windows 10/11 agent, not a local model runner. See how Able differs from a local model runner.
The right choice depends on whether you want to build the automation or use a finished agent on your own PC.
Common Failures and Troubleshooting
Common problems include:
- Misclicks on small or dense interface elements. Use a larger screen size or a clearer window layout.
- Loops that repeat the same action. Add a step limit and a check that the screen changed.
- Authentication or permission prompts that stop the run. Handle these explicitly or approve them manually.
Log every action and screenshot during testing. That makes failures far easier to diagnose.
Deciding Whether to Build on the API or Try Able on Your Windows PC
Build on the Anthropic API if you need a custom workflow inside your own software and you are comfortable writing and hosting the loop.
If you want a finished Windows agent that acts on your files and terminal after you say yes, try Able. Read an honest look at whether Able is worth trying on Windows. You can read more about Able on Windows, compare it with Lapu, or see how we handle trust and privacy. Start a 3-day trial at uable.store.
With Able, you are able to automate the file and terminal work you already do, on the PC you already own.
Frequently Asked Questions
What is Anthropic computer use?
Anthropic computer use is a beta capability that lets Claude look at a screen and take actions the way a person would. The model receives screenshots, returns requested actions such as mouse clicks and keyboard input, and your application executes those actions and sends back new screenshots. It is a developer capability, not a finished end-user app, so someone has to build the loop that carries it out.
Is Anthropic computer use safe?
Review and verify computer-use actions carefully. Screenshots and action requests are sent as part of each API call. Safety depends on how you run it: limit the environment, keep a human in the loop for sensitive actions, and avoid giving the model access to credentials it does not need.
Does Anthropic computer use work on Windows?
Anthropic's computer use is accessed through its API, and the implementation you build controls the operating system it targets. Teams that want to automate a Windows PC usually run the loop inside a sandboxed virtual machine or container. If you want an agent that runs directly on your Windows PC, Able is a licensed Windows desktop agent from Apex Empire Inc. that organizes files, runs terminal commands, and types vault logins after you say yes. Trial is 3 days with no card at uable.store.