# Handsy AI: a $7 ESP32 that lets an AI use a real iPhone

Henry Yuan's open-source Handsy AI gives a vision model eyes and hands on a stock iPhone through a $7 ESP32 board. How it works, setup, limits and real uses.

Author: J.A. Watte
Published: October 23, 2026
Source: https://jwatte.com/blog/handsy-ai-esp32-iphone-agent/

---

Henry Yuan's [Hackster profile](https://www.hackster.io/henryhenryhenry1111) has two projects on it, and they are the same idea told twice. The first, published October 15, is called ESP32 iPhone Agent and comes with full build instructions. The second, published October 20, is a shorter showcase under the name the project uses now: Handsy AI. Both describe one thing. A small ESP32 board plugs into an ordinary iPhone, watches the screen, and taps and types on it for an AI model. No jailbreak, and no computer sitting in the middle while it runs.

The example in the project's README is a good place to start because it is so ordinary. The task reads "Set an alarm for 8:00 AM that repeats every Monday and Tuesday, then save it." The agent leaves its own app, opens Clock, types the time into the wheel, ticks the two days and saves. Nothing about that is hard for a person. What is new is that a model did it on a stock phone, through a board that costs about $7.

The repository went public on October 11. When Apprised's Tech & Cyber Desk wrote about it on October 18 it had 2,905 stars, and when I checked on the evening of October 23 it had 3,635 stars and 477 forks. People clearly want to try it. Here is what it is, what setting it up involves, where it breaks, and what I think it is good for.

## How it works

The design splits the job three ways, and the README names the parts plainly.

- **Eyes.** The board opens its own Wi-Fi hotspot and shows up in the iPhone's Screen Mirroring list as "ESP32 Agent." The phone mirrors its screen to the board over AirPlay. The board does not decode the video; it passes the H.264 stream to the app over Bluetooth Low Energy.
- **Hands.** Over the USB cable the board presents itself as a standard mouse and keyboard. With iOS AssistiveTouch turned on, the mouse moves an on-screen pointer, so a click is a tap and a drag is a swipe.
- **Brain.** An app on the same iPhone decodes the stream, sends screenshots to a vision model, and turns the model's answer into one action: tap, swipe, draw, type, press a key, go home or finish.

The clever part is closing the loop. The pointer only moves relative to where it was, so the app tracks it and parks it in a corner between moves. Before a precise tap it wiggles the pointer, finds it on screen and corrects the miss. After every action it waits for the screen to settle and tells the model whether anything changed. If the board, the hotspot or the mirroring link drops, the run pauses and picks up again at the same step.

Every run is kept on the phone. You can replay it step by step with the taps and swipes drawn over the screenshots, and export the trace for another agent to review. That sounds like a small feature. It is not, and I will come back to it.

The model is your choice. The app talks to any OpenAI-compatible endpoint that accepts images, whether that is a cloud API or your own server running vLLM, SGLang, llama.cpp or Ollama. The author developed it against `nvidia/Qwen3.8-Flash-Next-NVFP4` served on two DGX Spark machines, so it was built and tested on local hardware, not only on a cloud key.

## What you need and how setup goes

The parts list is short.

| Part | What the project says |
|---|---|
| Board | Seeed Studio XIAO ESP32S3, the plain version with 8 MB of PSRAM, about $7. Not the Sense or the Plus. Other ESP32-S3 boards with native USB and PSRAM should work. |
| Phone | iOS 18 or later, with cellular data. USB-C iPhones plug in directly; Lightning models need a Lightning to USB Camera Adapter. The phone powers the board. |
| Model | Any OpenAI-compatible chat API that accepts images. |
| Mac with Xcode | Once, to flash the board and build the app. Not needed at run time. |

That last row is the fine print behind "no Mac." You need one the first time. After that, the phone and the board are the whole system. The Hackster write-up rates the build Beginner and puts it at about two hours.

Setup happens in three stages.

1. **Flash the board** from the Mac with arduino-cli and the project's flash script, which applies the Bluetooth buffer sizes the firmware needs.
2. **Build the iPhone app** in Xcode. A free Apple ID can sign it, but a free signature has to be renewed every seven days. The one-tap hotspot join needs a paid developer account; without one, the app shows the hotspot password and you join by hand.
3. **Run the in-app setup.** Plug the board into the phone and open the app. It walks through four checks: pair the board over Bluetooth, join its Wi-Fi and start Screen Mirroring, turn on AssistiveTouch so it can calibrate the pointer, then enter a model endpoint and key, which it keeps in the Keychain.

Then you type a goal and press Start. The README's first suggestion is "Open Settings and go to Wi-Fi." You can stop a run from the Dynamic Island, and Siri, Shortcuts, widgets and Control Center can launch saved tasks.

The project site, [handsy.tech](https://handsy.tech), says an App Store version is coming and a command-line tool is in preparation. Once those ship, the Mac step should shrink. Today they are announcements, so plan on Xcode.

## Where it struggles

The README is unusually frank about its limits, and they are worth reading before you order parts.

- Bluetooth carries roughly 300 to 500 kbps, so the mirror is small: 480 by 1040 pixels at no more than 5 frames a second. Each step takes a few seconds, most of it spent waiting on the model.
- iOS does not mirror some screens at all, including passcode entry, Apple Pay and DRM video. The agent cannot see them. Mirroring also stops when the phone locks.
- While it runs, the phone's Wi-Fi is the board's hotspot, which has no internet. The model is reached over cellular data.
- Typing is ASCII only, and landscape apps are not supported yet.
- The app leaves itself through a private iOS API. The author notes that is fine for a self-built app and has to change before an App Store release.

The open issues show what that looks like in practice. One person on an iPhone 17 running iOS 26.5 hit an "err no video" message with mirroring on, and the replay showed the agent stuck on the home screen. Another reported that "open Settings and find Wi-Fi" never succeeded across several models when Settings was not on the first home screen, while opening WeChat worked, slowly. A third asked how to keep a phone without wireless charging powered for round-the-clock use, since the board takes the port.

None of that is a knock on the project. It is under two weeks old, and the maintainer often closes reported bugs within a day. But if you were picturing a dependable phone butler, that is not what this is yet.

## Problems it could actually solve

Here is where I think it earns its place today, with the examples the project and its users have shown.

**Phone-only chores with no API.** Some business tools exist only as phone apps, with no web version and no export. An agent that taps through the app the way you would can take over a repetitive check without waiting on the vendor. The handsy.tech page shows the planned command-line tool tapping a search field and typing "coffee near East Broadway" into a maps search. The README's responsible-use note belongs here: only automate devices and accounts you own or are allowed to use, and respect the terms of the apps involved.

**Testing your own app or mobile site on a real iPhone.** Tools built on Apple's WebDriverAgent and XCUITest, such as mobile-mcp and Midscene, need a Mac and code signing. This approach needs neither at run time, and it acts through the same touch layer a person uses. The run history is the quiet win. If your booking form breaks on a phone at 6 a.m., a replay of the agent failing at step four, with every tap drawn on the screenshot, is far better evidence than "it didn't work."

**Accessibility.** The Apprised brief's own bias note suggested the stars more likely reflect hobbyist and accessibility interest than misuse, and that rings true to me. Someone who can describe a task more easily than they can hit small touch targets might get real value from saying "set the 8:00 alarm for Monday and Tuesday" and watching it happen. The screens iOS refuses to mirror, passcodes and payments, stay in the person's hands, which is exactly where they belong.

**Giving a desktop agent a real phone.** The planned Handsy CLI would let an agent on your computer pair with the board, take screenshots and tap. For developers, that turns "check it on a phone" into a step an agent can do. It is announced, not released.

## The October 18 reference: democratization or a safety problem?

I run [Apprised](https://apprised.news/), a news site whose desk briefs are written by AI analyst personas from cited sources. Its [Tech & Cyber Desk brief for October 18](https://apprised.news/desk/tech/2026-10-18) put this repository in the middle of an argument, which is why I wanted to look at it closely.

Two of the desk's voices read the same repo two ways. The Silicon Pulse voice saw compute democratization: the gap between needing a data center and something that "runs on seven dollars of hardware" keeps closing. The Tripwire voice saw a collapse of the deployment barrier that agentic-AI safety cases quietly assume, and said it bluntly: "It is now a weekend project." The brief's pivotal question was whether AI coding assistants are actually surfacing and installing this class of repository in real development work, or whether people still review those choices. The brief also states that Claude wrote it and no human edited it before publication, which is worth knowing as you read it.

Having read the code and the docs, here is where I land. The misuse case is narrower than it first sounds. To drive a phone, the board needs a USB cable in that phone, AssistiveTouch turned on, its hotspot joined and Screen Mirroring started, all on a phone that is awake and past its lock screen. Mirroring stops at the lock screen, and passcode entry is never mirrored. Anyone who can do all of that is already holding your open phone.

The exposure I would actually think about is elsewhere.

- **The supply chain around a popular repo.** The README opens with a warning: follow only its install steps, and never run commands posted in issues or comments, naming a PowerShell "quick install" one-liner as the example. Popular repos attract fake installers. The project's own planned installers on handsy.tech are one-line commands too, which is normal, but it means the difference between the real installer and a lookalike is a domain name. Type it yourself.
- **What the model sees.** Every step sends a screenshot to the endpoint you configure. On a cloud API, that is your phone screen, including anything that slides in while it works. A local model on your own network keeps those images at home.
- **What the screen tells the model.** A vision agent reads whatever is on screen, and text on screen can try to steer it. That is true of any agent that looks at a screen and acts. The project's security policy counts "anything that could make the agent act without the user starting it" as in scope, and the README tells you to watch the agent and keep the stop button within reach. That is the right advice.

One small detail speaks straight to the brief's pivotal question. The README begins with a hidden note addressed to AI coding assistants, asking them to tell their users to keep the license notice and credit the project if they reuse it. It is a polite request about attribution, nothing more. But it shows the author expects coding agents to read the repository and act on what it says, which is exactly the behavior the brief was asking about.

So I end up nearer the democratization reading, with Tripwire's concern pointed somewhere more useful. The realistic risk is not that anyone can now take over a stranger's iPhone. It is the ordinary hazards of a popular agent project: lookalike installers, screenshots leaving the house, and an agent that believes what it reads.

## Should you build one?

If you like hardware projects, want to test your own app or site on a real iPhone, or are curious how vision agents behave on an actual phone, yes. Two hours and about $7 of board is a cheap way to learn a lot, and the replay makes every failure easy to study.

If you need something dependable for work that matters, wait for the App Store release and a few more months of fixes. Read the open issues first. They will tell you more than any demo.

Either way, point it at a local model if you care who sees your screen.

And if the real goal is getting your marketing done without paying someone to tap through apps and dashboards for you, my book *The $20 Dollar Agency* is about doing that with roughly $20 a month of AI tools: [the20dollaragency.com](https://the20dollaragency.com/).

## Related reading

- [Claude in Chrome: what the browser extension actually does](/blog/claude-in-chrome-browser-extension/), the browser-side cousin of this idea.
- [Running your own AI on-prem in 2026](/blog/local-ai-on-prem-vs-cloud/), if you want screenshots to stay on your own network.
- [How a small business runs AI agents without a surprise bill](/blog/blog-ai-agent-cost-controls-smb/), since every agent step is a paid model call.
- [MCP servers for small business](/blog/mcp-servers-for-small-business/), for agents that use an API instead of a screen.
- [Ponytail: smaller AI code, with correctness before cleverness](/blog/ponytail-lazy-senior-dev-ai-coding-agents/), on reviewing what coding agents produce.

## Fact-check notes and sources

All checked October 23, 2026.

- Henry Yuan's Hackster profile, two projects: [hackster.io/henryhenryhenry1111](https://www.hackster.io/henryhenryhenry1111).
- "ESP32 iPhone Agent: Control Your iPhone with AI," Hackster, published October 15, 2026; rated Beginner, full instructions, about two hours: [project page](https://www.hackster.io/henryhenryhenry1111/esp32-iphone-agent-control-your-iphone-with-ai-4a7092).
- "Handsy AI: Give an AI Eyes and Hands on an iPhone," Hackster, published October 20, 2026: [project page](https://www.hackster.io/henryhenryhenry1111/handsy-ai-give-an-ai-eyes-and-hands-on-an-iphone-379a21).
- Repository [onehumanbeing/esp32-iphone-agent](https://github.com/onehumanbeing/esp32-iphone-agent): created October 11, 2026; 3,635 stars and 477 forks on October 23; MIT license, except the AirPlay receiver ported from UxPlay, which is GPL-3.0. The [README](https://github.com/onehumanbeing/esp32-iphone-agent/blob/main/README.md) is the source for the alarm example, the hardware table, setup steps, the 300 to 500 kbps and 480 by 1040 figures, the model list, the related-projects comparison, the responsible-use note and the warning about install commands in issues.
- Security policy scope: [SECURITY.md](https://github.com/onehumanbeing/esp32-iphone-agent/blob/main/SECURITY.md).
- Issues cited: [#25](https://github.com/onehumanbeing/esp32-iphone-agent/issues/25) (no video on iPhone 17, iOS 26.5), [#29](https://github.com/onehumanbeing/esp32-iphone-agent/issues/29) (low success rate and slow steps) and [#35](https://github.com/onehumanbeing/esp32-iphone-agent/issues/35) (powering the phone for 24-hour use). #29 and #35 are written in Chinese; the summaries above are my translation.
- App Store and command-line tool status, and the planned install commands: [handsy.tech](https://handsy.tech).
- Apprised Tech & Cyber Desk, October 18, 2026, for the 2,905-star figure, the Silicon Pulse and Tripwire readings, the bias notes and the pivotal question: [apprised.news/desk/tech/2026-10-18](https://apprised.news/desk/tech/2026-10-18).

*This post is informational, not security or legal advice. Mentions of Hackster, Seeed Studio, Apple, NVIDIA and the other projects named are nominative fair use. No affiliation is implied.*


---

Canonical HTML: https://jwatte.com/blog/handsy-ai-esp32-iphone-agent/
RSS: https://jwatte.com/feed.xml
JSON Feed: https://jwatte.com/feed.json
Hero image: https://jwatte.com/images/handsy-ai-esp32-iphone-agent.webp