magneto-browse

The problem

Language models are good at deciding what to do on a web page and bad at seeing one. Raw HTML is too large and too noisy; screenshots are expensive and lossy for text. And the moment an agent needs to log in somewhere, you face a choice between giving the model a password and not logging in. I wanted a tool that solved both: a small, faithful view of the page, and a way to authenticate that keeps secrets out of the model's context entirely.

My role

Everything: the protocol, the server, both browser back ends, the vault, the tests, and the operational hardening that followed a security finding in my own code.

What I built

• An MCP server that exposes browser tools (navigate, snapshot, click, fill, extract, wait) to any compatible model client over a token-gated local HTTP endpoint.
• Accessibility-tree snapshots. The page is digested into a compact tree of roles, names, and stable references. The model clicks a reference, not a CSS selector. Server-side diffing returns only what changed between actions.
• Two browser back ends behind one seam. A Chrome extension and, later, a native macOS app built on WKWebView. Both speak the same command protocol over a local WebSocket, so the server and every tool are shared. The native app can synthesize trusted input events, which matters for sites that check whether a click came from a human.
• Credential vault. Credentials are stored locally, keyed by origin. A login tool looks up the credential for the live origin, types it in the browser, submits, and returns booleans. The password appears in no tool result and no log.
• Origin gating and site memory. Every command carries the origin it expects; the tool refuses if the tab has drifted. Small per-site notes persist what worked last time.

Architecture

Difficult problems

The bug I found in my own tool. An audit turned up a cross-site WebSocket hijacking flaw: the local bridge accepted connections without validating the Origin header, so any web page could have connected and, one hop later, reached the vault. Fixed with strict origin validation and per-role tokens. It also became the thesis for a bug-bounty focus on local services and desktop attack surfaces, because that is the class of bug building this tool taught me to look for.

Time-of-check versus time-of-use on login. The vault looks up a credential by origin, but the page can navigate between lookup and typing. The native client re-checks the live origin immediately before typing and fails closed.

// vault login returns only facts about what happened, never the secret
{ "username_filled": true, "password_filled": true, "submitted": true, "final_origin": "https://github.com" }

Hidden file inputs. Upload widgets like Gmail's hide the real input element. A visibility gate that was right for buttons was wrong for file inputs; the check now asks "is this a file input that is enabled" rather than "can a human see it."

Widgets that check isTrusted. Some sites ignore synthetic clicks. The extension cannot fix that; the native back end can, by generating hardware-level events. That asymmetry is why there are two back ends.

Decisions and tradeoffs

• Accessibility tree over screenshots. Far cheaper per step and exact for text; weaker for purely visual layouts, so a visual snapshot tool exists as a fallback.
• WKWebView over embedding Chromium. A native app in a few thousand lines instead of a browser build. The tradeoff is Safari's engine and its quirks.
• Refusing beats guessing. If the origin does not match, the tool returns an error. Agents recover from an error more reliably than from a silent action on the wrong page.

Impact