Grounding Vision-Language Agents in the DOM: The Element Reference Pattern

The browser is the universal application runtime, yet programmatic interaction with web interfaces has remained remarkably brittle. Selenium scripts break when a class name changes. Scrapers fail when a site redesigns.

Vision-language models (VLMs) like Claude, GPT-4V, and Gemini offer a different approach: give a model a screenshot and let it see what a human sees. This has sparked a wave of autonomous browser agents that combine VLMs with Playwright to navigate websites and complete tasks without hand-written scripts.

We've been building one of these. Ours is an exploration agent that, given a URL, discovers every meaningful action on the page, executes them, and recursively maps the tree of reachable states. The output is a complete graph of user journeys: homepage to product listing to product detail to cart to checkout. This is harder than single-task agents because there's no predefined goal. The agent must discover what's possible, and every page presents fresh decisions: which elements are interactive, which are worth exploring, and how to click each one.

That last part is where things break down.

What You Actually Send the Model

A browser agent has to solve two problems for every action it takes:

1. Semantic understanding: What are the meaningful actions on this page? Which button is the primary CTA? Which links lead to new content vs. which are decorative?

2. Mechanical execution: Given a specific element the agent wants to interact with, produce a way to target it in the DOM and click it.

VLMs are extraordinarily good at problem #1. The question is how you give them enough information about the page to do it and whether the same representation can also solve problem #2. In practice, there are three things you can send the model. Each seems promising. Each breaks in its own way.

Full page HTML

The obvious first attempt. Send the DOM. The problems are immediate:

Size and noise. A typical e-commerce page is 200-500KB of markup, still 50K+ tokens after stripping scripts and styles. Most of it is nested <div>s, SVG icons, and tracking pixels. The model must find 5-10 meaningful actions in 50,000 tokens of angle brackets.

Hidden elements. Raw HTML contains every element, including mobile drawers, collapsed menus, and off-screen widgets. A typical page has 300+ interactive elements in the DOM; only 80-100 are visible. The model will confidently suggest clicking a hamburger menu that's invisible on desktop.

Selectors are still free-form. Even reading real attributes, the model generates selectors as text. It sees class="sc-1x2f3y4 kJmWbR" and writes button.add-to-cart. CSS-in-JS hashes bear no semantic relationship to purpose. And a[href="/products"] is valid CSS but matches 34 elements, an ambiguity the model can't detect.

Accessibility tree

Playwright's aria_snapshot() gives ~4K tokens of semantic structure instead of 50K. But it loses critical information: a hero CTA and footer link are structurally identical - link "Shop All"). Many elements lack labels entirely, and 30-40% of interactive elements on real sites are invisible to the tree. And the tree contains no CSS selectors and no spatial information. You can't target elements or reason about layout.

Screenshot

VLMs reason well about visual layout. They grasp that the big orange button is a CTA and the search bar is in the header. But a screenshot is pixels. You can't address a DOM element from an image alone.

The core tension

Each representation captures something the others miss. And across all three, the model is solving two fundamentally different tasks in one output: decide what to interact with (reasoning) and produce a selector to target it (mechanical precision). One is what LLMs are good at. The other is what they're bad at.

The Element Reference Pattern

The insight: stop asking the model to generate selectors. Instead, pre-extract visible elements, assign each a number, and let the model point by reference.

Before the LLM call, a JavaScript extraction script runs in the browser: find every interactive element, filter out hidden ones, generate a deterministic CSS selector for each, and return a numbered list. Send this alongside a screenshot (for visual reasoning) and the accessibility tree (for semantic structure). The model returns element numbers, not selectors:

1. [link] <a> "Shop All"
   selector: a[href="/collections/all"]
   position: (120, 45) size: 80x24

2. [button] <button> "Add to Cart"
   selector: #add-to-cart-btn
   position: (450, 620) size: 200x48

The model returns element_ref: 2 to click "Add to Cart." It doesn't need to generate the selector because the system already has one. Reasoning and targeting are completely separated.

The screenshot uses a scroll-and-stitch capture rather than Playwright's full_page=True (which breaks viewport units and duplicates sticky headers), and also triggers lazy-loaded content. The accessibility tree is truncated to ~4K tokens. The element list typically contains 80-100 entries after visibility filtering removes the 40-60% of DOM elements that aren't actually clickable.

The Selector Waterfall

he element list's value depends entirely on selector quality. Our extraction uses a 12-strategy waterfall where each strategy generates a candidate and verifies it matches exactly one element via querySelectorAll. The first unique match wins:

# Brief idea

Strategy 1-3:   #id, [data-testid], [data-test]         (most resilient)
Strategy 4-7:   a[href], [name], [aria-label], [data-*]  (semantic attributes)
Strategy 8-9:   #ancestor descendant, tag.class           (scoped combinations)
Strategy 10-12: nth-of-type, class paths, structural      (positional fallback)

The ordering is a deliberate tradeoff: early strategies produce selectors that survive deploys (IDs, test attributes); later strategies are fragile but guaranteed to produce a unique match for any element. For shadow DOM, the script recursively traverses open shadow roots and composes selectors as hostSelector innerSelector. Visibility checks cross shadow boundaries to catch hidden drawers and overlays.

Element Resolution

The model returns actions with an element_ref (usually correct) and a selector (often hallucinated). The resolution layer ignores the model's selector and looks up the real one.

if action.element_ref is not None:
  idx = action.element_ref - 1
    if 0 <= idx < len(elements):
      action = action.model_copy(update={"selector": elements[idx].selector})

About 60% of actions need selector correction because the model writes button.add-to-cart when the real selector is #product-widget button[data-action="addToCart"]. This is fine. The element reference is all that matters.

Selectors also pass ambiguity detection (rejecting 50+ matches) and visibility verification. When a selector matches 2-5 elements, the system refines it by re-running the waterfall on the resolved element, producing a unique selector for deterministic replay.

Key Takeaways

LLMs and deterministic code have complementary strengths that shouldn't be mixed. Asking a model to generate a CSS selector is asking it to do something a 50-line JavaScript function does better. Asking JavaScript to decide which actions matter is asking it to do something a vision model does better.

The model outputs an integer. The system turns it into a click. When one side fails, the failure is isolated and fixable without touching the other.

This generalizes beyond browsers. Any system using LLMs to drive actions in a structured environment faces the same tension between flexible reasoning and mechanical precision. Let the model reason about what. Let deterministic code handle how.