The sidebar agent, but on every web page

The in-app sidebar works because the agent is sitting next to the running app and knows the current route. But most of the time, the page you're trying to do something with isn't your app — it's somebody else's. A pricing page, a docs site, an admin console, a half-broken SaaS dashboard.

So we shipped the same agent as a Chrome extension. Same backend, same tools — but the side panel attaches to the active tab, and the agent can act on whatever page is loaded.

The shape

Manifest v3, side panel UI, persistent service worker. The relevant declaration:

{
  "manifest_version": 3,
  "name": "SkillFactory Agent",
  "permissions": [
    "sidePanel", "activeTab", "tabs",
    "tabGroups", "storage", "scripting", "offscreen"
  ],
  "host_permissions": ["https://*/*", "http://*/*"],
  "side_panel": { "default_path": "sidepanel.html" }
}

Side panel runs an agentic chat loop. Each user turn:

1. Send the message + current page context (URL, title, extracted text) to the server's tRPC endpoint.
2. Server returns either an assistant reply or one or more tool calls.
3. Tool calls split: page tools execute locally in the extension, server tools (web search, media, entity CRUD, memory, TTS) execute remotely.
4. Tool results feed back into the model. Loop until the model returns a plain reply or hits the iteration cap (8).

The page tool

The interesting tool is page. It's a single tool with a method parameter that gates a verb table:

The scan output is the load-bearing primitive. It's a single-pass plain-text representation:

# https://example.com | Example Domain | 47 nodes
[main]
  H1 Example Domain #h1
  L u4 More information... <https://www.iana.org/domains/example>
[nav]
  L u7 Privacy <https://example.com/privacy>
  L u8 Terms <https://example.com/terms>

Codes are single letters (L link, B button, T textbox, S combobox, H heading, etc.). Sections ([main], [nav], [form:login]) bracket their children. Same-origin iframes are inlined; cross-origin iframes show as - iframe#cross-origin <src>.

UIDs are scoped to a single scan. Any DOM mutation, navigation, or route change invalidates them. The error stale uid means exactly that — re-scan.

This format was chosen deliberately. JSON is too verbose for the model to chew through quickly; full a11y trees are too noisy. Compact plain text with stable codes turns "find the submit button" into a sub-second pattern match.

The architecture, briefly

┌──────────────────────┐     ┌────────────────────┐
│  side panel (UI)     │     │  active tab (page) │
│  - chat              │     │                    │
│  - tool dispatcher   │     │  content script    │
│  - voice I/O         │←──→│  (injected on      │
└─────────┬────────────┘     │   demand)          │
          │                  └────────────────────┘
          │ tRPC over fetch
          ▼
┌──────────────────────┐
│  server (df-ai-court)│
│  - agent loop        │
│  - server tools:     │
│      web, media,     │
│      entity, memory, │
│      speak           │
└──────────────────────┘

The service worker is light — mainly token management and dispatching scripting calls. The heavy lifting is split between the side panel (UI + page tool execution via chrome.scripting.executeScript) and the server (model calls + non-page tools).

What it's actually good for

A few patterns that have come out of using it:

• Form-filling on opaque admin tools. You're three layers deep in a SaaS console with 14 dropdowns and a poorly labelled toggle. "Find the option that controls X and turn it off" — agent scans, identifies, clicks. Faster than reading the docs.
• Content extraction. "Pull the prices and SKUs from this comparison table into a structured note." Page tool scans, agent calls entity server tool to persist.
• Cross-page workflows. Navigate → scan → extract → navigate to next URL → repeat. The model holds the loop; the page tool is its hands.
• Debugging your own apps from the user's perspective. Open the dev URL in a fresh tab, ask the agent to walk through the signup flow. It does, and reports anything that fails.

What's deliberately not there

• No autonomous form submission on financial or destructive endpoints without confirmation. The agent's system prompt gates these; the user has to type explicit consent for purchase, delete, or transfer actions.
• No cross-origin scraping at scale. host_permissions is broad, but rate limits and the per-tab attachment make this unsuited for crawling. Use a proper crawler.
• No ambient observation. The agent only acts on the current tab when invoked. It doesn't watch your tabs.

The friction points

• Stale uids are the #1 source of agent errors. Models forget to re-scan after click triggers a route change. We handle this with a clear error and an explicit instruction in the tool description, but it costs round-trips.
• Cross-origin iframes are blind spots. The scan marks them but can't reach into them. For SaaS apps that embed a different-origin checkout flow, the agent has to ask the user to navigate manually.
• Voice mode + side panel + Chrome's audio policies is the most fragile part of the codebase. Works, but every Chrome update is a coin flip.

Where this is going

The next obvious extension (no pun): page tool macros. "Save this sequence — login, navigate to billing, pull the latest invoice, save to entity store" — record once, replay anywhere. The infrastructure is mostly there; what's missing is a UI for naming and triggering them.

After that: multi-tab orchestration. The agent already has tabs and tabGroups permissions but doesn't use them yet. A workflow that opens five product pages, scans each, and assembles a comparison table — that's a side panel away.