LLM Testing in Practice: How To Test an AI Browser Extension End to End

A bug report, roughly as it reached me: “the extension broke on a student's Chromebook, inside a school portal's iframe, on a Google Drive PDF, on page three.”

One sentence. Four moving parts. Each one capable of failing on its own, and not one of them a crash.

That was the first real bug I got on the product, and it turned out to be the shape of every bug that followed: several layers you only partly control, and a model at the end of them that will not fail loudly. If you ship AI features, you have a version of it waiting. Ours arrived wearing Chromebooks and PDFs. Yours might be a retrieval pipeline that quietly returns the wrong chunk for one customer's document format.

The platform I test is an AI-powered product used in schools. It does text amplification, translation, read-aloud, and word support, and it delivers all of that through a browser extension that sits on top of content nobody on the team controls: Google Docs, Slides, PDFs in Drive, native browser PDFs, third-party curriculum sites. The surfaces are specific to us. The testing problem is general.

What follows is how we actually test it: what belongs at each layer, how to assert on output no one can predict, and where tests stop being the right tool.

Why LLM Testing Breaks the Usual Testing Playbook

Most testing advice quietly assumes two things.

You control the page. On an AI product that runs on top of someone else's content, you don't. Take one of the curriculum sites our extension works on. Our code goes looking for the lesson text in a particular place in the page's structure, the DOM: the page as code, which is what our extraction reads rather than what a person sees on screen.

Their team redesigns that lesson page next Tuesday. We ship nothing, and we change nothing, but the text has moved, so our extraction now finds half the lesson instead of all of it.

Nothing crashes. No error message appears anywhere for the student or for us. The student still gets an amplified passage in the side panel, and it still looks right. It was just built from half the lesson, and there is nothing on screen to indicate as much. Our tests pass, the build is green, and every student on that site gets a degraded lesson until somebody notices.

Ours is a curriculum site changing its pages. Yours might be a customer changing the format of the documents they send you. Same failure either way: something outside your control moves, and your product gets quietly worse without anyone getting an error.

Output is deterministic. It isn't. You cannot write expect(output).toBe('...') when a model chooses the wording. Every assertion you'd normally reach for is off the table.

An LLM testing strategy has to answer both at once: test across surfaces you don't own, and assert on output you cannot predict. Everything below follows from those two constraints, so if your stack looks nothing like ours, that is the part to map across.

Which LLM Testing Layer Each Bug Lives In

Our backend is a Node.js and Koa API with MongoDB and Redis, a managed auth provider, and calls out to several large language model providers. The frontend is a Manifest V3 Chrome extension with a side panel, a popup, and content scripts injected into third-party pages. The frontend is a Manifest V3 Chrome extension with a side panel, a popup, and content scripts injected into third-party pages.

The backend is conventional. It's an API, and the usual patterns for structuring a Node.js service apply to testing it too.

The frontend is not. Most frameworks are not built for "compile the extension, launch a real browser with it loaded, authenticate against a live backend, open a Google Doc, and assert that AI output appears in the side panel." That infrastructure had to be built.

Which produces three layers, roughly along the lines of the practical test pyramid, though on an AI product what each layer is for shifts.

Layer 1: Unit Tests, Around 300 Files

Mirror your test tree to your source tree, so every route, service, and utility has a test sitting next to it and gaps are obvious at a glance. Our Jest suite is laid out exactly that way.

Route tests confirm an endpoint enforces access control, validates input, and returns the right shape, without touching a real database. Mount the app with an in-memory session and a mocked model layer. A shared helper pins the session config and environment every test file starts from:

ts
export const TEST_SESSION_CONFIG = {
  key: 'app:session',
  maxAge: 86400000,
  httpOnly: true,
  secure: false,
  sameSite: 'lax' as const,
  renew: false,
};

export function setupTestEnvironment() {
  process.env.SESSION_SECRET = 'test-secret-key';
  process.env.NODE_ENV = 'test';
  process.env.CI = 'true';
}

These catch what fails silently. A student hitting an admin-only route should get a 403, not a 500. Products like this carry real role hierarchies: student, school admin, platform admin, each seeing a different slice of the product. Any B2B AI product has the same shape and the same obligation to test every boundary, including the ones only your lowest-privilege account will ever hit.

Input validation belongs here too. Zod schemas sit at the edge, and route tests confirm a malformed payload is rejected with the right status instead of sailing through to a service that assumed clean data.

Service tests cover the logic underneath: extraction pipelines, OCR caching, model dispatch, licensing. Mocked dependencies, no HTTP layer.

Asking a model to interpret a diagram in a textbook image is expensive, so its timeout is configurable, and the resolution logic has to be exact:

ts
it('TC-FUS-013: parses a valid numeric env override', () => {
  expect(resolveFigureUnderstandingTimeoutMs('15000')).toBe(15000);
});

it('TC-FUS-014: falls back to the default for missing, NaN, or non-positive values', () => {
  const DEFAULT = 20_000;
  expect(resolveFigureUnderstandingTimeoutMs(undefined)).toBe(DEFAULT);
});  

That TC-XXX-NNN prefix is worth stealing. Each test gets a stable ID that survives refactors, so six months later someone greps it from a ticket and lands on the exact assertion. Test names drift as behaviour gets reworded. IDs survive it.

Layer 2: Backend Integration Tests

A separate suite runs the full server against a real MongoDB instance, or an in-memory substitute in CI. Real routes, real middleware, full request lifecycle.

This catches what unit tests structurally cannot: session middleware interacting with route ordering, cross-middleware side effects, the mock that was subtly lying. Our layer here is deliberately thin, with the infrastructure ready to grow as the API surface does. If you're early, that's a reasonable trade to make, as long as you make it deliberately.

Layer 3: End-to-End Tests Against Surfaces You Don't Own

This is where an AI product diverges the hardest from a normal web app, and where most teams either skip E2E or fake it.

If your product ships as a browser extension, your E2E test has to do what a user's browser does: build the extension, launch Chromium with it loaded, derive the extension ID at runtime, authenticate, navigate to a third-party page, exercise the extension against live content, and assert on what appears in the side panel. Playwright loads an unpacked extension through a persistent context, which is the foundation. Auth is what decides whether the suite is worth anything.

We mint a server-side session directly, in the format the backend uses in production, and set the signed cookies before any test runs. The whole thing hangs on one contract: the suite has to sign with the same secret the backend verifies with, or every cookie it sets is garbage.

bash
TEST_API_URL=http://localhost:3005
SESSION_SECRET=dev-secret-key   # must match backend
MONGO_URL=mongodb://localhost:27017/app

The tests go through the real auth path. Mocking auth hides an entire class of session, cookie, and CORS bugs, and those are exactly the bugs that only appear on a locked-down, managed device in the field. Your equivalent is the enterprise customer behind SSO whose browser policy you will never reproduce with a stubbed token.

Organise E2E Test Coverage by Surface

Any LLM testing strategy running against pages you don't own needs a file structure that admits it. Nearly 40 spec files, grouped by where the content lives, because that is what actually varies:

• Standard HTML pages: amplify output exceeds 200 characters, no JS errors on the page.
• Google Docs: the popup triggers the panel, and both amplify and translate work on the same document.
• Google Docs formatting: output contains list HTML, and there's no selection flash after extraction.
• Google Slides: the full popup-to-output flow.
• Drive PDFs: page navigation triggers re-amplification; typing alone does not.
• Native PDFs: the handoff for non-selectable text, and scanned PDFs trigger OCR.
• Third-party curriculum sites and embedded iframes: each with contract-style assertions of its own.

That last group is where the surprises live, because those sites owe us nothing. One of them signs its content with a short-lived key, so a student who sat on a page too long used to lose access and have to close it and open it again. A content change observer handles that now. It's the kind of failure that stays invisible until it isn't, and it only surfaces if you test against the live site.

The general rule: your spec files should map to the axes of variation in your environment. If your product behaves differently per tenant, per document type, or per integration, that is your file structure. Organising by feature feels tidier and tells you nothing about where you're exposed.

A Worked Example: Testing a Confirmed-Intent Trigger

Pagination is where the logic gets genuinely subtle. When a student moves through a Drive PDF, the extension has to confirm that navigation actually happened before it re-amplifies. Someone touching the page-number box is not navigation.

Four scenarios encode it:

1. Typing a page number without pressing Enter: no re-amplification.
2. Typing a page number, waiting more than two seconds, then pressing Enter: re-amplify.
3. Scrolling to the next page: re-amplify.
4. Clicking a page thumbnail: re-amplify immediately.

Four tests, one product decision: act only on confirmed intent. Every AI product has a version of this because every model call costs money and latency. When do you fire the request? On debounce, on blur, on explicit submit? Whatever you chose, it's a decision living in code that nobody wrote down, and the day someone refactors the trigger, your most common user workflow breaks quietly.

Which is the rule worth taking away: test names should encode product decisions. "Typing a page number without Enter does not re-amplify" is a product rule, phrased the way a stakeholder would argue about it. When it fails a year from now, the name tells you what broke and why anyone cared.

Snapshot Testing and Surface-Tagged CI Runs

For extraction fidelity, Playwright snapshots compare what comes off a live page against a stored baseline. When a site quietly reshuffles its DOM and extraction degrades from good to mediocre, a snapshot diff catches it. A unit test never will. Anywhere your input is someone else's output, a snapshot is the cheapest regression net you can hang.

Every spec is tagged so CI targets a surface when only that surface changed:

bash
npx playwright test --grep "@googledrive"
npx playwright test --grep "@pdf"

The E2E suite runs on every PR alongside unit tests: 200 specs, 200 of 200 passing on the last run. The wall-clock time moves around, because these tests make live network calls, so the number worth watching is the pass result rather than the duration. Run locally first, because a base-branch conflict or a flaky spec is much cheaper to find before the CI pipeline is involved.

Test the Contract, Not the Content

Here's the core of it, and it's simpler than the tooling market makes it sound. Design every assertion around what the model must never do. You cannot pin what it will say, so pin the boundaries it cannot cross: error, time out, return empty, drop structure, drift from the source.

Four assertions carry almost all the load:

Output exists and is substantial. On the HTML page specs, that's a check that amplify output runs past 200 characters, which confirms the model returned something rather than timing out or erroring silently.

Output overlaps with the source. Semantic overlap tells you the model rewrote the input rather than inventing something new. On a product where a user trusts what they read, that assertion earns its place.

Format survived. Translate a formatted document, and the list structure has to come through. Losing structure is a real regression that a text-only assertion misses entirely.

The console is clean. An error scan catches silent client failures that never surface in the UI.

For backend AI services, mock the model call entirely and test the orchestration around it: timeout handling, error propagation, SSE streaming, caching decisions. What's under test is your handling of the response. The model itself is out of scope.

One structural choice makes all of this tractable. Every model call goes through a single callModel() function with a taskName that maps to a versioned prompt. Change a prompt, bump the version. Evaluation traces stay coherent, and you can compare behaviour across versions instead of mixing them into one undifferentiated blob. It's the same discipline our AI engineering team applies to model calls in client systems. If you're building LLM features and your prompts are scattered across the codebase, that's the refactor to do before you write a single test.

LLM Monitoring in Production Is Part of QA

Tests cover the known unknowns. Monitoring covers the rest, and on a product running against content you don't control, the rest is most of your risk.

If your product runs against pages you don't own, sample real output on a schedule and alert on it, because that is the only way you hear about a broken site before your users do. Our extension runs exactly this: a pipeline sampling extraction output across real user URLs.

When a site starts failing because it changed its HTML, added an iframe layer, or put up a login gate, the pipeline catches it before a user reports it.

The backend traces every model call with its task name, an input hash, the output, and latency. When a prompt change degrades quality in production, it shows up in the traces even though every unit test still passes. That's the normal failure mode on an AI product, and it's why traces, metrics, and logs belong in the QA conversation rather than only the ops one. The line between "test" and "monitor" is really just the line between failures you predicted and failures you didn't.

What Holds Up Across AI Products

Test at the right layer. Route tests catch authorization bugs. Service tests catch logic bugs. E2E catches integration bugs. Over-investing in unit tests is tempting because they're fast, but the bugs that reach real users live at the integration layer.

Make E2E auth real. The overhead of minting a genuine session in setup pays for itself the first time a cookie policy changes.

Automate second. Manual exploration is how you learn what's worth automating, so resist automating first, which only encodes your assumptions faster. That is not theory: on this product the E2E count went from around 30 to 200 precisely because manual testing came first on every feature.

Let the suite track the product. New feature, manual pass, then promote the specs most prone to regression or flakiness into E2E, so the suite follows the product.

None of this is finished. No production system's QA ever is. But testing at every layer, from unit through integration and E2E out into production monitoring, is what lets a small team put AI features in front of real users and sleep at night.

We've built and run this kind of test infrastructure for engineering teams shipping AI to production. Follow along on LinkedIn for more from the team.

If your test suite still assumes deterministic output, that gap tends to surface in production rather than in CI.

Talk to our QA and AI engineering team.

FAQ

How do you test LLM output when it is non-deterministic?

Assert on the contract rather than the content. Check that output exists and clears a length floor, that it semantically overlaps the source, that formatting survived, and that no errors hit the console. Never assert on exact model wording.

What is a good LLM testing framework?

There is no single framework. In practice, it's a stack: Jest for unit and service tests, a real-database integration layer for the API, and Playwright for end-to-end tests that load the actual client. The framework matters less than deciding what belongs at each layer.

Should you mock the LLM in tests?

For backend tests, yes. Mock the model call and test the orchestration around it: timeouts, error propagation, streaming, caching. Leave real model calls to E2E and production monitoring.

How do you test a Chrome extension end-to-end?

Playwright can launch Chromium with an unpacked extension loaded via a persistent context. The hard parts are deriving the extension ID at runtime and authenticating through the real session path rather than a mock.

Is monitoring part of QA for AI products?

Yes. Tests only cover what you anticipated. For a product running against third-party content and non-deterministic models, production traces and sampled output monitoring catch the failures your suite was never written to see.