Why Testing AI Is Different: A QA Engineer’s First Look at Evals

Summary: Testing AI is different from testing traditional software because users can type anything, the same input can give a different output on the next run, and wrong answers fail quietly. AI teams handle this with evals: realistic inputs, written criteria for a good answer, and a way to judge each output through human review, code checks, or an LLM judge. Like a regression suite, evals run again on every prompt, model, or data change.

I've tested software products for over a decade. Testing AI broke most of my instincts.

For more than ten years, my job has come down to one question: does it work? I've answered it across products, platforms, and the layers in between. I thought I had a method that worked everywhere.

When our team started building AI systems, I assumed that method would carry over. Surprisingly, it doesn't, and figuring out why has been the most interesting part.

Why My QA Playbook Breaks When Testing AI Applications

In traditional QA, the product limits what users can do: fixed fields, dropdowns, buttons. I can partition the inputs, write cases for each partition, and trust that if input X gave output Y today, it will tomorrow. When something breaks, it breaks loudly with an error, a failed transaction, or a stack trace.

AI products remove all three comforts.

• The input is anything. Users type whatever they want, in whatever words they want. Equivalence partitioning doesn't hold up when the input space is all of human language.
• The output isn't repeatable. The same question can get a different answer on the next run. In my old world, that's a flaky test. Here it's normal behavior.
• Failures are quiet. A wrong answer arrives confident, well-written, and polite. Nothing is logged. You find out when a customer does.

The comparison that made it click for me is to think of an examiner checking answer sheets. With a multiple-choice paper, the examiner has an answer key. Each answer is either right or wrong, and they just count the correct ones. That's testing normal software.

With an essay paper, there's no answer key. Two students can write completely different answers, and both deserve full marks. The examiner needs a marking scheme and their own judgment to decide what's good. That's testing AI.

Model Evals vs Product Evals: What's the Difference?

The word "evals" gets used for two different things, which confused me at first.

Model evals are what AI labs run on their own models. They test general abilities like reasoning, coding, maths, and general knowledge against standard question sets called benchmarks, and publish the scores. That's how you see headlines like "new model tops coding leaderboard."

Product evals test whether a specific AI system does its job: your chatbot, with your data, your tools, and your users. No lab can run these for you, because nobody else has your product.

So when I first heard a model "scores 90% on benchmarks," my question was: why test again?

As a tester, I think of it like a candidate's exam results. A strong score tells you they know the subject in general. It doesn't tell you how they'll do in your team, on your projects, with your clients. You'd still interview them and watch them work. Benchmarks are the exam results: useful for shortlisting a model. Product evals are the interview and the probation period, where you find out whether it actually works for you. Product evals are what the rest of this post is about.

How to Evaluate LLM Outputs: Human Review, Code Checks and LLM-as-a-Judge

There are three ways to judge an output, and one of them needs its own testing.

1. Human review. Someone who knows the domain reads the output and decides if it's good. This is the ground truth, like SME sign-off. It's accurate, but it doesn't scale.
2. Code checks. These are assertions, the part that felt most familiar. Is the JSON valid? Are required fields present? Did it call the right tool? That last check matters most for AI agents, which act through tools. Code checks are fast and cheap, and they only work where "correct" is objective.
3. LLM-as-a-judge. You give another AI model your evaluation criteria and ask it to judge the outputs, much like an examiner working from a marking scheme. It scales to thousands of responses and can assess things code can't, like tone, helpfulness, or whether an answer is backed by the source.

Why an LLM Judge Needs Testing Too

LLM-as-a-judge is where my QA instincts came back. An LLM judge is a system under test too. It has known biases, like favoring longer answers, and it can simply be wrong. The researchers behind the MT-Bench and Chatbot Arena study documented this verbosity bias, along with a tendency to favor whichever answer is shown first. So before trusting a judge, you compare its verdicts with human verdicts on a sample. If they don't agree, the judge isn't ready.

Real user behavior adds a fourth signal: thumbs down, edited answers, people asking the same question twice. It's noisy and only arrives after launch, but it's the most honest feedback there is.

An AI Eval Example: Testing an HR Leave-Policy Assistant

To make this concrete, picture an internal HR assistant that answers leave-policy questions. Here's how I'd approach testing it.

• I'd collect around 30 realistic questions, run them through the assistant, and write down what the right answer should be.
• Then I'd read every response. Say a pattern shows up: when a question falls outside the handbook, the assistant invents a plausible policy instead of saying it doesn't know.
• That pattern becomes the eval, rather than a generic "accuracy" score. The criteria are written the way I'd explain them to a new tester: if the handbook doesn't cover it, the correct behavior is to say so and point to HR.
• An LLM judge applies those criteria to every response. I label the same set myself and check how often we agree.

Reading the outputs first felt like exploratory testing, and the failure buckets felt like defect triage. It turns out a lot of my old job carries over. Here's how my QA vocabulary maps across:

Why AI Evals Run Like a Regression Suite, Not a One-Time Sign-Off

Evals aren't something you run once before launch. Every prompt tweak, model swap, or retrieval change can fix one thing and quietly break another, so the suite runs on every change, the same way regression tests do.

Model providers also update their models behind the scenes, so behavior can shift even when you changed nothing. A Stanford and UC Berkeley study measured this directly: the March and June 2023 versions of GPT-4 behaved noticeably differently on the same tasks. Failures from real users get added back into the test set, and the cycle repeats.

If you want to see what LLM testing looks like on a shipped product, a teammate wrote about: testing an AI browser extension end to end.

What's Next for Me

My next step is to do this for real: read 100 actual conversations from a system we've built and sort what goes wrong. I'll share what I find, including what doesn't go to plan. If you're on the same path from QA to AI, or just want to swap notes, connect with me on LinkedIn.

If you lead an engineering team that's about to ship an AI feature, your testers are closer to running evals than you might think. Most of their instincts carry over once the way of judging changes. At Procedure, this is how we're approaching software testing and QA for the AI systems we build.

Where I Started Learning

Two videos helped me get started: Aish Reganti's "The Most Important Skill for AI Engineers: Evals Explained" and Aishwarya Srinivasan's "AI Evals Explained: How to Evaluate AI Agents". Both are worth watching.

For reading, Anthropic's engineering post Demystifying evals for AI agents and Hamel Husain's notes on AI evals go deeper on the ideas above.

Frequently Asked Questions

What are AI evals? AI evals are tests for AI systems. Because an AI's output can change from run to run and there's often no single correct answer, an eval pairs realistic inputs with written criteria for a good answer, then judges each output against those criteria using human review, code checks or an LLM judge.

What's the difference between model evals and product evals? Model evals are run by AI labs to measure a model's general abilities, like reasoning or coding, on public benchmarks. Product evals test whether your specific AI system does its job with your data, tools, and users. Benchmarks help you shortlist a model. Product evals tell you whether it works for you.

What is LLM-as-a-judge? LLM-as-a-judge means giving an AI model your evaluation criteria and asking it to grade outputs, much like an examiner working from a marking scheme. It scales to thousands of responses, but it has known biases, such as favoring longer answers, so its verdicts should be checked against human labels on a sample before you rely on it.

Can QA engineers run AI evals? Yes. Much of the work maps to existing QA skills: reading outputs is like exploratory testing, grouping failures is like defect triage, and running evals on every change is like maintaining a regression suite. The main shift is moving from pass/fail answer keys to written criteria and judgment.