AI Testing Is a Different Beast

Ask any two engineers to run the same prompt against the same LLM with the same inputs, twice, and you’ll get two different answers.

Not wildly different — but different enough that “compare output to expected output,” the backbone of QA as we’ve known it for decades, starts to break down.

We ran into this firsthand while building a six-agent AI pipeline for an EdTech assessment platform. One agent reads a teacher’s raw observation, and five downstream agents turn it into developmental insight, pattern tracking, and an intervention plan.

Run the same observation through twice, and the first agent might phrase its interpretation slightly differently each time. Not necessarily wrong — just different. That variation can then propagate through the downstream agents.

In a traditional pipeline, that would look like a bug.

Here, the more useful question became:

Does the output stay within the assessment framework, schema and intended meaning?

That shift in thinking changed how we approached testing.

At BUSoft, we’re working across generative AI, individual agents, agentic workflows, traditional ML and Vision AI. What we’re finding is that each introduces a slightly different testing challenge. But there are some common principles.

The first is having a golden dataset — a curated, versioned set of representative inputs with known-good characteristics. Without one, every evaluation run becomes a new snapshot that is difficult to compare. With one, we can see whether a prompt change, model upgrade or RAG index rebuild actually improved the system.

We saw exactly why this matters when comparing a commercial model against a self-hosted open-source one for cost reasons. Feed the same sprint snapshot through each, and one might recommend pulling a ticket out of the sprint while the other suggests adding a buffer day before the demo — both reasonable reads of the same signal, neither one “the” answer. Changing the model didn’t just shift latency and cost; it shifted the whole distribution of recommendations. An exact expected-output test would need re-baselining every time we swapped models — and once cost pressure makes model-swapping routine, re-baselining stops being an edge case and becomes part of the job.

The second is evaluation rather than exact matching.

We increasingly ask questions such as: is the response semantically relevant, or does it wander off-topic? Is it factually grounded, or does it hallucinate — inventing facts from the model’s own memory rather than any real source? Is it complete, covering every part of a multi-part prompt? Is it coherent — well-formed, consistent in tone and structure? And is it appropriate for the intended user?

LLM-as-judge can help us do this at scale, but it isn’t a magic solution. We still need to periodically compare automated evaluation against human review and refine the rubric when they disagree.

Underneath all of this sit more familiar infrastructure metrics, though they carry new weight — latency and cost now scale with how much a model reasons, not just how much load it’s under. Time to first token, total generation time, throughput, cost per request, and increasingly, how the system behaves under a provider’s own rate limits, not just our own.

The testing also has to move beyond the final answer.

For RAG systems, are we retrieving the right context in the first place? For MCP-based tool integrations, are the schemas well-formed, is the tool lifecycle managed correctly, and is access properly scoped? For database agents, is the generated SQL not only valid, but actually answering the user’s intent? For agentic workflows, did the system choose the right tool or agent, and did the user’s intent survive the handoffs? For Vision AI, are the annotations reliable and does the test data represent the real-world conditions the model will encounter?

Then there is security — prompt injection, unauthorized data access, unsafe tool calls, PII exposure, and adversarial or jailbreak attempts. Increasingly, that also means testing for over-refusal: a model that declines things it shouldn’t is its own failure mode, not a safe default.

Closely related is bias and fairness — do outputs stay consistent and equitable across different framings of demographic or cultural inputs, and for Vision AI, do annotation quality and model accuracy hold up evenly across sub-populations in the data rather than just on the majority class.

Neither is a one-time check; both need retesting whenever the data or model changes.

And perhaps the most overlooked piece is observability.

When an AI system produces an unexpected answer, can we trace it back to the exact prompt, context, retrieved information, model, tool calls and inputs that produced it? If we can’t, debugging becomes guesswork.

And testing doesn’t stop when the system goes into production. Data changes. User behavior changes. Models change. A model provider can even update the underlying model without any change to our application code. So data drift and model drift become part of the quality conversation as well.

I’m increasingly convinced that AI testing isn’t simply traditional QA with a few additional test cases.

The definition of “correctness” itself changes.

Traditional software often asks:

“Did we get the expected result?”

AI systems increasingly require us to ask:

“Did we get an acceptable result, within the boundaries we intended?”

Not every AI system needs every type of evaluation. The challenge is understanding which risks matter for the system you’ve built, and how to test them effectively.

That risk-based approach is becoming increasingly important as we move from experimenting with AI to governing and operating it in production.

But there is another part of the challenge that is just as important: having people who can evaluate these systems with the right technical context and judgment.

At BUSoft, this has raised a practical question:

Do our QA and engineering teams have the skills needed to work this way?

In my next post, I’ll explore how the QA and developer roles are evolving, and the skills teams need as AI becomes part of the development and quality lifecycle.