In recent years, the evaluation of AI-based systems has become a mainstay of business decision-making. Product teams, researchers, and CTO rely on dashboards that show hit rates, average scores, or pass rates to decide if a new model, a tweaked prompt, or an update to an AI agent is better than the previous version. However, there is a widespread methodological error that can turn those numbers into a trap: comparing two averages as if they were independent samples when in fact they come from the same set of test items. What looks like solid improvement may be noise, and what looks like a tie may hide a quiet regress in a particular subset of data.
Let's imagine the typical scenario: we have 500 questions or test cases. We run system A and get 71.4% hits. We run system B and get 74.0%. The immediate conclusion is that B is better. However, that reasoning ignores the crucial fact that both systems answered the exact same questions. The answers are not independent; they are correlated because the difficulty of each item is shared. A complex, ambiguous, or context-rich question will often cause both systems to fail. That positive correlation reduces the variance of the difference between the two systems, and if it is not taken into account, uncertainty is overestimated or, worse, overlapping confidence intervals are misinterpreted as a sign that there is no significant difference.
This problem is not new in the field of statistics, but it is surprising to see it repeated in AI evaluation reports, even in companies with mature data teams. The origin is in treating two executions on the same set of tests as if they were two experiments with different samples. The solution is as elegant as it is simple: match the results by item and directly analyze the differences. For binary evaluations (pass/fail), the appropriate test is the McNemar test, which concentrates only on items where the systems disagree. If system A is right and B is wrong in 40 cases, and B is right and A is wrong in 53, the statistical question is whether that 40-53 split is compatible with a balanced currency. An exact binomial test will reveal whether the observed difference is credible or simply the result of chance. On many occasions, this gap of 2.6 percentage points turns out not to be significant.
For assessments with continuous scores (a score from 0 to 1, a Likert scale rating, a response time), the recommended approach is paired bootstrap. The difference of each item (B minus A) is calculated, and then those differences are resampled with replacement to obtain confidence intervals over the mean of the difference. This method does not assume normality and respects the dependency structure between the two measurements. The important thing is to resample the pairs, not the columns separately, because otherwise the pairing is broken and the independence error is returned.
Beyond the technical procedure, there is a cultural implication in product teams: the temptation to make decisions by looking at two numbers side by side is enormous. Dashboards invite you to do so. But when you're evaluating a change that can affect the experience of thousands of users, or when deciding whether to deploy a new AI agent into production, statistical accuracy isn't an academic luxury. It is a necessity to avoid false positives that lead to investment in changes that do not add value, or false negatives that prevent real improvements from being detected.
A specific case that illustrates this risk was experienced when reviewing a regression in a custom software project for a client in the logistics sector. The prompt of a conversational assistant to answer queries about shipments had been modified. The new prompt showed a 68.9% success rate compared to 69.7% in the previous one. Seeing that the confidence intervals overlapped, the team dismissed the difference as noise and went ahead with the change. Weeks later, users were reporting incorrect answers on a particular subset of international route queries. When reanalyzing the data with a paired test, it was found that the difference was not only significant, but that the delta confidence interval was completely below zero. The new prompt was consistently worse in that segment, but the global average hid it. If a paired analysis had been applied from the beginning, the regression would have been detected before reaching production.
The problem is compounded when multiple metrics are compared at once. Modern AI assessments for enterprise measure not only the hit rate, but also fidelity, security, latency, format compliance, and various rubric criteria. Each metric receives its own test, and if it is not corrected for multiple comparisons, the probability of finding at least one false positive grows dramatically. With twelve metrics and an alpha threshold of 0.05, the probability of having a false positive is approximately 46%. That is why it is advisable to apply procedures such as the Benjamini-Hochberg to control the rate of false discoveries, reserving stricter corrections for metrics that act as launch gates.
In Q2BSTUDIO, when we develop custom applications with artificial intelligence components, we incorporate these best practices from the evaluation phase. We know that a customer who relies on a poorly constructed dashboard can make wrong decisions that affect their operations. That's why, in our AWS and Azure cloud service projects, where we deploy AI models and agents, we ensure that evaluation pipelines include paired tests and correction by multiple comparisons. We also apply this rigor in business intelligence services projects with power BI, where the comparison of indicators before and after a change must be done with solid statistical methodologies.
Cybersecurity is not exempt from this problem either. When evaluating the effectiveness of an intrusion detection system trained with machine learning, hit rates are compared over the same set of test traffic. If you don't match the results, you run the risk of selecting a model that doesn't actually improve detection, or discarding one that does. In this area, a statistical error can result in undetected vulnerabilities.
The original article that inspired this reflection (used only as a conceptual reference) clearly states that the key is to treat data as pairs when sharing items. He advises reporting the effect size and a confidence interval about the difference, not two independent averages. And remember that even paired bootstrap has limitations when items aren't independent of each other (e.g., multiple questions pulled from the same document). In those cases, the solution is to resample at the group level or use mixed models, although the practice is not yet standardized.
In my experience working with teams that develop AI agents for customer service or process automation, the practical recommendation is clear: always store the results at the item level for each execution. It is not enough to keep the stocking. The pair (result of A, result of B) for each case is the raw material of the correct analysis. And when that's not possible, be honest about the limitation and consider any decision based on unpaired averages as uncertain.
If you want to delve into how to apply these techniques in your own evaluation systems, I recommend exploring how at Q2BSTUDIO we integrate these methodologies into artificial intelligence solutions for companies, where rigor in measurement is as important as the power of the model. We also cover bespoke application development aspects that require reliable assessments to make deployment decisions with confidence.
In short, comparing evaluations by their average success rate assuming independence is a mistake that can cost dearly. The good news is that the solution exists, is well known, and is not complex to implement. It just requires a change in mindset: stop looking at two loose numbers and start analyzing the difference within each pair. The leap in quality in decision-making deserves it.
