The promise of AI agent systems is that they do not just execute fixed tasks, but learn, accumulate experience and improve over time. To achieve this, many designs use a skill library: the agent composes its responses from pieces of knowledge, execution routines and heuristics that have worked in the past. This architecture is especially attractive in business environments, where complex processes need to be automated and adapted to changing data. However, a library that grows without control also accumulates noise. Skills that are no longer useful, that are contradictory, or that only worked in a specific situation end up degrading global performance. That is why self-evolving systems need a removal mechanism, a curator capable of eliminating what no longer adds value. At Q2BSTUDIO we often compare it to good code governance: it is not enough to add functions; you also have to dare to delete. But there is a crucial nuance: if the evaluator that decides what gets retired is biased, the curator becomes a blind curator.
That automatic curator works thanks to an evaluation signal. When a skill produces a correct result, it is reinforced; when it fails, it is weakened until it disappears. The minimum condition for the system to be viable is that the skill library never makes the agent perform worse than an agent with no skills at all. This baseline is non-negotiable. In tasks with an objective answer, a code executor, a unit test or a formal verifier can detect failure with precision. But in tasks without a ground truth, such as writing a report, summarizing a conversation or evaluating the quality of a text, there is no external correct answer. Evaluation then falls into the hands of a language judge, usually a large language model trained to score or compare outputs. And that is where the chain of trust starts to break.
A language judge is not an oracle. It can have systematic biases: preferring long texts, penalizing unusual structures, or being too lenient with factual errors. The study behind this reflection distinguishes two types of noise. One is symmetric noise: the judge sometimes gets it right and sometimes gets it wrong, but does not systematically favor bad outputs. That noise, though uncomfortable, does not destroy the retirement mechanism. The other is false-pass bias: a real failure that the judge marks as a success. This second case is much more dangerous, because the curator does not see the failure and therefore does not retire the defective skill. The system can become trapped in a zone where all bad skills receive a green light and the library fills with useless content without anyone intervening.
The most relevant point is that there is a sharp threshold. As long as the percentage of failures that receive approval stays low, the curator continues to do its job. But beyond a certain point, the mechanism stops working completely. It does not worsen gradually; it switches off. What makes this result counterintuitive is that it is not a data quantity problem. More training data, more examples or more evaluations cannot cross the threshold if the judge keeps approving failures. Bias is not averaged away; it becomes a structural feature of the system. For a company trying to scale AI solutions, this idea is fundamental: an aggregate quality metric can remain stable while the internal safety mechanism has already been disabled.
The research also shows that the mechanism problem is universal. It appears across domains, with different failure rates and with different types of agents. Only evaluators with an almost zero false-pass rate avoid it, something that in practice is only achieved with external verifiers or very strict checking systems. In other words, the failure does not depend on the task, but on the relationship between the evaluator and the reality it evaluates. A judge that cannot distinguish failure from success is not just a noisy evaluator: it is a switch that turns off supervision. The immediate consequence is that the curator becomes blind. It is still there, still processing skills, but it can no longer tell good skills from bad ones.
Imagine a factory where a quality supervisor never rejects a defective part because its measurement system is poorly calibrated. The production line does not stop; production reports continue to show an acceptable pace; but defective parts pile up in inventory. At some point the warehouse is full of unusable product and nobody remembers when the problem started. Something similar happens with skill libraries. If the judge approves failures, bad skills are not retired, they accumulate, and the system loses its structural safeguard. The most dangerous part is that external indicators do not always reflect the damage: overall quality can remain steady because the synthesis of new skills is still working, masking the fact that pruning has stopped.
In the business world, this distinction between mechanism and outcome is key. Organizations usually measure an agent by the quality of its deliveries, not by the internal health of its skill library. If a system begins to produce reports that seem correct but contain undetected errors, the first warning sign will not appear in average accuracy, but in the silent disappearance of the ability to correct itself. That is why, when we design AI solutions for clients at Q2BSTUDIO, we insist that model governance is part of the architecture from day one, not a layer added at the end. An autonomous agent without a reliable curator is an operational risk, no matter how good its metrics look.
Our experience as a custom software development company has shown us that evaluating language models should be treated with the same rigor as a cybersecurity audit. It is not enough to trust the model's judgment; you have to audit the judge. The proposal to inject known defects into the evaluation flow is a practical way of doing this: take skills that we know fail, introduce them into the system and check whether the judge detects them. If the judge approves them, we know it is on the wrong side of the threshold and that skill retirement is disabled. This test is cheap, can be run before deployment, and provides very valuable information about the system's limits. It is a practice directly connected to cybersecurity: attacking the system with defective cases to discover blind spots before a real problem does.
The implications for custom software development are profound. When a company orders an AI agent platform, it is not buying a model; it is buying a behavioral system. That system needs retirement policies, observability, and correction mechanisms. At Q2BSTUDIO we work with cloud AWS/Azure architectures, data pipelines and BI/Power BI dashboards, because all of these elements are part of the infrastructure that makes it possible to monitor an agent's behavior. It is not about adding more intelligence, but about knowing when a capability should be retired. A dashboard that shows the judge's approval rate and the skills retirement rate can alert in time that the curator has been disconnected, even when quality metrics have not moved yet.
The practical conclusion is that operators of self-evolving systems cannot wait for performance to worsen to detect a biased judge. As soon as evaluation without ground truth depends on a language model, the right question is not 'how often is it right?' but 'what happens when it is wrong?'. If errors are systematically approved, the system will have no way to retire its bad skills. And even if outcome metrics remain stable, the self-assessment capability will be broken. Defect-injection auditing offers a fast way to know the terrain before stepping on it. At a time when AI agents are beginning to take on critical tasks, this distinction between noise and bias, and between mechanism and outcome, should be part of the common language of any engineering team.
Ultimately, the blind curator is a reminder that autonomy does not eliminate the need for supervision; it multiplies it. An agent capable of learning must also be capable of forgetting. And for forgetting to be useful, something has to see the failure. If that something is a biased judge that approves everything, the system loses its ability to renew itself without making noise. The good news is that there are practical methods to detect the problem before it causes harm. The bad news is that most companies still do not apply them. Building responsible AI agents requires looking where nobody looks: not only at the quality of responses, but at the health of the mechanism that decides which skills deserve to remain alive. At Q2BSTUDIO we help organizations take that look with technical rigor, combining custom software, artificial intelligence, cybersecurity and data so that autonomous evolution does not become a silent accident.
