AI evaluation is at a turning point. For decades, static tests with known answers have been the main thermometer of progress: labeled datasets, reasoning exams, computer vision tasks. The problem appears when a system outperforms the best humans on one of these tests: from that moment on, the test stops providing useful information. Each one-point improvement may not reflect a new capability, but rather statistical tuning. To keep moving forward, we need a way to measure intelligence that does not depend on the human ceiling.
The challenge is not only statistical, but epistemological. To grade an exam, one must know the correct answer. If examiners cannot solve a difficult task, how can they claim that a machine-generated answer is correct? In closed domains, such as mathematics or programming, formal verification offers a way out. In open domains, such as business strategy or organizational diagnosis, there is no single solution. This asymmetry explains why traditional benchmarks saturate and why building an absolute intelligence scale is so difficult.
An alternative is to change the approach: measure in relative terms, letting systems themselves generate the challenges. The idea is simple: one model creates a task it considers representative of a specific capability; another model must solve it. If the challenger fails, the generator scores points; if it succeeds, the challenger scores. Repeating the process at scale yields a dynamic ranking, a kind of adversarial psychometric scoring. What matters is that no human needs to know the solution: it is enough that the generator produces valid tasks and that a verifier can check the answers.
This approach has a particularly valuable property: it scales with the systems it aims to measure. When current models can generate problems of increasing difficulty, future models will have to solve even harder problems. Evaluation is no longer frozen at the human limit. That is why it makes sense to speak of intelligence beyond the human scale: not because some magic formula exists, but because the difficulty of tests continuously adapts to the capacity of those being evaluated.
For such a system to be reliable, privileged-information attacks must be avoided. If the generator model knows the solution, it could introduce subtle clues that only another similarly trained model can detect, distorting results. One practical solution is a commit-and-reveal protocol: the generator publishes a digital digest of the challenge, sends it to the evaluators, and only later reveals the full content, verifying that it matches the announcement. Another is to separate generation and verification in isolated environments, so that the generator cannot alter the answers of others. These protocols reduce incentives to manipulate evaluation and make it possible to issue verdicts without a human judge in domains where formal verification exists.
In open, non-verifiable domains, adjudication is more delicate. One path is to use consistency among agents: if several independent systems agree that an answer is coherent and find no contradictions, the answer is accepted. Another path is to use meta-evaluators that only assess the quality of the question, not the solution. In any case, the key is that the verdict does not depend on a single person or a single model, but on a process designed to minimize biases and maximize reproducibility.
This shift has immediate business implications. Organizations need to know which AI model is right for each task and how to detect when a system stops improving. Agent-generated tests allow continuous competency assessments, far more realistic than multiple-choice tests. They can measure whether a virtual assistant solves never-before-seen incidents, whether a cybersecurity system identifies a new vulnerability, or whether a language model produces useful reports for decision making.
Imagine a language model evaluation platform. A generator agent creates a simulated medical record and asks for the most likely diagnosis. Another agent must respond in natural language. An automatic verifier checks whether the answer mentions key data and does not invent information. Each interaction updates both ratings. After thousands of rounds, the ranking stops reflecting memory and begins to reflect applied reasoning. The same pattern can be applied to recruitment, process auditing, or continuous improvement of an ERP.
The role of humans does not disappear; it is elevated. Instead of grading thousands of exercises, experts define quality criteria, review edge cases, and decide which skills are strategic. The machine multiplies scenarios and systematically searches for weak points in other systems. It is a form of collaborative evaluation among artificial intelligences, supervised by people with judgment and guided by real business objectives.
At Q2BSTUDIO, a software and technology development company, we apply this philosophy to concrete projects. When we develop custom software, we integrate evaluation systems that learn with each use, rather than relying on fixed exams. Our team designs architectures where AI agents generate test cases, environments are deployed on cloud AWS/Azure, and results are visualized through dashboards in BI/Power BI. We also incorporate Artificial Intelligence solutions in every layer of the process, from challenge generation to result interpretation. Cybersecurity is essential: protecting evaluation data is as important as metric accuracy.
Implementing this model in a company requires several steps. First, identify the critical competencies to measure. Second, choose the right generators and verifiers, whether proprietary models or external services. Third, deploy the infrastructure in the cloud with strict privacy policies. Fourth, define the scoring algorithm and update rules. Finally, visualize evolution with indicators that support decision making. It is not a trivial process, but the technology to do it exists today, and companies that adopt it earlier will have a significant competitive advantage.
Measuring intelligence beyond the human scale is not a utopia. It is a paradigm shift that can be implemented with adversarial test generation, formal verification, elastic cloud computing, and data analytics. The challenge lies in designing fair protocols and interpreting results critically. Those who achieve it will be able to say precisely which systems are truly capable, even when no human knows the answer.
