The emergence of large language models (LLMs) in the cybersecurity field has sparked a crucial debate: how to rigorously and automatically measure their actual knowledge about threats, vulnerabilities, and best practices? Until now, predominant approaches relied on manually creating sets of questions or benchmarks, a process that requires significant effort from security experts and quickly becomes outdated as new attack techniques arise. In this context, a partially automated methodology emerges that promises to transform LLM evaluation: using authoritative information from Consumer Protection Agencies (CPAs) to detect instabilities in model responses, indicative of knowledge gaps.
The proposal is based on a simple yet powerful principle: if an LLM lacks solid knowledge about a specific security topic —such as identity theft or impersonation scams— its responses will exhibit significant variations when asked equivalent questions with slight modifications. This instability, measured through consistency metrics, becomes a reliable indicator that the model lacks the necessary foundation to address the problem accurately. Instead of relying on experts to craft hundreds of test cases, the method leverages already existing public corpora, such as CPA reports and guides, to semi-automatically generate the inputs that test the model. This drastically reduces the cost and time of evaluation, while enabling a continuous update cycle as agencies publish new information.
From a technical perspective, the process involves selecting official CPA documents —for example, from the Federal Trade Commission (FTC) or the Spanish Data Protection Agency— extracting threat definitions and descriptions, and constructing variant questions that explore the same concepts from different angles. The LLM is then subjected to those questions, and the consistency of its responses is analyzed using natural language processing algorithms. If the model responds contradictorily or shows uncertainty patterns, it is identified as a knowledge gap in that specific area. In experiments with models from the Gemini and GPT families, the method was able to clearly discriminate between those possessing sufficient knowledge and those showing significant gaps regarding identity theft and impersonation scams.
This approach has direct implications for companies integrating LLMs into their cybersecurity systems. When an AI-based assistant is used to filter phishing emails or advise employees on safe practices, the reliability of its knowledge is critical. A model that does not fully understand the nuances of a social engineering attack might miss warning signs or, worse, provide incorrect recommendations. Automated evaluation using CPAs offers an objective and scalable method to certify that the model meets the necessary knowledge standards before deployment.
At Q2BSTUDIO, as a company specialized in software and technology development, we understand that the quality of knowledge embedded in AI systems is as important as the technical architecture itself. That is why we offer services ranging from creating custom software to integrating AI solutions that require knowledge evaluations like the one described. The ability to automatically audit an LLM's cybersecurity knowledge aligns with our vision of providing robust and reliable tools to our clients.
Moreover, the methodology is not limited to security: it is extensible to any domain where authoritative sources of structured knowledge exist. For example, it could be applied to verify that a technical support chatbot correctly understands configuration procedures for cloud services like AWS or Azure. At Q2BSTUDIO, we offer consulting and development in cloud AWS/Azure, and we know that accuracy of technical knowledge is essential to avoid misconfigurations that compromise security. Similarly, in the business intelligence field, AI agents trained to analyze data with Power BI require precise knowledge of key indicators to avoid generating misleading reports.
Another relevant aspect is the possibility of using this technique as part of a continuous testing pipeline within the software development lifecycle. By integrating knowledge evaluation into continuous integration practices, companies can automatically detect when an updated model has lost competencies in critical cybersecurity areas, enabling a rapid response before the change affects end users. In this sense, test automation becomes a key enabler for maintaining the quality of AI-based software.
Of course, the method is not without limitations. The quality of the evaluation directly depends on the comprehensiveness and timeliness of CPA corpora. If an agency does not cover an emerging threat, the model might pass the test without truly demonstrating its competence. However, the advantage of using official sources is that they are usually the most rigorous and industry-accepted, providing a solid reference point. Additionally, the approach can be complemented with other techniques, such as adversarial question generation or complex scenario simulation.
Looking ahead, the industry is moving toward AI systems that are not only powerful but also verifiable and transparent. Tools like automated knowledge testing based on CPAs represent an important step in that direction. At Q2BSTUDIO, we are committed to innovation in areas such as cybersecurity, artificial intelligence, and custom software development, helping organizations build solutions that are as secure as they are intelligent. The combination of automated evaluations with advanced cybersecurity and BI/Power BI services allows us to offer a complete ecosystem where knowledge reliability is a fundamental pillar.
