FreeEval presents a modular architecture divided into Evaluation Methods, Meta-Evaluation, and LLM Inference Backends. The Evaluation Methods include various datasets and implemented methods. Meta-Evaluation ensures integrity and fairness in evaluations, providing data contamination detection methods. The LLM Inference Backends act as the computational foundation, enabling distributed and concurrent inference with performance optimization.
FreeEval's modular architecture is designed to adapt to the constant evolution in language model evaluation. Its implementation is based on three key concepts: step, dataset, and config. A step encapsulates an evaluation method, data augmentation technique, or metric calculation logic. Each step has three phases: preprocess, which loads and prepares the data; run, which executes the main logic; and postprocess, which interprets results and releases resources.
Datasets contain the data used in evaluations, managing their preprocessing and adjustments such as few-shot settings, prompting, and instance augmentation. The config configuration allows defining evaluation pipelines through files with all necessary parameters, ensuring transparency and a complete record of the process.
This modular approach allows flexible data reuse and makes it easy for researchers to incorporate new methods without affecting the overall framework structure. By defining each evaluator as an independent unit, FreeEval promotes code reusability and maintainability. Additionally, it eliminates the need to write Python code to run evaluations, as everything is configured through configuration files.
At Q2BSTUDIO, we understand the importance of efficient and reliable evaluation of language models and other AI-based solutions. As a technology development and services company, we offer specialized consulting and develop custom solutions that meet our clients' technological needs. Our team is dedicated to innovation, adapting modular architectures like FreeEval to improve the accuracy and scalability of our projects.
