QIMMA قِمّة: Leaderboard of quality Arabic LLMs

The launch of QIMMA represents a new stage in the evaluation of language models for the Arabic language, offering a systematic approach to validation and comparative analysis.

Recently, QIMMA قِمّة was launched, a new leaderboard for evaluating language models in Arabic, which has the potential to improve the quality of analysis through a rigorous validation process. The initiative was a response to growing concerns about existing standards and evaluation methods in the field of Arabic natural language processing (NLP).

QIMMA combines 109 subsets from 14 sources into a single evaluation platform, including over 52,000 samples, covering seven different domains—from culture and STEM to law and medicine. An important feature of QIMMA is its emphasis on 99% Arabic content and the integration of programming evaluation through adapted tasks.

With the increasing interest in Arabic LLMs, competition in this field is intensifying. Existing platforms often suffer from fragmentation and lack of data validation, making QIMMA a unique project as it systematically addresses these issues, providing a more reliable and reproducible evaluation.

The initial results from QIMMA revealed serious shortcomings in the evaluation samples, including factual errors and low-quality texts, highlighting the need for new approaches to data validation. These findings not only contribute to improving the quality of LLMs but also create a foundation for more reliable and fair comparisons of models in the field of Arabic NLP.