Benchmarking LLMs in Russian: What, How, and Why Should We Evaluate?

Abstract

How do you choose the right language model for your task? What benchmarks are available for evaluating LLMs in Russian, and how do you determine which ones best suit your needs? Why can a model that tops one leaderboard perform poorly on another?

In this workshop, we will explore modern approaches to evaluating LLMs and AI agents.

We will review existing Russian-language benchmarks, discuss how they differ, and explain how to select appropriate evaluation tasks and metrics and interpret the results. We will focus on MERA Text 2.0, a new generation of Russian-language benchmarks covering complex reasoning, agentic capabilities, cultural and linguistic nuances, and human-centric evaluation. Using WebPageBench as an example, we will also demonstrate how to evaluate AI agents in interactive web environments and verify whether they successfully complete real-world tasks.

Finally, we will discuss how to ensure reproducible evaluations, avoid common pitfalls when comparing models, and use benchmarking results to select and improve LLM-based systems.