Семинар НУЛ ММВП "RussianSuperGLUE"

Аннотация: In this talk, we introduce an advanced Russian general language understanding evaluation benchmark -- RussianGLUE.
Recent advances in the field of universal language models and transformers require the development of a methodology for their broad diagnostics and testing for general intellectual skills - detection of natural language inference, commonsense reasoning, ability to perform simple logical operations regardless of text subject or lexicon. For the first time, a benchmark of nine tasks, collected and organized analogically to the SuperGLUE methodology, was developed from scratch for the Russian language. We provide baselines, human level evaluation, an open-source framework for evaluating models.