Семинар НУЛ ММВП "Non-Autoregressive Island in Autoregressive World (Неавторегрессионные языковые модели)"
12 марта 2020, в четверг состоится заседания семинара Научно-учебной лаборатории моделей и методов вычислительной прагматики Департамента анализа данных и искусственного интеллекта.
Тема: "Non-Autoregressive Island in Autoregressive World" (Неавторегрессионные языковые модели)
Докладчик: Михаил Архипов, МФТИ, Лаборатория Нейронных Систем и Глубокого Обучения, DeepPavlov // Mikhail Arkhipov, MIPT, DeepPavlov.
The vast majority of current state-of-the-art models rely on autoregressive inference for modeling sequences. While showing top quality metrics this approach has several intrinsic drawbacks such as sequential inference and exposure bias. Despite the struggles* of the research community current parallel approaches show lower quality being in particular cases an order of magnitude faster. In this talk, we will review approaches to parallel inference and discuss recent papers devoted to the subject.
*an approximate list of struggles:
- Non-Autoregressive Neural Machine Translation
- Noisy parallel approximate decoding for conditional recurrent language model
- Fast Decoding in Sequence Models Using Discrete Latent Variables
- On the Discrepancy between Density Estimation and Sequence Generation
- Mask-Predict: Parallel Decoding of Conditional Masked Language Models