Michał Chromiak's blog ERNIE 2.0: Continual multi-task pre-training for language understanding
What Baidu's 2019 framework changed, how its seven objectives work, and what its results establish
ERNIE 2.0 extends masked-language-model pre-training with seven lexical, structural, and semantic objectives learned continually through a shared Transformer. This guide explains the training schedule, task embeddings, 2019 English and Chinese evaluations, and the limits of the reported comparison with BERT and XLNet.
more ...