부산대학교 도서관

Most models in natural language processing (NLP) are being pretrained on English text. When the parameters' number in the model gets more, the gap between English and other languages becomes harder to fill, and performance issues in other languages become problematic. To solve this issue, we present a new transfer learning approach that concentrates on knowledge transfer between two languages and makes a possible transfer to model size. After the transfer, we are looking to get a model of the same size as before. To make sure that model's size doesn't become a bottleneck in our approach, we train from scratch a model in our target language with a smaller size. After that, we use that model combined with the initial source model to construct token embeddings for the target model (which should be at the same size as source model) by contacting the vocabulary of both languages. The rest of the weights in the target model are same as in source model. This approach achieves the same outperforms standard language transfer method and gets 4 times faster convergence. [ABSTRACT FROM AUTHOR]