학술논문
OLMo: Accelerating the Science of Language Models
Document Type
Working Paper
Author
Groeneveld, Dirk; Beltagy, Iz; Walsh, Pete; Bhagia, Akshita; Kinney, Rodney; Tafjord, Oyvind; Jha, Ananya Harsh; Ivison, Hamish; Magnusson, Ian; Wang, Yizhong; Arora, Shane; Atkinson, David; Authur, Russell; Chandu, Khyathi Raghavi; Cohan, Arman; Dumas, Jennifer; Elazar, Yanai; Gu, Yuling; Hessel, Jack; Khot, Tushar; Merrill, William; Morrison, Jacob; Muennighoff, Niklas; Naik, Aakanksha; Nam, Crystal; Peters, Matthew E.; Pyatkin, Valentina; Ravichander, Abhilasha; Schwenk, Dustin; Shah, Saurabh; Smith, Will; Strubell, Emma; Subramani, Nishant; Wortsman, Mitchell; Dasigi, Pradeep; Lambert, Nathan; Richardson, Kyle; Zettlemoyer, Luke; Dodge, Jesse; Lo, Kyle; Soldaini, Luca; Smith, Noah A.; Hajishirzi, Hannaneh
Source
Subject
Language
Abstract
Language models (LMs) have become ubiquitous in both NLP research and in commercial product offerings. As their commercial importance has surged, the most powerful models have become closed off, gated behind proprietary interfaces, with important details of their training data, architectures, and development undisclosed. Given the importance of these details in scientifically studying these models, including their biases and potential risks, we believe it is essential for the research community to have access to powerful, truly open LMs. To this end, this technical report details the first release of OLMo, a state-of-the-art, truly Open Language Model and its framework to build and study the science of language modeling. Unlike most prior efforts that have only released model weights and inference code, we release OLMo and the whole framework, including training data and training and evaluation code. We hope this release will empower and strengthen the open research community and inspire a new wave of innovation.