학술논문
BigBIO: A Framework for Data-Centric Biomedical Natural Language Processing
Document Type
Working Paper
Author
Fries, Jason Alan; Weber, Leon; Seelam, Natasha; Altay, Gabriel; Datta, Debajyoti; Garda, Samuele; Kang, Myungsun; Su, Ruisi; Kusa, Wojciech; Cahyawijaya, Samuel; Barth, Fabio; Ott, Simon; Samwald, Matthias; Bach, Stephen; Biderman, Stella; Sänger, Mario; Wang, Bo; Callahan, Alison; Periñán, Daniel León; Gigant, Théo; Haller, Patrick; Chim, Jenny; Posada, Jose David; Giorgi, John Michael; Sivaraman, Karthik Rangasai; Pàmies, Marc; Nezhurina, Marianna; Martin, Robert; Cullan, Michael; Freidank, Moritz; Dahlberg, Nathan; Mishra, Shubhanshu; Bose, Shamik; Broad, Nicholas Michio; Labrak, Yanis; Deshmukh, Shlok S; Kiblawi, Sid; Singh, Ayush; Vu, Minh Chien; Neeraj, Trishala; Golde, Jonas; del Moral, Albert Villanova; Beilharz, Benjamin
Source
Subject
Language
Abstract
Training and evaluating language models increasingly requires the construction of meta-datasets --diverse collections of curated data with clear provenance. Natural language prompting has recently lead to improved zero-shot generalization by transforming existing, supervised datasets into a diversity of novel pretraining tasks, highlighting the benefits of meta-dataset curation. While successful in general-domain text, translating these data-centric approaches to biomedical language modeling remains challenging, as labeled biomedical datasets are significantly underrepresented in popular data hubs. To address this challenge, we introduce BigBIO a community library of 126+ biomedical NLP datasets, currently covering 12 task categories and 10+ languages. BigBIO facilitates reproducible meta-dataset curation via programmatic access to datasets and their metadata, and is compatible with current platforms for prompt engineering and end-to-end few/zero shot language model evaluation. We discuss our process for task schema harmonization, data auditing, contribution guidelines, and outline two illustrative use cases: zero-shot evaluation of biomedical prompts and large-scale, multi-task learning. BigBIO is an ongoing community effort and is available at https://github.com/bigscience-workshop/biomedical
Comment: Submitted to NeurIPS 2022 Datasets and Benchmarks Track
Comment: Submitted to NeurIPS 2022 Datasets and Benchmarks Track