학술논문
A Labeling Task Design for Supporting Recent Algorithmic Needs
Document Type
Conference
Author
Source
2022 IEEE International Conference on Big Data (Big Data) Big Data (Big Data), 2022 IEEE International Conference on. :2689-2698 Dec, 2022
Subject
Language
Abstract
Studies on supervised machine learning (ML) recommend involving workers from various backgrounds in training dataset labeling to reduce algorithmic bias. Moreover, sophisticated tasks for categorizing objects in images are necessary to improve ML performance, further complicating micro-tasks. This study aims to develop a task design incorporating the fair participation of people, regardless of their specific backgrounds or task’s difficulty. By collaborating with 75 labelers from diverse backgrounds for 3 months, we analyzed workers’ log-data and relevant narratives to identify the task’s hurdles and helpers. The findings revealed that workers’ decision-making tendencies were affected by the "community" that positively helps workers. Also, the machine’s feedback perceived by workers could make people easily engaged in works. Based on these findings, we suggest an extended human-in-the-loop approach that connects labelers, machines, and communities rather than isolating individual workers.