In this year’s CHI conference in Glasgow Scotland, members of the Alexandria project will be presenting their work on mitigating crowd biases in subjective tasks.
- Christoph Hube, Besnik Fetahu, and Ujwal Gadiraju: “Understanding and Mitigating Worker Biases in the Crowdsourced Collection of Subjective Judgments”
The problem is framed in the context of subjective tasks and how the inherent worker biases impact the collection of judgements for such tasks. A description of the work is shown below.
Crowdsourced data acquired from tasks that comprise a subjective component (e.g. opinion detection, sentiment analysis) is potentially affected by the inherent bias of crowd workers who contribute to the tasks. This can lead to biased and noisy ground-truth data, propagating the undesirable bias and noise when used in turn to train machine learning models or evaluate systems. In this work, we aim to understand the influence of workers’ own opinions on their performance in the subjective task of bias detection. We analyze the influence of workers’ opinions on their annotations corresponding to different topics. Our findings reveal that workers with strong opinions tend to produce biased annotations. We show that such bias can be mitigated to improve the overall quality of the data collected. Experienced crowd workers also fail to distance themselves from their own opinions to provide objective annotations.
The work in this project is funded by the