Improving Multiclass Classification in Crowdsourcing by Using Hierarchical Schemes
by Xiaoni Duan, Keishi Tajima
Abstract
In this paper, we propose a method of improving accuracy of multiclass
classification tasks in crowdsourcing. In crowdsourcing, it is
important to assign appropriate workers to appropriate tasks. In
multiclass classification, different workers are good at different
subcategories. In our method, we reorganize a given flat
classification task into a hierarchical classification task consisting
of several subtasks, and assign each worker to an appropriate subtask.
In this approach, it is important to choose a good hierarchy. In our
method, we first post a flat classification task with a part of data
and collect statistics on each worker's ability. Based on the
obtained statistics, we simulate all candidate hierarchical schemes,
estimate their expected accuracy, choose the best scheme, and post it
with the rest of data. In our method, it is also important to
allocate workers to appropriate subtasks. We designed several greedy
worker allocation algorithms. The results of our experiments show
that our method improves the accuracy of multiclass classification
tasks.
/ Fax: +81(Japan) 75-753-5978 / Office: Research Bldg. #7, room 404