- Title
- A robust consistency model of crowd workers in text labeling tasks
- Creator
- Alqershi, Fattoh; Al-Qurishi, Muhammad; Aksoy, Mehmet; Alrubaian, Majed; Imran, Muhammad
- Date
- 2020
- Type
- Text; Journal article
- Identifier
- http://researchonline.federation.edu.au/vital/access/HandleResolver/1959.17/186105
- Identifier
- vital:16806
- Identifier
-
https://doi.org/10.1109/ACCESS.2020.3022773
- Identifier
- ISBN:2169-3536 (ISSN)
- Abstract
- Crowdsourcing is a popular human-based model to acquire labeled data. Despite its ability to generate huge amounts of labelled data at moderate costs, it is susceptible to low quality labels. This can happen through unintentional or intentional errors by the crowd workers. Consistency is an important attribute of reliability. It is a practical metric that evaluates a crowd workers' reliability based on their ability to conform to themselves by yielding the same output when repeatedly given a particular input. Consistency has not yet been sufficiently explored in the literature. In this work, we propose a novel consistency model based on the pairwise comparisons method. We apply this model on unpaid workers. We measure the workers' consistency on tasks of labeling political text-based claims and study the effects of different duplicate task characteristics on their consistency. Our results show that the proposed model outperforms the current state-of-the-art models in terms of accuracy. This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
- Publisher
- Institute of Electrical and Electronics Engineers Inc.
- Relation
- IEEE Access Vol. 8, no. (2020), p. 168381-168393
- Rights
- All metadata describing materials held in, or linked to, the repository is freely available under a CC0 licence
- Rights
- https://creativecommons.org/licenses/by/4.0/
- Rights
- Copyright @ IEEE
- Rights
- Open Access
- Subject
- 40 Engineering; 46 Information and computing sciences; Consistency; Crowdsourcing; Fake news; Reliability; Text labeling
- Full Text
- Reviewed
- Funder
- This work was supported by the Deanship of Scientific Research through the initiative of DSR Graduate Students Research Support (GSR), King Saud University.
- Hits: 540
- Visitors: 548
- Downloads: 52
| Thumbnail | File | Description | Size | Format | |||
|---|---|---|---|---|---|---|---|
| View Details Download | SOURCE1 | Published version | 3 MB | Adobe Acrobat PDF | View Details Download |