Influence of deliverable evaluation feedback and additional reward on worker's motivation in crowdsourcing services
IntroductionTo create training data for AI systems, it is necessary to manually assign correct labels to a large number of objects; this task is often performed by crowdsourcing. This task is usually divided into a certain number of smaller and more manageable segments, and workers work on them one...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Frontiers Media S.A.
2025-04-01
|
| Series: | Frontiers in Organizational Psychology |
| Subjects: | |
| Online Access: | https://www.frontiersin.org/articles/10.3389/forgp.2025.1500016/full |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849731266373484544 |
|---|---|
| author | Yoshinori Hijikata Yoshinori Hijikata Hikari Ishizaki |
| author_facet | Yoshinori Hijikata Yoshinori Hijikata Hikari Ishizaki |
| author_sort | Yoshinori Hijikata |
| collection | DOAJ |
| description | IntroductionTo create training data for AI systems, it is necessary to manually assign correct labels to a large number of objects; this task is often performed by crowdsourcing. This task is usually divided into a certain number of smaller and more manageable segments, and workers work on them one after the other. In this study, assuming the above task, we investigated whether the deliverable evaluation feedback and provision of additional rewards contribute to the improvement of workers' motivation, that is, the persistence of the tasks and performance.MethodWe conducted a user experiment on a real crowdsourcing service platform. This provided first and second round of tasks, which ask workers input correct labels to a flower species. We developed an experimental system that assessed the work products of the first-round task performed by a worker and presented the results to the worker. Six hundred forty-five workers participated in this experiment. They were divided into high and low performing groups according to their first-round scores (correct answer ratio). The workers' performance and task continuation ratio under the high and low performance group and with and without evaluation feedback and additional rewards were compared.ResultsWe found that the presentation of deliverable evaluations increased the task continuation rate of high-quality workers, but did not contribute to an increase in the task performance (correct answer rate) for either type of worker. The providing additional rewards reduced workers' task continuation rate, and the amount of reduction was larger for low-quality workers than that for high-quality workers. However, it largely increased the low-quality worker's task performance. Although not statistically significant, the low-quality worker's task performance of the second round was highest for those who were shown both feedback and additional rewards.DiscussionIt was found that rewards positively affected worker motivation in previous studies. This is inconsistent with the results of our study. One possible reason is that previous studies have examined workers' future engagements on different tasks, whereas our study examined workers' successive tackles on the almost same task. In conclusion, it is better to offer both feedback and additional rewards when the quality of the deliverables is a priority, and to give only feedback when the quantity of deliverables is a priority. |
| format | Article |
| id | doaj-art-cf4180ec6d6c406cb30144675c8df9ec |
| institution | DOAJ |
| issn | 2813-771X |
| language | English |
| publishDate | 2025-04-01 |
| publisher | Frontiers Media S.A. |
| record_format | Article |
| series | Frontiers in Organizational Psychology |
| spelling | doaj-art-cf4180ec6d6c406cb30144675c8df9ec2025-08-20T03:08:36ZengFrontiers Media S.A.Frontiers in Organizational Psychology2813-771X2025-04-01310.3389/forgp.2025.15000161500016Influence of deliverable evaluation feedback and additional reward on worker's motivation in crowdsourcing servicesYoshinori Hijikata0Yoshinori Hijikata1Hikari Ishizaki2Graduate School of Information Science, University of Hyogo, Kobe, Hyogo, JapanSchool of Business Administration, Kwansei Gakuin University, Nishinomiya, Hyogo, JapanSchool of Business Administration, Kwansei Gakuin University, Nishinomiya, Hyogo, JapanIntroductionTo create training data for AI systems, it is necessary to manually assign correct labels to a large number of objects; this task is often performed by crowdsourcing. This task is usually divided into a certain number of smaller and more manageable segments, and workers work on them one after the other. In this study, assuming the above task, we investigated whether the deliverable evaluation feedback and provision of additional rewards contribute to the improvement of workers' motivation, that is, the persistence of the tasks and performance.MethodWe conducted a user experiment on a real crowdsourcing service platform. This provided first and second round of tasks, which ask workers input correct labels to a flower species. We developed an experimental system that assessed the work products of the first-round task performed by a worker and presented the results to the worker. Six hundred forty-five workers participated in this experiment. They were divided into high and low performing groups according to their first-round scores (correct answer ratio). The workers' performance and task continuation ratio under the high and low performance group and with and without evaluation feedback and additional rewards were compared.ResultsWe found that the presentation of deliverable evaluations increased the task continuation rate of high-quality workers, but did not contribute to an increase in the task performance (correct answer rate) for either type of worker. The providing additional rewards reduced workers' task continuation rate, and the amount of reduction was larger for low-quality workers than that for high-quality workers. However, it largely increased the low-quality worker's task performance. Although not statistically significant, the low-quality worker's task performance of the second round was highest for those who were shown both feedback and additional rewards.DiscussionIt was found that rewards positively affected worker motivation in previous studies. This is inconsistent with the results of our study. One possible reason is that previous studies have examined workers' future engagements on different tasks, whereas our study examined workers' successive tackles on the almost same task. In conclusion, it is better to offer both feedback and additional rewards when the quality of the deliverables is a priority, and to give only feedback when the quantity of deliverables is a priority.https://www.frontiersin.org/articles/10.3389/forgp.2025.1500016/fullcrowdsourcingdeliverable assessmentevaluation feedbackadditional rewardworker motivationtask continuation |
| spellingShingle | Yoshinori Hijikata Yoshinori Hijikata Hikari Ishizaki Influence of deliverable evaluation feedback and additional reward on worker's motivation in crowdsourcing services Frontiers in Organizational Psychology crowdsourcing deliverable assessment evaluation feedback additional reward worker motivation task continuation |
| title | Influence of deliverable evaluation feedback and additional reward on worker's motivation in crowdsourcing services |
| title_full | Influence of deliverable evaluation feedback and additional reward on worker's motivation in crowdsourcing services |
| title_fullStr | Influence of deliverable evaluation feedback and additional reward on worker's motivation in crowdsourcing services |
| title_full_unstemmed | Influence of deliverable evaluation feedback and additional reward on worker's motivation in crowdsourcing services |
| title_short | Influence of deliverable evaluation feedback and additional reward on worker's motivation in crowdsourcing services |
| title_sort | influence of deliverable evaluation feedback and additional reward on worker s motivation in crowdsourcing services |
| topic | crowdsourcing deliverable assessment evaluation feedback additional reward worker motivation task continuation |
| url | https://www.frontiersin.org/articles/10.3389/forgp.2025.1500016/full |
| work_keys_str_mv | AT yoshinorihijikata influenceofdeliverableevaluationfeedbackandadditionalrewardonworkersmotivationincrowdsourcingservices AT yoshinorihijikata influenceofdeliverableevaluationfeedbackandadditionalrewardonworkersmotivationincrowdsourcingservices AT hikariishizaki influenceofdeliverableevaluationfeedbackandadditionalrewardonworkersmotivationincrowdsourcingservices |