Use of Mechanical Turk as a MapReduce Framework for Macular OCT Segmentation

Purpose. To evaluate the feasibility of using Mechanical Turk as a massively parallel platform to perform manual segmentations of macular spectral domain optical coherence tomography (SD-OCT) images using a MapReduce framework. Methods. A macular SD-OCT volume of 61 slice images was map-distributed...

Full description

Saved in:
Bibliographic Details
Main Authors: Aaron Y. Lee, Cecilia S. Lee, Pearse A. Keane, Adnan Tufail
Format: Article
Language:English
Published: Wiley 2016-01-01
Series:Journal of Ophthalmology
Online Access:http://dx.doi.org/10.1155/2016/6571547
Tags: Add Tag
No Tags, Be the first to tag this record!
_version_ 1832549411786326016
author Aaron Y. Lee
Cecilia S. Lee
Pearse A. Keane
Adnan Tufail
author_facet Aaron Y. Lee
Cecilia S. Lee
Pearse A. Keane
Adnan Tufail
author_sort Aaron Y. Lee
collection DOAJ
description Purpose. To evaluate the feasibility of using Mechanical Turk as a massively parallel platform to perform manual segmentations of macular spectral domain optical coherence tomography (SD-OCT) images using a MapReduce framework. Methods. A macular SD-OCT volume of 61 slice images was map-distributed to Amazon Mechanical Turk. Each Human Intelligence Task was set to $0.01 and required the user to draw five lines to outline the sublayers of the retinal OCT image after being shown example images. Each image was submitted twice for segmentation, and interrater reliability was calculated. The interface was created using custom HTML5 and JavaScript code, and data analysis was performed using R. An automated pipeline was developed to handle the map and reduce steps of the framework. Results. More than 93,500 data points were collected using this framework for the 61 images submitted. Pearson’s correlation of interrater reliability was 0.995 (p<0.0001) and coefficient of determination was 0.991. The cost of segmenting the macular volume was $1.21. A total of 22 individual Mechanical Turk users provided segmentations, each completing an average of 5.5 HITs. Each HIT was completed in an average of 4.43 minutes. Conclusions. Amazon Mechanical Turk provides a cost-effective, scalable, high-availability infrastructure for manual segmentation of OCT images.
format Article
id doaj-art-ef69ad25a40644a588b0ff442eaefe23
institution Kabale University
issn 2090-004X
2090-0058
language English
publishDate 2016-01-01
publisher Wiley
record_format Article
series Journal of Ophthalmology
spelling doaj-art-ef69ad25a40644a588b0ff442eaefe232025-02-03T06:11:24ZengWileyJournal of Ophthalmology2090-004X2090-00582016-01-01201610.1155/2016/65715476571547Use of Mechanical Turk as a MapReduce Framework for Macular OCT SegmentationAaron Y. Lee0Cecilia S. Lee1Pearse A. Keane2Adnan Tufail3Department of Ophthalmology, University of Washington, Seattle, WA 98104, USADepartment of Ophthalmology, University of Washington, Seattle, WA 98104, USAMedical Retina Service, Moorfields Eye Hospital NHS Foundation Trust, London EC1V 2PD, UKMedical Retina Service, Moorfields Eye Hospital NHS Foundation Trust, London EC1V 2PD, UKPurpose. To evaluate the feasibility of using Mechanical Turk as a massively parallel platform to perform manual segmentations of macular spectral domain optical coherence tomography (SD-OCT) images using a MapReduce framework. Methods. A macular SD-OCT volume of 61 slice images was map-distributed to Amazon Mechanical Turk. Each Human Intelligence Task was set to $0.01 and required the user to draw five lines to outline the sublayers of the retinal OCT image after being shown example images. Each image was submitted twice for segmentation, and interrater reliability was calculated. The interface was created using custom HTML5 and JavaScript code, and data analysis was performed using R. An automated pipeline was developed to handle the map and reduce steps of the framework. Results. More than 93,500 data points were collected using this framework for the 61 images submitted. Pearson’s correlation of interrater reliability was 0.995 (p<0.0001) and coefficient of determination was 0.991. The cost of segmenting the macular volume was $1.21. A total of 22 individual Mechanical Turk users provided segmentations, each completing an average of 5.5 HITs. Each HIT was completed in an average of 4.43 minutes. Conclusions. Amazon Mechanical Turk provides a cost-effective, scalable, high-availability infrastructure for manual segmentation of OCT images.http://dx.doi.org/10.1155/2016/6571547
spellingShingle Aaron Y. Lee
Cecilia S. Lee
Pearse A. Keane
Adnan Tufail
Use of Mechanical Turk as a MapReduce Framework for Macular OCT Segmentation
Journal of Ophthalmology
title Use of Mechanical Turk as a MapReduce Framework for Macular OCT Segmentation
title_full Use of Mechanical Turk as a MapReduce Framework for Macular OCT Segmentation
title_fullStr Use of Mechanical Turk as a MapReduce Framework for Macular OCT Segmentation
title_full_unstemmed Use of Mechanical Turk as a MapReduce Framework for Macular OCT Segmentation
title_short Use of Mechanical Turk as a MapReduce Framework for Macular OCT Segmentation
title_sort use of mechanical turk as a mapreduce framework for macular oct segmentation
url http://dx.doi.org/10.1155/2016/6571547
work_keys_str_mv AT aaronylee useofmechanicalturkasamapreduceframeworkformacularoctsegmentation
AT ceciliaslee useofmechanicalturkasamapreduceframeworkformacularoctsegmentation
AT pearseakeane useofmechanicalturkasamapreduceframeworkformacularoctsegmentation
AT adnantufail useofmechanicalturkasamapreduceframeworkformacularoctsegmentation