Caveat emptor, computational social science: Large-scale missing data in a widely-published Reddit corpus.
As researchers use computational methods to study complex social behaviors at scale, the validity of this computational social science depends on the integrity of the data. On July 2, 2015, Jason Baumgartner published a dataset advertised to include "every publicly available Reddit comment"...
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2018-01-01
|
| Series: | PLoS ONE |
| Online Access: | https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0200162&type=printable |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850230134880075776 |
|---|---|
| author | Devin Gaffney J Nathan Matias |
| author_facet | Devin Gaffney J Nathan Matias |
| author_sort | Devin Gaffney |
| collection | DOAJ |
| description | As researchers use computational methods to study complex social behaviors at scale, the validity of this computational social science depends on the integrity of the data. On July 2, 2015, Jason Baumgartner published a dataset advertised to include "every publicly available Reddit comment" which was quickly shared on Bittorrent and the Internet Archive. This data quickly became the basis of many academic papers on topics including machine learning, social behavior, politics, breaking news, and hate speech. We have discovered substantial gaps and limitations in this dataset which may contribute to bias in the findings of that research. In this paper, we document the dataset, substantial missing observations in the dataset, and the risks to research validity from those gaps. In summary, we identify strong risks to research that considers user histories or network analysis, moderate risks to research that compares counts of participation, and lesser risk to machine learning research that avoids making representative claims about behavior and participation on Reddit. |
| format | Article |
| id | doaj-art-333fd7efca614a22b5ba544821348e70 |
| institution | OA Journals |
| issn | 1932-6203 |
| language | English |
| publishDate | 2018-01-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLoS ONE |
| spelling | doaj-art-333fd7efca614a22b5ba544821348e702025-08-20T02:03:58ZengPublic Library of Science (PLoS)PLoS ONE1932-62032018-01-01137e020016210.1371/journal.pone.0200162Caveat emptor, computational social science: Large-scale missing data in a widely-published Reddit corpus.Devin GaffneyJ Nathan MatiasAs researchers use computational methods to study complex social behaviors at scale, the validity of this computational social science depends on the integrity of the data. On July 2, 2015, Jason Baumgartner published a dataset advertised to include "every publicly available Reddit comment" which was quickly shared on Bittorrent and the Internet Archive. This data quickly became the basis of many academic papers on topics including machine learning, social behavior, politics, breaking news, and hate speech. We have discovered substantial gaps and limitations in this dataset which may contribute to bias in the findings of that research. In this paper, we document the dataset, substantial missing observations in the dataset, and the risks to research validity from those gaps. In summary, we identify strong risks to research that considers user histories or network analysis, moderate risks to research that compares counts of participation, and lesser risk to machine learning research that avoids making representative claims about behavior and participation on Reddit.https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0200162&type=printable |
| spellingShingle | Devin Gaffney J Nathan Matias Caveat emptor, computational social science: Large-scale missing data in a widely-published Reddit corpus. PLoS ONE |
| title | Caveat emptor, computational social science: Large-scale missing data in a widely-published Reddit corpus. |
| title_full | Caveat emptor, computational social science: Large-scale missing data in a widely-published Reddit corpus. |
| title_fullStr | Caveat emptor, computational social science: Large-scale missing data in a widely-published Reddit corpus. |
| title_full_unstemmed | Caveat emptor, computational social science: Large-scale missing data in a widely-published Reddit corpus. |
| title_short | Caveat emptor, computational social science: Large-scale missing data in a widely-published Reddit corpus. |
| title_sort | caveat emptor computational social science large scale missing data in a widely published reddit corpus |
| url | https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0200162&type=printable |
| work_keys_str_mv | AT devingaffney caveatemptorcomputationalsocialsciencelargescalemissingdatainawidelypublishedredditcorpus AT jnathanmatias caveatemptorcomputationalsocialsciencelargescalemissingdatainawidelypublishedredditcorpus |