The InterModel Vigorish (IMV) as a flexible and portable approach for quantifying predictive accuracy with binary outcomes.
Understanding the "fit" of models designed to predict binary outcomes has been a long-standing problem across the social sciences. We propose a flexible, portable, and intuitive metric for quantifying the change in accuracy between two predictive systems in the case of a binary outcome: th...
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2025-01-01
|
| Series: | PLoS ONE |
| Online Access: | https://doi.org/10.1371/journal.pone.0316491 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850265777037377536 |
|---|---|
| author | Benjamin W Domingue Charles Rahal Jessica Faul Jeremy Freese Klint Kanopka Alexandros Rigos Ben Stenhaug Ajay Shanker Tripathi |
| author_facet | Benjamin W Domingue Charles Rahal Jessica Faul Jeremy Freese Klint Kanopka Alexandros Rigos Ben Stenhaug Ajay Shanker Tripathi |
| author_sort | Benjamin W Domingue |
| collection | DOAJ |
| description | Understanding the "fit" of models designed to predict binary outcomes has been a long-standing problem across the social sciences. We propose a flexible, portable, and intuitive metric for quantifying the change in accuracy between two predictive systems in the case of a binary outcome: the InterModel Vigorish (IMV). The IMV is based on an analogy to weighted coins, well-characterized physical systems with tractable probabilities. The IMV is always a statement about the change in fit relative to some baseline model-which can be as simple as the prevalence-whereas other metrics are stand-alone measures that need to be further manipulated to yield indices related to differences in fit across models. Moreover, the IMV is consistently interpretable independent of baseline prevalence. We contrast this metric with alternatives in numerous simulations. The IMV is more sensitive to estimation error than many alternatives and also shows distinctive sensitivity to prevalence. We consider its performance using examples spanning the social and natural sciences. The IMV allows for precise answers to questions about changes in model fit in a variety of settings in a manner that will be useful for furthering research and the understanding of social outcomes. |
| format | Article |
| id | doaj-art-722d2ef90eb948eca4ce896380a9143c |
| institution | OA Journals |
| issn | 1932-6203 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLoS ONE |
| spelling | doaj-art-722d2ef90eb948eca4ce896380a9143c2025-08-20T01:54:19ZengPublic Library of Science (PLoS)PLoS ONE1932-62032025-01-01203e031649110.1371/journal.pone.0316491The InterModel Vigorish (IMV) as a flexible and portable approach for quantifying predictive accuracy with binary outcomes.Benjamin W DomingueCharles RahalJessica FaulJeremy FreeseKlint KanopkaAlexandros RigosBen StenhaugAjay Shanker TripathiUnderstanding the "fit" of models designed to predict binary outcomes has been a long-standing problem across the social sciences. We propose a flexible, portable, and intuitive metric for quantifying the change in accuracy between two predictive systems in the case of a binary outcome: the InterModel Vigorish (IMV). The IMV is based on an analogy to weighted coins, well-characterized physical systems with tractable probabilities. The IMV is always a statement about the change in fit relative to some baseline model-which can be as simple as the prevalence-whereas other metrics are stand-alone measures that need to be further manipulated to yield indices related to differences in fit across models. Moreover, the IMV is consistently interpretable independent of baseline prevalence. We contrast this metric with alternatives in numerous simulations. The IMV is more sensitive to estimation error than many alternatives and also shows distinctive sensitivity to prevalence. We consider its performance using examples spanning the social and natural sciences. The IMV allows for precise answers to questions about changes in model fit in a variety of settings in a manner that will be useful for furthering research and the understanding of social outcomes.https://doi.org/10.1371/journal.pone.0316491 |
| spellingShingle | Benjamin W Domingue Charles Rahal Jessica Faul Jeremy Freese Klint Kanopka Alexandros Rigos Ben Stenhaug Ajay Shanker Tripathi The InterModel Vigorish (IMV) as a flexible and portable approach for quantifying predictive accuracy with binary outcomes. PLoS ONE |
| title | The InterModel Vigorish (IMV) as a flexible and portable approach for quantifying predictive accuracy with binary outcomes. |
| title_full | The InterModel Vigorish (IMV) as a flexible and portable approach for quantifying predictive accuracy with binary outcomes. |
| title_fullStr | The InterModel Vigorish (IMV) as a flexible and portable approach for quantifying predictive accuracy with binary outcomes. |
| title_full_unstemmed | The InterModel Vigorish (IMV) as a flexible and portable approach for quantifying predictive accuracy with binary outcomes. |
| title_short | The InterModel Vigorish (IMV) as a flexible and portable approach for quantifying predictive accuracy with binary outcomes. |
| title_sort | intermodel vigorish imv as a flexible and portable approach for quantifying predictive accuracy with binary outcomes |
| url | https://doi.org/10.1371/journal.pone.0316491 |
| work_keys_str_mv | AT benjaminwdomingue theintermodelvigorishimvasaflexibleandportableapproachforquantifyingpredictiveaccuracywithbinaryoutcomes AT charlesrahal theintermodelvigorishimvasaflexibleandportableapproachforquantifyingpredictiveaccuracywithbinaryoutcomes AT jessicafaul theintermodelvigorishimvasaflexibleandportableapproachforquantifyingpredictiveaccuracywithbinaryoutcomes AT jeremyfreese theintermodelvigorishimvasaflexibleandportableapproachforquantifyingpredictiveaccuracywithbinaryoutcomes AT klintkanopka theintermodelvigorishimvasaflexibleandportableapproachforquantifyingpredictiveaccuracywithbinaryoutcomes AT alexandrosrigos theintermodelvigorishimvasaflexibleandportableapproachforquantifyingpredictiveaccuracywithbinaryoutcomes AT benstenhaug theintermodelvigorishimvasaflexibleandportableapproachforquantifyingpredictiveaccuracywithbinaryoutcomes AT ajayshankertripathi theintermodelvigorishimvasaflexibleandportableapproachforquantifyingpredictiveaccuracywithbinaryoutcomes AT benjaminwdomingue intermodelvigorishimvasaflexibleandportableapproachforquantifyingpredictiveaccuracywithbinaryoutcomes AT charlesrahal intermodelvigorishimvasaflexibleandportableapproachforquantifyingpredictiveaccuracywithbinaryoutcomes AT jessicafaul intermodelvigorishimvasaflexibleandportableapproachforquantifyingpredictiveaccuracywithbinaryoutcomes AT jeremyfreese intermodelvigorishimvasaflexibleandportableapproachforquantifyingpredictiveaccuracywithbinaryoutcomes AT klintkanopka intermodelvigorishimvasaflexibleandportableapproachforquantifyingpredictiveaccuracywithbinaryoutcomes AT alexandrosrigos intermodelvigorishimvasaflexibleandportableapproachforquantifyingpredictiveaccuracywithbinaryoutcomes AT benstenhaug intermodelvigorishimvasaflexibleandportableapproachforquantifyingpredictiveaccuracywithbinaryoutcomes AT ajayshankertripathi intermodelvigorishimvasaflexibleandportableapproachforquantifyingpredictiveaccuracywithbinaryoutcomes |