Invalid Permutation Tests

Permutation tests are often presented in a rather casual manner, in both introductory and advanced statistics textbooks. The appeal of the cleverness of the procedure seems to replace the need for a rigorous argument that it produces valid hypothesis tests. The consequence of this educational failin...

Full description

Saved in:
Bibliographic Details
Main Author: Mikel Aickin
Format: Article
Language:English
Published: Wiley 2010-01-01
Series:International Journal of Mathematics and Mathematical Sciences
Online Access:http://dx.doi.org/10.1155/2010/769780
Tags: Add Tag
No Tags, Be the first to tag this record!
_version_ 1832557011449937920
author Mikel Aickin
author_facet Mikel Aickin
author_sort Mikel Aickin
collection DOAJ
description Permutation tests are often presented in a rather casual manner, in both introductory and advanced statistics textbooks. The appeal of the cleverness of the procedure seems to replace the need for a rigorous argument that it produces valid hypothesis tests. The consequence of this educational failing has been a widespread belief in a “permutation principle”, which is supposed invariably to give tests that are valid by construction, under an absolute minimum of statistical assumptions. Several lines of argument are presented here to show that the permutation principle itself can be invalid, concentrating on the Fisher-Pitman permutation test for two means. A simple counterfactual example illustrates the general problem, and a slightly more elaborate counterfactual argument is used to explain why the main mathematical proof of the validity of permutation tests is mistaken. Two modifications of the permutation test are suggested to be valid in a very modest simulation. In instances where simulation software is readily available, investigating the validity of a specific permutation test can be done easily, requiring only a minimum understanding of statistical technicalities.
format Article
id doaj-art-39f193a99846494e907b69dfc0b731b9
institution Kabale University
issn 0161-1712
1687-0425
language English
publishDate 2010-01-01
publisher Wiley
record_format Article
series International Journal of Mathematics and Mathematical Sciences
spelling doaj-art-39f193a99846494e907b69dfc0b731b92025-02-03T05:43:45ZengWileyInternational Journal of Mathematics and Mathematical Sciences0161-17121687-04252010-01-01201010.1155/2010/769780769780Invalid Permutation TestsMikel Aickin0Department of Family & Community Medicine, University of Arizona, Tucson, AZ 85724, USAPermutation tests are often presented in a rather casual manner, in both introductory and advanced statistics textbooks. The appeal of the cleverness of the procedure seems to replace the need for a rigorous argument that it produces valid hypothesis tests. The consequence of this educational failing has been a widespread belief in a “permutation principle”, which is supposed invariably to give tests that are valid by construction, under an absolute minimum of statistical assumptions. Several lines of argument are presented here to show that the permutation principle itself can be invalid, concentrating on the Fisher-Pitman permutation test for two means. A simple counterfactual example illustrates the general problem, and a slightly more elaborate counterfactual argument is used to explain why the main mathematical proof of the validity of permutation tests is mistaken. Two modifications of the permutation test are suggested to be valid in a very modest simulation. In instances where simulation software is readily available, investigating the validity of a specific permutation test can be done easily, requiring only a minimum understanding of statistical technicalities.http://dx.doi.org/10.1155/2010/769780
spellingShingle Mikel Aickin
Invalid Permutation Tests
International Journal of Mathematics and Mathematical Sciences
title Invalid Permutation Tests
title_full Invalid Permutation Tests
title_fullStr Invalid Permutation Tests
title_full_unstemmed Invalid Permutation Tests
title_short Invalid Permutation Tests
title_sort invalid permutation tests
url http://dx.doi.org/10.1155/2010/769780
work_keys_str_mv AT mikelaickin invalidpermutationtests