Demonstration of Simpson’s Paradox in PISA 2015 Data: Confusing Differences between Boys and Girls

This paper explores the occurrence of a Simpson’s paradox in PISA 2015 science literacy data. Simpson’s paradox, a case of contradicting interpretations when results are analysed by groups or aggregated as a whole, has both a practical and an academic significance. It is an interesting phenomenon that is far from theoretical and when it happens, it has profound effects on the interpretation and if left unidentified can cause confusion and misunderstanding. This paper demonstrates best ways to detect Simpson’s paradox through appropriate tables and graphs. Actual occurrences of a Simpson’s paradox and conditions leading to them are explored using PISA 2015 gender differences in science literacy data in five central European countries − Austria, Croatia, Czech Republic, Slovakia and Slovenia. In countries where the occurrence of a Simpson’s paradox was detected, we provide correct interpretation of the results. Beside creating problems with interpretation an occurrence of a Simpson’s paradox also provides new insight − it signifies that there is very different gender composition in different educational tracks which has important implications for the educational governance. We will discuss implications of these findings in context of Slovenian educational system.

1. Blyth, C. R. (1972). On Simpson's paradox and the sure-thing principle. Journal of the American Statistical Association, 67(338), 364–366. CrossRef

2. Baker, S. G., & Kramer, B. S. (2001). Good for women, good for men, bad for people: Simpson's paradox and the importance of sex-specific analysis in observational studies. Journal of Women's Health and Gender-Based Medicine, 10(9), 867–872. CrossRef

3. Cohen, B. S., & Moch, P. L. (2017). Guarding against Simpson's paradox when combining data sets. Curriculum and Teaching Dialogue, 19(1 & 2), 153.

4. Demers, S., & Rossmo, D. K. (2015). Simpson's paradox in Canadian police clearance rates. Canadian Journal of Criminology and Criminal Justice, 57(3), 424–434. CrossRef

5. European Commission. (2016). She figures 2015. Luxembourg: Publications Office of the European Union. Retrieved from http://ec.europa.eu/research/swafs/pdf/pub_gender_equality/she_figures_2015-final.pdf.

6. Gardner, M. (1982). Aha! Gotcha: paradoxes to puzzle and delight. San Francisco: Freeman.

7. Lesser, L.M. (2001), Representations of reversal: An exploration of Simpson's paradox. In A. A. Cuoco, & F. R. Curcio (Eds.), The roles of representation in school mathematics (pp. 129-145). Reston, Virginia: National Council of Teachers of Mathematics.

8. OECD. (2017). PISA 2015 technical report. Paris: OECD Publishing.

9. R Core Team (2017). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. On-line: https://www.R-project.org.

10. Rücker, G., & Schumacher, M. (2008). Simpson's paradox visualized: The example of the Rosiglitazone meta-analysis. BMC Medical Research Methodology, 8(1). CrossRef

11. Simpson, E. H. (1951). The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society, 13(2), 238–241.

12. Smith, E. (2008). Using secondary data in educational and social research. New York, NY : McGraw Hill/Open University Press.

13. Tan, A. (1986). A geometric interpretation of Simpson's paradox. College Mathematics Journal, 17, 340-341. CrossRef

14. UNESCO (2014). UNESCO's promise: Gender equality, a global priority. Paris: UNESCO. Retrieved from http://unesdoc.unesco.org/images/0022/002269/226923m.pdf.

15. Wright, B. (2012). Best of N contests: Implications of Simpson's paradox in tennis [Masters Thesis]. The Florida State University.

16. Yule, G. U. (1903). Notes on the theory of association of attributes in statistics. Biometrika, 2(2), 121–134. CrossRef