Comparing Results of TIMSS and the Hungarian National Assessment of Basic Competencies

In our paper, we compared some characteristics of TIMSS 2015 and the National Assessment of Basic Competencies (NABC) 2015. The NABC assesses all students’ reading and mathematics performance in Grades 6, 8 and 10. Both studies assessed Hungarian Grade 8 students’ mathematics abilities in the spring of 2015. We linked data of the two studies on the student level using Student Measurement IDs. We compared TIMSS and NABC mathematics scales based on the Assessment Framework of the two studies along with the results of students in the two assessments. The comparison of the Frameworks revealed that although the two tests use similar content and cognitive categorizations, there are crucial differences between the two constructs. While the basis of TIMSS’s mathematics construct is the common part of mathematics curricula of participating countries, NABC intends to measure mathematical literacy, the ability of students to use their mathematical knowledge and competencies in real life situations. The correlation between the TIMSS and NABC mathematics test results (0.79) also confirms that the two tests measure related, but not identical abilities. To evaluate the representativeness of the TIMSS sample we used school- and class-level weight factors of TIMSS and the student-level weights of NABC combined. The mean performances of the TIMSS sample are only slightly lower than the full NABC cohort’s, the effect size of the difference is 0.042 and 0.046 in mathematics and reading respectively. The differences in the standard deviations are somewhat but not considerably larger. The SES-index shows a very good match with no statistically significant differences in the mean and standard deviation of the sample and the full cohort. Our analysis confirms that estimations of population parameters based on TIMSS samples are of a good quality.

1. Aux-Bánfi, I., Balázsi, I., Balkányi, P., Balogh, V. K., Gyapay, J., Lak, Á. R., Ostorics, L. I., Palincsár, I., Rábai-Szabó, A., Rózsa, Cs., Szabó, Á., Szabó, L. D., Szepesi, I., Szipőcs-Krolopp, J., & Vadász, Cs. (2015). Országos kompetenciamérés. Technikai leírás. Retrieved from https://www.oktatas.hu/kozneveles/meresek/kompetenciameres/tanulmanyok_publikaciok.

2. Balázsi, I., Balkányi, P., Bánfi, I., Szalay, B., & Szepesi, I. (2012). A PIRLS és TIMSS 2011 tartalmi és technikai jellemzői. Retrieved from https://www.oktatas.hu/kozneveles/meresek/timss/timss_2011_meres.

3. Balázsi, I., Balkányi, P., Ostorics, L., Palincsár, I., Rábai-Szabó, A., Szepesi, I., Szipőcs-Krolopp, J., & Vadász, Cs. (2014). Az Országos kompetenciamérés tartalmi keretei – Szövegértés, matematika, háttérkérdőívek. Retrieved from https://www.oktatas.hu/kozneveles/meresek/kompetenciameres/tanulmanyok_publikaciok.

4. Balázsi, I., Lak, Á. R., Ostorics, L., Szabó, L. D., & Vadász, Cs. (2016). Országos kompetenciamérés 2015 – Országos jelentés. Retrieved from https://www.oktatas.hu/kozneveles/meresek/kompetenciameres/eredmenyek.

5. Foy, P. & Yin, L. (2016). Scaling the TIMSS 2015 Achievement Data. In M. O. Martin, I. V. S. Mullis, & M. Hooper, (Eds.), Methods and procedures in TIMSS 2015. Retrieved from http://timssandpirls.bc.edu/publications/timss/2015-methods.html.

6. Hedges, L. V. (2007). Effect Sizes in Cluster-Randomized Designs. Journal of Educational and Behavioral Statistics, 32(4), 341–370. CrossRef

7. Horn, D. (2013). Diverging performances: The detrimental effects of early educational selection on equality of opportunity in Hungary. Research in Social Stratification and Mobility, 32(June), 25–43. CrossRef

8. Kertesi, G. & Kézdi, G. (2016). On the Test Score Gap between Roma and non-Roma Students in Hungary and its Potential Causes. Economics of Transition, 24(1), 135–162. CrossRef

9. Lak, Á. R., Palincsár, I., Szabó, L. D., Szepesi, I., & Szipőcs-Krolopp, J. (2016). Országos kompetenciamérés 2015 – Feladatok és jellemzőik. Matematika, 8. évfolyam. Retrieved from https://www.oktatas.hu/kozneveles/meresek/kompetenciameres/feladatsorok.

10. LaRoche, S., Joncas, M., & Foy, P. (2016). Sampling design in TIMSS 2015. In M. O. Martin, I. V. S. Mullis, & M. Hooper, (Eds.), Methods and procedures in TIMSS 2015. Retrieved from http://timssandpirls.bc.edu/publications/timss/2015-methods.html.

11. Martin, M. O., Mullis, I. V. S., and Hooper, M. (Eds.). (2016). Methods and procedures in TIMSS 2015. Retrieved from http://timssandpirls.bc.edu/publications/timss/2015-methods.html.

12. Mullis, I. V. S., Martin, M. O., Foy, P., & Hooper, M. (2016). TIMSS 2015 International results in mathematics. Retrieved from http://timssandpirls.bc.edu/timss2015/international-results.

13. Mullis, I.V.S. & Martin, M.O. (Eds.) (2013). TIMSS 2015 assessment frameworks. Retrieved from http://timssandpirls.bc.edu/timss2015/frameworks.html.

14. Neidorf, T.S., Binkley, M., Gattis, K., & Nohara, D. (2006). Comparing mathematics content in the National Assessment of Educational Progress (NAEP), Trends in International Mathematics and Science Study (TIMSS), and Program for International Student Assessment (PISA) 2003 assessments (NCES 2006-029). Retrieved from http://nces.ed.gov/pubsearch.

15. Sinka, E. (2010). OECD review on evaluation and assessment frameworks for improving school outcomes. Country background report. Hungary. Retrieved from http://www.oecd.org/edu/school/50484774.pdf.

16. Szalay, B., Szepesi, I., & Vadász, Cs. (2016). TIMSS 2015 Összefoglaló jelentés. Retrieved from https://www.oktatas.hu/kozneveles/meresek/timss/timss_2015_meres.

17. Yamamoto, K. & Kulick, E. (2016). TIMSS 2015 achievement scaling methodology. In M. O. Martin, I. V. S. Mullis, & M. Hooper, (Eds.), Methods and procedures in TIMSS 2015. Retrieved from http://timssandpirls.bc.edu/publications/timss/2015-methods.html.