How representative is the Swedish PISA sample? A comparison of PISA and register data

Abstract

PISA aims to serve as a “global yardstick” for educational success, as measured by student performance. For comparisons to be meaningful across countries or over time, PISA samples must be representative of the population of 15-year-old students in each country. Exclusions and non-response can undermine this representativeness and potentially bias estimates of student performance. Unfortunately, testing the representativeness of PISA samples is typically infeasible due to unknown population parameters. To address this issue, we integrate PISA 2018 data with comprehensive Swedish registry data, which includes all students in Sweden. Utilizing various achievement measures, we find that the Swedish PISA sample significantly overestimates the achievement levels in Sweden. The observed difference equates to standardized effect sizes ranging from d = .19 to .28, corresponding to approximately 25 points on the PISA scale or an additional year of schooling. The paper concludes with a plea for more rigorous quality standards and their strict enforcement.