Articles
Authors: Hannah Silverstein (University of North Carolina - Chapel Hill) , Sudhanshu Handa (University of North Carolina - Chapel Hill) , David Thissen (University of North Carolina - Chapel Hill)
Raven’s Progressive Matrices measure logical reasoning and are often included in large multi-topic surveys in low and middle-income countries. The matrices are image-based items that do not require formal knowledge of language or math to complete. As such, they are attractive items to measure logical reasoning in international development contexts. Many of these large field surveys include short item Raven’s sets because space is too limited to fit a full suite. However, short sets can result in restricted variation in terms of test scores. In this paper, we use a nominal response model (NRM) form of item response theory (IRT) to uncover hidden variation in right and wrong answers using short-item Raven’s tests from two large field surveys in Malawi and Zambia. We also analyze relationships between a set of other variables, comparing performance of different versions of the logical reasoning scores as both independent and dependent variables, checking the validity of the new scores. The new NRM-estimated logical reasoning scores follow a more normal distribution in both samples. Validity checks suggest that when relationships are less strong, NRM-estimated scores can capture more nuance than summed scores or even 2 parameter logistic IRT-estimated scores. NRM can uncover differences that are not apparent when using simple summed scores.
Keywords: Raven’s Progressive Matrices, item response theory, cognitive ability, Malawi, Zambia
How to Cite: Silverstein, H. , Handa, S. & Thissen, D. (2023) “Large Field Surveys and Short Items Sets: Estimating Reasoning Skills among Distinct Samples in Malawi and Zambia”, Journal of Methods and Measurement in the Social Sciences. 14(1). doi: https://doi.org/10.2458/jmmss.6047