Math items about real-world content lower test-scores of students from families with low socioeconomic status

Socioeconomic status (SES) remains a significant predictor of academic performance, despite efforts to reduce achievement gaps. Standardized math tests, crucial for academic progression, are designed to measure mathematical abilities but may contain unintended biases. If personal characteristics disproportionately affect performance on specific items for certain groups (like low-SES students), test results are biased. Previous research has shown that language complexity in math items negatively impacts low-SES students more than high-SES students. This study expands on this by investigating whether the *content* of math items—the context surrounding the mathematical question—introduces bias against low-SES students. The researchers hypothesized that items focusing on money, food, and social relationships (ecologically relevant for low-SES students) would improve their performance relative to their overall average math scores. This expectation stems from the 'hidden talents' approach, which suggests that adversity can enhance certain cognitive abilities relevant to overcoming specific challenges. Conversely, deficit models posit that adversity impairs cognitive function. The study uses TIMSS data to test this hypothesis, employing 'number of books at home' as an SES indicator due to its established correlation with other SES components in various countries. The study acknowledges the multifaceted nature of SES, noting that low-SES individuals experience higher levels of adversity, such as limited resources and increased exposure to threat, which may impact cognitive function. However, the 'hidden talents' approach proposes that adversity can also foster adaptive cognitive abilities.

Existing literature establishes that low-SES students generally score lower on math tests than high-SES students. However, the definitions and measures of SES vary across studies. This paper defines SES based on wealth and education, using the number of books in the home as a proxy. This measure, frequently used in cross-national educational research, correlates moderately with other SES indicators like financial resources and parental occupational prestige. It's crucial to distinguish between SES and the factors driving the SES-academic performance relationship. Low-SES conditions are often associated with various forms of adversity, including limited access to resources and increased exposure to threat. While acknowledging the impact of such factors (which deficit models emphasize), the study also considers adaptive developmental processes that could enhance cognitive abilities in response to adversity. The 'hidden talents' approach proposes that adversity can strengthen specific cognitive skills, leading to enhanced performance on ecologically relevant tasks. Research from anthropology and cultural psychology supports this, showing that low-SES individuals can effectively solve real-world problems with ecologically relevant content but struggle with equivalent abstract problems in formal settings. This suggests that existing educational systems may not fully capture the cognitive abilities of low-SES students. The study highlights the importance of recognizing both deficits and strengths in low-SES students to reduce stigma and improve academic persistence.

This study uses data from the Trends in International Mathematics and Science Studies (TIMSS) waves 2007 and 2011, encompassing data from 58 countries and students in grades 4 and 8 (N = 5,501,165). The 'number of books at home' served as the SES indicator, categorized into five levels. Items were classified into 'low-SES ecologically relevant content' (money, food, social interaction) and 'low-SES neutral content' (neutral word problems and mathematical notation). Two independent coders classified items, achieving an 82% agreement rate. Linguistic features of items (word count, sentence length, etc.) were also coded using Textalyser. Student-level analyses used mixed logistic regression, predicting correct answers based on SES, item relevance (interaction term), individual test scores, and item features. Separate analyses were conducted for grades 4 and 8 and for specific types of relevant content (money, food, social interaction). Item-level analyses employed Differential Item Functioning (DIF) analyses, specifically the Mantel-Haenszel (MH) procedure and Logistic Regression, to determine if items with ecologically relevant content showed bias against low-SES students. This involved comparing the probability of a correct response for low- and high-SES students with the same overall test scores. Linear regression was then used to examine the relationship between DIF odds and item characteristics, controlling for various factors.

Contrary to the initial hypothesis, analyses at both the student and item levels revealed that low-SES students performed significantly worse on items with ecologically relevant content (money, food, social interaction) compared to items with neutral content. Student-level analyses showed a significant interaction between SES and item relevance. Low-SES students had an 18% (grade 4) and 16% (grade 8) lower chance of answering correctly on ecologically relevant items, controlling for overall math ability and item characteristics (Table 1). This effect was particularly strong for items involving money and social interaction in both grades and food in grade 4 (but not grade 8). (Figure 1 illustrates the interaction effect in grade 8). Item-level analyses (DIF) using the Mantel-Haenszel procedure showed that items with ecologically relevant content had significantly lower odds (0.91) of being correctly answered by low-SES students compared to neutral items (1.02 and 1.06, Table 3). Linear regression analysis confirmed this difference, even after controlling for item characteristics (b=-0.09, t(160) = -2.55, p = 0.012, Cohen's d = 0.70, a medium effect size). This effect was especially pronounced for items related to money and social interaction, and to a lesser extent, food (Figure 2).

The findings contradict the initial hypothesis and the 'hidden talents' approach, which predicted enhanced performance for low-SES students on ecologically relevant items. Instead, the results suggest that this type of content hinders performance. Several explanations are proposed. First, the 'attention capture' hypothesis suggests that salient content related to scarcity (money, food) might distract low-SES students, reducing cognitive resources. Second, difficulties transferring between informal and formal mathematical knowledge may be exacerbated by the high salience of ecologically relevant content, making it difficult for students to apply learned school-based algorithms. Finally, stereotype threat, where cues related to marginalized group identity negatively impact performance, could also be a factor. The study contrasts its findings with research showing enhanced performance on executive function tasks with ecologically relevant content, speculating that the type of cognitive skills tested (executive function vs. formal math knowledge) might explain the discrepancy. Future research should investigate the different phases of problem-solving (understanding, planning, carrying out, evaluating) to understand how ecologically relevant content impacts performance at each stage. Experimental studies incorporating self-affirmation interventions are needed to assess the role of stereotype threat.

This study demonstrates an unexpected negative relationship between ecologically relevant content in math items and the performance of low-SES students. This raises concerns about bias in standardized testing and challenges the simple application of the 'hidden talents' approach. Future research should focus on experimental manipulations to clarify the underlying mechanisms (attentional, cognitive, affective) and explore which types of content are universally biased. Addressing this bias is crucial for ensuring fair and equitable assessment in education, potentially involving teacher-level and test-level interventions. The study also prompts a broader discussion on the appropriate balance between comparing absolute test scores and tracking individual student progress.

The observational nature of this study limits causal inferences. While controlling for key item features, unobserved variables may have influenced the results. The use of 'number of books at home' as the sole SES indicator is also a limitation. The method of calculating the students' average test score as the matching criterion for the DIF analysis potentially underestimates the true impact of ecologically relevant content on low-SES students' performance. Future experimental studies are needed to address these limitations.