# A Partial Simulation Study of Phantom Effects in Multilevel Analysis of School Effects: The Case of School Socioeconomic Composition:

Abstract: Hierarchical linear modeling (HLM) is often used to estimate the effects of socioeconomic status (SES) on academic achievement at different levels of an educational system. However, if a prior acad...

## Summary (4 min read)

### 1.2 Phantom Effects

- Phantom effects can also occur at other levels of an educational system.
- At the student level, in the absence of SES, the racial-ethnic background often indicates statistically significant effects on academic achievement of students; however, in the presence of SES, such significant effects often disappear, which makes the racial-ethnic effects phantom effects.
- With a focus on school SES, the present study investigates school contextual effects as a potential source of phantom effects in the school effectiveness research literature.

### 1.3 Contextual Effects

- At the level 2, school SES was the mean SES for each school.
- A two-level random intercept HLM was applied.
- The results showed that after controlling for student's prior academic achievement, school SES effect disappeared in both cases (i.e., SES_1 and SES_2).
- The effects of school SES were phantom effects.

### 1.4 Purpose of Research

- The purpose of the present study is to examine the extent to which the effects.
- The combination of empirical answers to both questions will provide evidence to address the issue of the extent to which the effects of school SES on science achievement of students are phantom effects.

### 1.5.1 Informing Policy Change

- As argued earlier, many researchers have shown that school SES largely affects students ' academic achievement (e.g., OECD, 2015) .
- The phenomenon of phantom effects associated with school SES may threaten the credibility of claims like this.
- To some degree, education policymakers may have been misinformed on research evidence due to the complexity concerning school contextual effects, especially school SES.
- This study aims to provide empirical evidence on whether phantom effects of school SES on students' academic achievement exist and, if yes, the extent to which school SES produces phantom effects on students' academic achievement.
- The significance of this study is that it may promote policy change through a revisiting of educational policies and practices concerning school SES.

### 1.5.2 Promoting Pioneer Research

- Among many important school characteristics, school socioeconomic status (SES)-a school background variable-plays a critical role in many educational policies and practices.
- In New Zealand and the United Kingdom, schools adopt a funding model that provides similar resources to all schools and provides additional funding to schools with high needs (e.g. rural school, high percentage of students from low SES, etc.) (Perry and McConney, 2010) .
- In the United States, policymakers issued different polices aimed to adjust school SES for better distribution of educational resources, such as magnet schools and school assignment policy.
- The present study considers this important school characteristic.

### 2.2 School socioeconomic composition

- At school level, school SES related with student academic achievement (e.g., Ma, 2010) .
- School SES is often measured in two ways, either as the proportion of students enrolled in a reduced-price or free lunch program (Sirin 2005).
- In general, School SES is dependent upon student SES.

### 2.3 Effects of School SES on Academic Achievement

- If the coefficient is positive, school SES improves student academic achievement.
- If the coefficient is negative, school SES hinders student academic achievement.
- All studies referenced earlier showed evidence to support that students in high SES schools performed better than students in low SES schools.
- Some of them, however, indicated that the effects of school SES can be conditional.
- Lam and Lau's study (2014) , after controlling school size on school's level, showed that the school SES effects disappeared.

### 2.5 Phantom effects of School SES

- The first method is that MLM includes variables that highly correlated with students' present academic achievement (Harker & Tymms, 2004) .
- The above illustration pertains to this approach.
- In other words, the effects of school SES are phantom effects because the model cannot adequately control for measurement errors.
- There are few empirical studies to support this view.
- Showed the same pattern as taking Year 5 student achievement as prior ability.

### 3.5 Partial Simulation

- Simulated data would then work with actual data to address a statistical issue, thus named partial simulation.
- The partial simulation procedure can generate a random variable with a defined correlation to an existing variable.
- Once these prior measures of science achievement were generated, a separate multilevel analysis was performed with models that were discussed in the previous section.
- As a result, 10 sets of multilevel analyses were conducted for each of the ten correlation conditions.

### 3.6 Working with Plausible Values in Partial Simulation

- A short discussion on the variables employed at the student level and at the school level is in order before the modeling activities.
- In addition, 26percent of the students are native, and 81 percent of the students speak English at home.
- At the school level, the average school size is 1,251 students with a standard deviation of 887 students Meanwhile, 38 percent of the schools are located in city areas, 49 percent of the schools are located in town areas, and 13 percent of the schools are located in rural areas.
- The average school SES is .07 with a standard deviation of .54.
- Disciplinary climate is an index, and the average disciplinary climate is 0.28 with a standard deviation of 0.38.

### 4.1 The Null Model

- The null model (see Chapter 3) provides the background for all the subsequent analyses.
- The results of the null model show that the average science achievement of U.S. students is 494 points.
- Therefore, according to the PISA science scale (M = 500, SD = 100), U.S. students scored a little lower than the international average.
- The variance in science achievement at the student level is 7727.50, and variance in science achievement at the school level is 1876.65.
- Intra-class correlation is approximately 0.20, which indicates that 20 percent of the total variance in science achievement is due to the school level.

### 4.2 The Absolute Effects Models

- It is important to emphasize that, although all effects are statistically significant at the alpha level of .05 in Table 4 .2, some effects have rather small effect sizes.
- If 25 percent of a SD can be considered practically important (e.g., Cohen, 1988) , then phantom effects of school SES appear when a prior measure has a correlation of .65 (even .55) with the present measure.
- Compared with the base model, student SES effects and school SES effects in the model with .75 correlation between prior and present measures are decreased by 51 percent and 53 percent respectively.

### 4.3 The Relative Effects Models

- As discussed in Chapter 3, the relative effects models examine student SES effects and school SES effects in the presence of student and school background variables (at student and school levels).
- Meanwhile, with the increasing correlation between prior science achievement and present science achievement, the association between student SES and student science achievement decreases dramatically as well.
- In the relative model, all effects associated with student SES are below .25 SD.
- Phantom effects of student SES disappear when a prior science achievement reaches .35 (even .25) in correlation to present science achievement measurement.

### 5.3 Implications for Empirical Research

- This study also offers a way to help create prior academic achievement measures when they are not available for data analysis.
- Researchers are encouraged to conduct a thorough literature review to locate possible correlations between prior academic achievement measures and current academic achievement measures.
- When these correlations are known, this study developed a procedure (in the programming language of R) to create prior academic achievement measurements, which will help researchers conduct data analysis based on correctly specified models.

### 5.4 Implications for Policy and Practice

- The effectiveness of these policy practices is open to question based on the evidence in this and other studies.
- The association between school SES and student academic achievement may be attenuated by misspecified contextual models.
- In other words, student SES and school SES may not have as strong effects on student academic achievement as previous studies indicated, if the school contextual models are correctly specified.
- In order to make appropriate policies, policymakers may want to encourage (e.g., fund) research projects that gather appropriate evidence with a fuller data collection from students and schools, particularly including prior student academic achievement measures.

### 5.6 Suggestions for Further Research

- On the other hand, the approach that focuses on potential measurement errors may also be explored further.
- Measurement error and model specification are often tangled up with each other to produce effects on parameter estimation.
- Based on simulation study, the author gave a thumbs-up rule for applying each approach.
- Pokropek (2015) provides only limited information to answer some of the questions, but more comprehensive studies should be conducted.

### 5.7 Conclusion

- The result of this study can be summarized by several important points.
- First, based on partial simulation procedure, phantom effects of school SES and student SES are real.
- The stronger the correlation between prior science achievement measure and present science achievement measure, the greater the decrease in both student SES effects and school SES effects.
- Third, the procedure of partial simulation provides a new angle to conduct theoretical studies (full simulation), which is entirely based on ideal assumption.
- Finally, the procedure of partial simulation offers researchers a way to create prior student academic achievement measures when they are not available for data analysis.

Did you find this useful? Give us your feedback

...read more

##### Citations

12 citations

### Cites methods from "A Partial Simulation Study of Phant..."

...Analysing PISA data, Zhou and Ma (2021) found the stronger the correlation between prior achievement and present achievement, the greater the chance of phantom effects for school SES....

[...]

1 citations

##### References

103,911 citations

### "A Partial Simulation Study of Phant..." refers background in this paper

...If 25 percent of a SD can be considered practically important (e.g., Cohen, 1988), then phantom effects of school SES appear when a prior measure has a correlation of .65 (even .55) with the present measure....

[...]

22,551 citations

19,250 citations

### "A Partial Simulation Study of Phant..." refers background or methods in this paper

...1.3 Contextual Effects In the studies cited above, the compositional model was applied to examine the effects of school SES on student academic achievement (see Raudenbush & Bryk, 2002)....

[...]

...Estimation of the above HLM involves two stages centering around the variance and covariance components (see Raudenbush & Bryk, 2002)....

[...]

3,180 citations

### "A Partial Simulation Study of Phant..." refers background or methods in this paper

...Home resources include household possessions, such as books, a study room, and a computer (Sirin, 2005)....

[...]

...12 Sirin (2005) conducted a meta-analytic review of the relationship between SES and academic performance, which included 58 published journal articles from 1990 to 2000....

[...]

...Scholars showed that student SES positively significant correlated with student academic achievement (e.g., Sirin, 2005; White, 1982)....

[...]

...School SES is often measured in two ways, either as the proportion of students enrolled in a reduced-price or free lunch program (Sirin 2005), or as aggregated from student SES....

[...]

...School SES is often measured in two ways, either as the proportion of students enrolled in a reduced-price or free lunch program (Sirin 2005), or as...

[...]

1,482 citations

### "A Partial Simulation Study of Phant..." refers background in this paper

...Scholars showed that student SES positively significant correlated with student academic achievement (e.g., Sirin, 2005; White, 1982)....

[...]

...Willms (2010) showed a similar case where after controlling...

[...]

##### Related Papers (5)

[...]