STEMscopes Science School Parishes in Louisiana Outperform on 2021 LEAP

This current study was designed to be consistent with a Level 2 rating for the Every Student Succeeds Act (ESSA) guidance for evidence-based research. The study uses a quasi-experimental design with baseline equivalence established between the treatment and control groups. For the analysis, a standardized outcome measure was used, and baseline achievement was included as a covariate. Findings indicate that school usage of STEMscopes Science (Jefferson Parish + 5 East Baton Rouge schools) resulted in a higher percentage of students meeting mastery or above on the Louisiana Education Assessment Program (LEAP 25) Science test compared separately to two control districts with similar schools and students who did not use STEMscopes Science.

The purpose of this study was to estimate the impact of using the STEMscopes Science Curriculum on science achievement as assessed by state standardized testing. Particularly, the research was focused on one primary research question: What is the impact of the STEMscopes Science Curriculum on the percentage of students in a school who demonstrate mastery or above on the science test of the Louisiana Education Assessment Program (LEAP 25)? Results were analyzed for schools in Jefferson Parish, LA; Caddo Parish, LA; and East Baton Rouge Parish, LA. Caddo Parish uses Ph.D. Science from Great Minds as their core science curriculum program in 3rd-5th grade and IQWST from Activate Learning in 6th-8th grade. Apart from five East Baton Rouge (EBR) Parish schools that use STEMscopes Science, EBR uses Amplify as their core science curriculum program. Comparisons were made overall as well as by grade level between 3rd and 8th grade.

Included Schools

Jefferson Parish is the largest district in Louisiana, serving over 48,000 students from diverse racial and ethnic backgrounds, and over 80% are considered as economically disadvantaged. This makes them unique compared to the many smaller districts and school systems in Louisiana that serve more affluent student populations or students with similar racial/ethnic heritages. In the 2020-2021 school year, 57 of Jefferson Parish’s 66 schools that reported LEAP 25 data used the STEMscopes Science curriculum.

We compared Jefferson Parish to two school districts: Caddo Parish and East Baton Rouge. Both Caddo and East Baton Rouge are large parishes that also serve diverse and economically disadvantaged populations. Caddo Parish included 44 schools that reported LEAP 25 data, and EBR included 67 (5 STEMscopes) schools with LEAP 25 data. Caddo Parish serves a large and diverse student body, including over 36,000 students, with over 70% considered economically disadvantaged. East Baton Rouge Parish serves over 40,000 students, with over 78% considered economically disadvantaged.

To ensure districts were similar, we compared Jefferson Parish to Caddo Parish and East Baton Rouge Parish schools, respectively, with LEAP 25 data on: 1) 2019 LEAP 25 Science percent passing as a measure of baseline science achievement, 2) school total student enrollment, 3) school reported percentage of the student body who was considered a racial/ethnic minority, and 4) school reported percent of students considered economically disadvantaged. In all cases, Jefferson Parish did not significantly differ from Caddo Parish. Jefferson Parish did, however, differ from East Baton Rouge on the school-reported percentage of students considered a minority, with EBR serving a higher percentage of minority students. Importantly, there were no significant baseline differences in any other measured background variables, including the 2019 LEAP Science percent passing. This establishes baseline equivalence in prior school-level science achievement.

We then used multiple regression analyses to estimate the impact of STEMscopes Science Curriculum with full information maximum likelihood estimation to handle any missing data. Our outcome variable is the percentage of students in a given school that reached Louisiana’s passing standard of “mastery or higher” on the LEAP 25 science assessment. Please note that initially, the 2021 LEAP 25 outcome variable included score outliers that affected the distribution of the data. Analyses were conducted with both a negative binomial correction (all cases) and with outliers removed (which also corrected the distribution to normal as needed in regression analyses without a distribution correction). Both analyses resulted in similar results— we present estimates from both analyses in the Tables below for full comparison. Baseline science achievement (2019 percent passing), percent of students classified as racial/ethnic minorities, and percent of students considered economically disadvantaged were included in models as covariates (WWC, 2017). To indicate the size of STEMScopes Science Curriculum impacts, effect sizes were computed for all comparisons using Hedge’s g and odds-ratios (i.e., How many times more likely schools that use STEMscopes include students who meet or exceed the mastery benchmark) are reported for the analyses using binomial distributions. Models evaluated overall 3-8 grade level LEAP 25 2021 Science percent passing, as well as individual grade percent passing.

Results

Despite school level science passing rates that did not significantly differ in 2019, the Jefferson Parish schools (+ 5 EBR schools) using STEMscopes Science Curriculum outperformed Caddo Parish and East Baton Rouge Parish schools (that did not use STEMscopes) on the 2021 LEAP 25 science achievement test. The percent of students achieving mastery or above was significantly higher for STEMscopes schools compared to Caddo Parish schools for students in 4th - 7th grade, and for the overall 3-8 grade science passing rate, see Figure 1 and Table 1. The overall adjusted 3-8 grade “percent passing” difference between Jefferson Parish + 5 EBR (i.e., STEMscopes) schools and Caddo Parish (control group schools) was +7.53%. Put another way, schools using STEMscopes were 1.49 times more likely to have students who scored mastery or above compared to non-STEMscopes schools in Caddo Parish. Across grades, the effect sizes, as measured by Hedge’s g (see Table 1), ranged from 0.45 to 1.00. Odds-ratios ranged from 0.94 - 3.36.

Likewise, the percent of students achieving mastery or above was significantly higher for schools using STEMscopes Science compared to East Baton Rouge Parish schools that did not use STEMscopes Science for students in 4th - 6th grade, and for the overall 3-8 grade science passing rate, see Figure 1 and Table 2. The overall 3-8 grade adjusted “percent passing” difference between Jefferson Parish (and EBR) STEMscopes schools and EBR non-STEMscopes schools was 2.43%. Put another way, schools using STEMscopes were 1.16 times more likely to have students who scored mastery or above compared to non-STEMscopes schools in East Baton Rouge Parish. Across grades, the effect sizes, as measured by Hedge’s g (see Table 1), ranged from 0.07 to 0.43. Odds-ratios ranged from 1.04 - 1.79.

TABLE 1

ANALYSIS RESULTS FOR STEMSCOPES SCIENCE SCHOOLS COMPARED TO NON-STEMSCOPES SCHOOLS IN CADDO PARISH, LA IN LEAP 25 SCIENCE ACHIEVEMENT, GRADES 3-8

Grade	Group	Schools	Adj. LEAP 25% Passing Mean	Adj. Mean Diff.*	Effect Size**	Odds Ratio* (CI95)**	p-value For Odds Ratio
3rd	Control	37	5.36	7.32	1.00	0.94 (0.54 - 1.67)	0.85
	STEMscopes	50	12.68
4th	Control	37	14.71	10.26	0.67	2.34 (1.70 - 3.23)	<.001
	STEMscopes	50	24.97
5th	Control	37	16.23	10.84	0.64	2.22 (1.54 - 3.19)	<.001
	STEMscopes	50	27.07
6th	Control	17	12.30	10.08	0.80	3.36 (2.16 - 5.32)	<.001
	STEMscopes	34	22.38
7th	Control	15	17.15	5.88	0.45	2.56 (1.69 - 3.94)	<.001
	STEMscopes	34	23.03
8th	Control	15	11.66	7.44	0.60	2.14 (0.88 - 5.48)	0.06
	STEMscopes	33	19.10
3-8	Control	44	15.43	7.53	0.71	1.49 (1.24 - 1.79)	<.001
	STEMscopes	62	22.96

*Adj. Mean Diff. (SE) = the adjusted mean difference and standard error when controlling for the influence of standardized 2019 passing rate, as well as school percent of minority and economically disadvantaged students. **Effect sizes are how scientists measure the strength of an educational intervention or curriculum. While there are many ways to evaluate effect sizes, Hedge’s g is a widely used method for quantifying the differences in the means or averages between two groups, measured in standard deviations. Hedge’s g is a lot like Cohen's d, but it is better for smaller sample sizes. The larger the Hedges g value, the greater the effect. Effects under 0.25 are considered small, between 0.25 - 0.75 are considered medium and 0.76 and above are considered large in education, psychology, and other similar fields measuring human traits, abilities, scores, and behaviors. ***An odds ratio is the best measure of effect size when the analysis includes a negative binomial distribution to correct for non-normal data; CI95 = 95% confidence interval. In 3rd grade, outliers affected results. Although the odd-ratio was not significant, the adjusted means was significant p < .01.

Table 2

ANALYSIS RESULTS FOR STEMSCOPES SCIENCE SCHOOLS COMPARED TO NON-STEMSCOPES SCHOOLS IN EAST BATON ROUGE PARISH, LA IN LEAP 25 SCIENCE ACHIEVEMENT

Grade	Group	Schools	Adj. LEAP 25% Passing Mean	Adj. Mean Diff.*	Effect Size**	Odds Ratio* (CI95))**	p-value For Odds Ratio
3rd	Control	51	13.62	-0.56	0.07	1.04 (0.78 - 1.38)	0.78
	STEMscopes	45	13.06
4th	Control	51	18.83	3.75	0.24	1.47 (1.11 - 1.94)	<.01
	STEMscopes	45	22.58
5th	Control	51	19.27	5.12	0.34	1.51 (1.13 - 2.04)	<.01
	STEMscopes	45	24.29
6th	Control	25	16.51	6.03	0.43	1.79 (1.23 - 2.63)	<.01
	STEMscopes	33	22.54
7th	Control	22	21.75	3.89	0.26	1.19 (0.85 - 1.65)	0.33
	STEMscopes	33	25.64
8th	Control	22	16.42	3.90	0.30	1.26 (0.66 - 2.53)	0.45
	STEMscopes	32	20.80
3-8	Control	67	19.57	2.43	0.20	1.16 (1.01 - 1.34)	<.05*
	STEMscopes	57	22.00

*Adj. Mean Diff. (SE) = the adjusted mean difference and standard error when controlling for the influence of standardized 2019 passing rate, as well as school percent of minority and economically disadvantaged students. **Effect sizes are how scientists measure the strength of an educational intervention or curriculum. While there are many ways to evaluate effect sizes, Hedge’s g is a widely used method for quantifying the differences in the means or averages between two groups, measured in standard deviations. Hedge’s g is a lot like Cohen's d, but it is better for smaller sample sizes. The larger the Hedges g value, the greater the effect. Effects under 0.25 are considered small, between 0.25 - 0.75 are considered medium and 0.76 and above are considered large in education, psychology, and other similar fields measuring human traits, abilities, scores, and behaviors. ***An odds ratio is the best measure of effect size when the analysis includes a negative binomial distribution to correct for non-normal data; CI95 = 95% confidence interval.

Conclusion

In conclusion, the findings suggest that using the STEMscopes Science curriculum resulted in a higher percent of students passing the 2021 LEAP 25 science achievement test in Jefferson Parish and 5 EBR schools compared to Caddo Parish and East Baton Rouge Parish schools that did not use STEMscopes. Jefferson Parish, Caddo Parish and East Baton Rouge Parish have similar percentages of students who are economically disadvantaged. Although East Baton Rouge has significantly more students classified as a racial/ethnic minority than Jefferson Parish, Caddo has similar and the effect of this background characteristic was accounted for in all analyses along percent of economically disadvantaged students. Importantly (and included in analyses), participating Jefferson Parish, East Baton Rouge, and Caddo Parish schools had similar 2019 LEAP science achievement scores. However, in 2021 the percent of students who achieved mastery or above in the Jefferson Parish and East Baton Rouge schools that used STEMscopes Science was statistically significantly higher than Caddo parish and East Baton Rouge non-STEMscopes schools across 3-8 grade, and specifically in grades 4, 5, and 6 (and 7th in Caddo Parish as well). Of note, there were fewer available schools that included 7th and 8th grades in the analyses, and this smaller sample size may have affected the significance of findings. Yet— across all grades, the effect sizes and odd-ratios, taken together, generally show benefit to STEMscopes Science curriculum adoption and implementation, again, with largest effects in grades 4, 5, and 6. Results also tended to be larger for STEMscopes schools compared to Caddo Parish with schools in some grades two to three times more likely to have students that reached mastery and above compared to Caddo Parish students in those same grades. Because this study includes: 1) multiple school sites with over 350 students total, 2) a matched control group that had similar 2019 (baseline) LEAP 25 science achievement scores (and similar economic disadvantage and similar sized schools), and 3) positive significant effects demonstrated for the STEMscopes Science Curriculum, it yields evidence that is consistent with ESSA Level 2.

Louisiana Parish that uses STEMscopes Science Outperforms Two Other Parishes on 2021 LEAP