Background

The current report focuses on evidence that the STEMscopes Science curriculum effectively raises student science achievement. We use a post-facto quasi-experimental design (QED) with a matched control group to evaluate the potential associations between STEMscopes Science and science achievement for 5th and 8th graders in Illinois. QEDs with a matching attempt to overcome the hurdle of “non-random” assignment. In this case, the schools that chose to purchase and use STEMscopes Science may be different in some way (e.g., serve different student populations) than the schools that did not choose to use STEMscopes Science during the 2021-2022 school year. Our primary analyses focus on whether schools that use STEMscopes Science have a higher percentage of students who are ‘proficient and above’ on the 2022 Illinois Science Assessment (ISA) than schools that use some other science curricula. The ISA includes four performance levels: emerging, developing, proficient, and exemplary. We also include follow-up analyses focused on sub-group findings.

Results

We conducted multiple regression analyses to examine the impact of STEMscopes Science on increasing ISA school performance. Our 5th grade matched sample included 116 schools. Our 8th grade matched sample included 88 matched schools. Our primary analyses focused on predicting the percent of students who were proficient or above on the 2022 Illinois Science Assessment (ISA) test with a binary variable indicating whether a school was a STEMscopes or non-STEMscopes school.

5th Grade. In 5th grade, the main effect of STEMscopes, including all student scores, although positive, was not significant (non-STEMscopes: 57.16%, STEMscopes: 59.12%, b = 1.96, p = 0.39, ES = 0.08). However, results were significant for economically disadvantaged students in STEMscopes schools (non-STEMscopes: 39.47, STEMscopes: 45.78, b = 6.31, p < .05, ES = 0.48) as compared to economically disadvantaged students in non-STEMscopes schools, as well as for students with an Individualized Education Plan (non-STEMscopes: 27.13, STEMscopes: 35.24, b = 8.11, p < .05, ES = 0.43), see Figure 1. No other subgroup analyses were significant.

Figure 1

In 8th grade, on average, schools that purchased and used STEMscopes had a 5.97-point increase in the percent of students (all) who were proficient or above on the 2022 ISA (non-STEMscopes: 49.77%, STEMscopes: 55.74%; B = 5.94, p < 0.05, ES = 0.30); see Figure 2.

Figure 2

In addition to the main analyses, we included several follow-up analyses focused on proficiency rates among different sub-groups of students. There were significant findings among economically disadvantaged students (B = 6.75, p < .05, ES =0.37) such that economically disadvantaged students in STEMscopes Science schools outperformed economically disadvantaged students in non-STEMscopes schools. There were also significant findings for English Language Learners in STEMscopes versus non-STEMscopes schools (B = 6.71, p < .05, ES = 0.65) and Black./African American students in STEMscopes versus non-STEMscopes schools (B = 8.07, p = 0.05, ES = 0.42). No other subgroup analyses were significant.

Methods

In this section, details are provided about study procedures including the data sources, variables used, and participating districts

Data Sources

Data for this study came from two sources. First, schools that used STEMscopes Science for 5th and 8th grade in the 2021 - 2022 school year were identified through the STEMscopes analytics platform and Sales Force systems. Within the analytics reports, we used the unique number of 5th and 8th grade scopes accessed as a metric of use, and then confirmed usage with our internal Sales Force reports (to rule out schools that had free trial usage). If a school demonstrated use of at least 5 scopes in 5th or 8th grade science scope usage, they were considered a STEMscopes Science school. As a note, scopes are similar to a unit. The scope restriction was applied as a cleaning measure. The analytics system does not track whether a school was given access to STEMscopes via free trial, nor does it track whether a teacher/student from a different grade level accessed the scope (e.g., if a 6th grade teacher used a 5th grade scope as a review of previous knowledge). An internal study suggested that when a school uses 5 or more scopes in a given grade, this eliminated over 90% of cases that were out of grade level usage, thus ensuring that schools were appropriately included.

Second, school demographic data and school performance on the Illinois Science Assessment (ISA) were accessed through the Illinois State Board of Education website. For 5th grade, we used the previous 4th grade Illinois Assessment of Readiness (IAR) math data file (spring 2021 results) and focused on schools’ percent of students who 'met expectations’ on IAR math as a baseline measure of academic achievement. Specifically, the state of Illinois sets 5 levels of math achievement: did not meet expectations, partially met expectations, approached expectations, met expectations, exceeded expectations. We use 4th grade math as a proxy for prior achievement (as the Illinois Science Assessment does not have 4th grade level results). For 8th grade, we used the spring 2021 7th grade percent of students who met expectations on the IAR math test (by school) as a baseline measure of prior academic achievement.

We also downloaded 2021-2022 school year school fall enrollment data, including enrollment data by race/ethnicity, total enrollment, and enrollment by special populations including economically disadvantaged students and English language learners. All count data was then converted to percentage data by school (e.g., number of economically disadvantaged students/ total number of students in a school). These variables were used to match STEMscopes and non-STEMscopes schools (see participants section below for details on matching). Once matching was complete, we downloaded the 2023 ISA data file. This file includes a column for all students who are proficient or above as well as sub-population percent proficient and above columns. It is organized by grade.

Participants

In the 2021-2022 school year, the overall number of public Illinois schools that used STEMscopes for 5th grade was 60, and 45 for 8th grade (STEMscopes is also used by numerous private and parochial schools in Illinois). Overall, 2,039 regular and/or public charter schools may have contributed 2022 5th grade ISA data and 2,043 schools that may have 8th grade scores. There was missingness in all publicly available state files; please see missingness section below; thus, this number is the number of schools who submitted data but not necessarily scores.

Missing data

As a measure of privacy, state data does not include a numeric value for any variable where less than 10 students contributed data. This led to missing data (by design- this means we know what caused the missingness) with variables that included fewer than 10 students not reporting numbers. In addition to not reporting categories with 10 or fewer, for race/ethnicity data, the state of Illinois often masks two columns to discourage “figuring out” missingness based on the other categories. To account for missing data in the covariates, we used multiple imputation by chained equations (MICE). We use the ‘mice’ package in R (5 imputations, 20 iterations per imputation). Baseline math scores, race/ethnicity percentages, ell percentages, and school size were used in both matching analyses.

Matching

To match schools based on the data available from the Illinois State Board of Education, we matched as closely as possible across nine school demographic and achievement variables, including 2021 IAR math scores, school size, the percentage of students that were classified as economically disadvantaged, the percent of ELL students, and the percent of students in a school across race/ethnicity categories (i.e., Asian, Black/African American, Hispanic/Latinx, White/Caucasian, and Two or more race/ethnicities). We used the ‘Match-it” package in R with Mahalanobis Distance matching. When matching, Mahalanobis Distance is designed to consider the multivariate space between numerous covariates. Specifically, rather than propensity scores, Mahalanobis Distance (Gu & Rosenbaum, 1993) is used as the distance metric, and it is considered both more robust to multiple covariate usage and correlated covariates. Finally, using Mahalanobis distance in Match-it has the added benefit of prespecifying an allowable multivariate distance between matched school pairs. In this case, we used a multivariate distance of 0.10. By setting a pre-specified distance, the program will not return school pairs that are too dissimilar to a degree outside this distance. Matching with the Mahalanobis Distance method resulted in 58/60 5th grade STEMscopes Science schools matched, resulting in a total sample of 116, and 44/45 8th grade STEMscopes Science schools matched, resulting in a total sample of 88 schools. There were no significant differences between matched groups for all covariate variables (the variables used for matching), including baseline math performance.

Planned analyses

Substantive analyses were conducted with R-studio’s Lavaan structural equation modeling package because this package includes estimation with full information maximum likelihood (FIML) to handle missing data. Specifically, as mentioned above, as a measure of privacy, state data does not include a numeric value for any variable where less than ten students contributed data. This led to missing data in the ISA outcome data. We used FIML procedures to handle missing data estimation to ensure that the final analysis was not biased by this missing outcome data.