We also downloaded 2021-2022 school year school enrollment data including race/ethnicity percentages data, student program data including percentage data for students considered economically disadvantaged, Limited English Proficient (LEP), students who were eligible to receive special education services, as well as gifted and talented program students. We focused on these covariates as previous research has indicated they are associated with science achievement. These variables were used to match STEMscopes and non-STEMscopes schools (see participants section below for details on matching). Once matching was complete and baseline analyses were conducted (see baseline equivalence), we downloaded the 2022 Georgia Milestones End-of-Course assessments by grade from the govenor’s office website. We used this third data source because files on the govenor’s website included score breakout by student sub- populations. We used the percent of students in each school who were “proficient and above” grade level standard (passing rate) on the spring 2022 Georgia Milestones as the outcome variable(s).

Missing Data
If a variable (e.g., program category/sub-population) had less than 10 students, the data was removed by the state and identified “TFS” (too few students), which is included instead in order to protect student privacy under the Family Educational Rights and Privacy Act (FERPA). This leads to missingness by design in both covariate and outcome variables. We handled this missing data in two ways. For any covariate variable used to match schools, we used multiple imputation by chained equations via R-studio’s “MICE” package with the “CART” imputation method (see Van Buuren and Groothuis-Oudshoorn, 2011). We use MICE procedures during this step to ensure complete data for matching procedures via R-Studio's “Match-it” package, although few covariates included missing data (math 2021 passing rate n = 23 [~12% of total unmatched sample], and percent of gifted and talented students n = 24 missing [~12%]). Once data were matched, in all final analyses we used R-Studio’s “Lavaan” package which uses full information maximum likelihood procedures to handle missing outcome data. Many of the student sub-populations had extensive missing data. We only ran analyses among sub-population data that included over 50% of the sample.

Participants
Before collecting data, we used the National Center of Education Statistics (NCES) website to classify “urban districts.” Specifically, we focused on districts that were in “midsize” or “large” cities via NCES classifications. Six districts fit this classification and these districts included 197 schools. In the 2021-2022 school year, the overall number of Georgia urban schools that purchased and used STEMscopes (in any capacity) for 5th grade was 86 out of 197 schools (~44%). Of these 86 schools, 72 met the criterion of schoolwide usage of at least 80% of 5th grade science units (scopes) accessed and were eligible for the study.

As mentioned previously, to match schools we used R-Studio’s “Match-it” package with the “nearest neighbor” method. We included 11 school level variables to match data: 5 race/ethnicity variables representing the percent of the school population that fit each category: White/Caucasian, Hispanic/Latinx, Black/African American, Asian, and two or more races/ethnicities; as well as the total number of students enrolled in a school, the percent of students that qualified for special education or gifted and talented programs, the percent of Limited English Proficieny students, and the percent of economically disadvantaged students. Finally, schools were matched based on ‘baseline’ academic achievement as indexed by the Georgia Milestones 2021 4th grade math passing rate for a given school. All 72 STEMscopes schools were matched, resulting in a final sample size of 144 urban schools.

Baseline Equivalence
For all covariate variables (the variables used for matching) including baseline academic achievement, there were no significant differences between matched groups (see Table 2). However, the What Works Clearinghouse (WWC) standards require that baseline differences with a Hedge’s G effect size greater than 0.05 must be controlled for statistically. Following the advice of Stuart, 2010, we included all covariates that were not collinear in the final analyses as a complementary approach to matching, and a more stringent test of effects. Several race/ethnicity variables were correlated at or above -/+0.70. Specifically, the percent of students that were Black/African American was highly negatively correlated both with the percent of students who reported being White/Caucasian (r = -0.91) and with the percent of students who reported two or more ethnicities (r = -0.70). The percent of students that was Hispanic/Latinx was highly correlated with the percent of students who reported being Limited English Proficient (r = 0.94). With this in mind, we present models above excluding these highly correlated variables (however, we re-ran models that separately included each of race/ethnicity percent variables along with the other not collinear covariates and the pattern of findings was the same). Inclusion of these covariates satisfies the WWC standard as several variables had effect sizes greater than 0.05.