Background

The effectiveness of the STEMscopes Science curriculum among California districts has previously been established. The current report adds to this evidence in several important ways. For 5th and 8th grade, it is the first report to focus on a post-facto quasi-experimental design (QED) that utilizes a matched control group to evaluate the potential associations between STEMscopes Science and science achievement for 5th and 8th graders in California. This type of evidence is consistent with the “Every Student Succeeds Act’s (ESSA) Tier Two evidence. QEDs with matching attempt to overcome the hurdle of “non-random” assignment. In this case, the districts that chose to purchase and use STEMscopes Science may be different in some way (e.g., serve different student populations) than the districts that did not choose to use STEMscopes Science during the 2021-2022 school year. In addition to the main analyses that focus on whether districts that use STEMscopes Science have higher student passing rates than districts that use some other science curriculum, we include several follow-up analyses focused on 5th grade. Specifically, we present sub-group findings and findings associated with districts using STEMscopes Science for multiple years. 

Results

To examine the effectiveness of STEMscopes Science to increase CAST district 5th and 8th grade passing rates, we conducted multiple regression analyses. Our 5th grade matched sample included 216 districts. Our 8th grade matched sample included 274 matched districts. Our main analyses focused on predicting 2022 CAST district passing rates (outcome) with a binary variable indicating whether a district was a STEMscopes district or non-STEMscopes district, and covariates (see methods). Results for both grades were significant. In 5th grade, on average, districts that purchased and used STEMcopes had a 1.58 increase in the percent of students passed (non-STEMscopes: 32.23%, STEMscopes: 33.81%; B = 1.58, p < 0.05, ES = 0.08), see Figure 1. We estimate that  approximately 897 additional students passed in STEMscopes districts *among the students who were tested (1.58% * 56,795 tested students in STEMscopes districts) relative to non-STEMscopes districts. In 8th grade, on average, districts that purchased and used STEMcopes had a 2.01 increase in the percent of students passed (non-STEMscopes: 29.69%, STEMscopes: 31.70%; B = 2.01, p < 0.05, ES = 0.14); see Figure 2. We estimate that approximately 2,415 additional 8th graders relative to the non-STEMscopes districts may have passed their 2022 CAST based on this weighted estimate. 

In addition to the main analyses, we included several follow-up analyses for 5th grade including an analysis of passing rates among different sub-groups of students. There were also significant findings such that: among economically disadvantaged students, there was a positive association between STEMscopes Science and CAST passing rates (B = 1.83, p < .05, ES = 0.10). There was also a significant finding among African American students (B = 4.76, p < .05, ES =0.25). Our last follow-up analysis for 5th grade was to evaluate the association between multiple years of STEMscopes usage by districts and their 2022 CAST passing rate. Specifically, controlling for all other covariates, for every additional year of STEMscopes use, CAST district passing rates increased an estimated 0.51 points (about ½ a percent a year every year the district uses STEMscopes). Overall, these findings are in-line with previous findings suggesting a positive association between STEMscopes Science curriculum use in 5th grade and a higher percentage of district students passing the 2022 CAST, with the longer a district uses STEMscopes predicting greater and greater passing rates.

Methods

In this section, we provide details about study procedures including the data sources, variables used, and participating districts. 

Data for this study came from two sources. First, schools that used STEMscopes for 5th and 8th grade in the 2021 - 2022 school year were identified through the STEMscopes analytics platform. Within the analytics reports, we used the number of 5th and 8th grade scopes accessed as a metric of use. If a district demonstrated any 5th or 8th grade scope usage within at least 50% of a district’s schools across the year, they were considered a STEMscopes district.

Second, school demographic data and school performance on the California Science Test were accessed through the California Assessment of Student Performance and Progress website and the California Department of Education website. We used the 2018 - 2019 CAST data file and focused on the district passing rate as a baseline measure of science achievement. Specifically, the state of California sets 4 levels of science achievement: grade standards not met, grade standards nearly met, grade standards met, and grade standards exceeded. The “passing rate” refers to the percent of students in a district that either “met” or “exceeded” grade level standards. We use the 2018-2019 district science passing rate because it occurred prior to the global Covid-19 pandemic. Due to the pandemic, standardized scores were not collected in the 2019-2020 school year, and those that were collected in 2020-2021 included extensive missingness as many districts did not submit scores and those that did often included far fewer tested students (which is also reported within the CAST data files). We also downloaded 2021-2022 school year school enrollment data, student ‘lunch status’ data (i.e., school percent of students eligible for free and reduced lunch) as an indicator of student economic disadvantage, percent of english language learners, and race/ethnicity count data. Data were cleaned such that counts were converted to percentages (e.g., number of students in a given racial category/ total number of students). These variables were used to match STEMscopes and non-STEMscopes districts (see participants section below for details on matching). Once matching was complete and baseline analyses were conducted (see participants), we downloaded the 2022 CAST data including subgroups and the number of students tested per district. We used the percent of students in each district who met or exceeded grade level standards on the spring 2022 CAST as the main outcome variable. 

In the 2021-2022 school year, the overall number of public California districts that used STEMscopes for 5th grade was 119 and 138 for 8th grade (STEMscopes is also used by numerous private and parochial schools in CA). Overall, the California Dept. of Ed. website reports up to 935 districts that may have 2022 5th and 8th grade scores for the district (345 Unified, 519 Elementary, and 81 ‘other’). The 2019 CAST baseline dataset included 931 districts that contributed 5th grade data; of these 803 had valid scores. For 8th grade, there were 896 districts that contributed data; of these 788 had valid scores. As a measure of privacy, state data does not include a numeric value for any variable where less than 10 students contributed data. This led to missing data (by design- this means we know what caused the missingness) with districts that included fewer than 10 students passing not reporting scores. Using the 2019 number of districts, we estimate districts that used STEMscopes in 5th grade represent  ~12.8% of California public districts that could include state testing data. This number increases to ~17.5%  of California schools using STEMscopes for 8th grade.

To match districts based on the data available from the California Department of Education, we matched as closely as possible across 8 school demographic and achievement variables including their 2019 percent meeting the CAST passing score, district size, the percent of students that were classified as economically disadvantaged (i.e., free and reduced lunch), and the percent of students in the district across race/ethnicity categories (i.e., Asian, Black/African American, Hispanic/Latinx, and White/Caucasian). Our final 5th grade sample included 218 matched districts (109 STEMscopes and 109 non-STEMscopes). Among the 5th grade STEMscopes districts, 33 districts used STEMscopes Science only during the 2021-2022 school year, 11 districts used STEMscopes for two years, 37 districts used STEMscopes for three years, and 28 districts used STEMscopes for four or more years. To denote this multi-year usage, we created a variable where non-STEMscopes districts = 0,  1 = one year, 2 = 2 years, 3 = 3 years and 4 = 4 years (hence the study focuses roughly on a four year time period of usage; we request readers also recall these four years corresponded in part with the global Covid-19 pandemic). The sample districts served a range of sizes and percentages of minority and economically disadvantaged students, please see Table 1 below for details. A similar process was used to match samples for 8th grade. Our final 8th grade sample included 274 matched districts (137 STEMscopes and 137 non-STEMscopes). The sample districts served a range of sizes and percentages of minority and economically disadvantaged students, please see Table 2 below for details. 

All analyses were conducted in R-studio. For all covariate variables (the variables used for matching) including baseline science performance, there are no significant differences between matched groups (see Table 1). However, the WWC standards require that baseline differences greater than 0.05 must be controlled for statistically. Following the advice of Stuart, 2010, we include all covariates (apart from collinear variables, please see below) in the final analyses as a complementary approach to matching, and a more stringent test of effects. This also satisfies the WWC standard as several variables had effect sizes greater than or equal to 0.05.]

Method continued

Analyses were conducted with R-studio’s Lavaan structural equation modeling package because this package includes estimation with full information maximum likelihood (FIML) to handle missing data. Specifically, as mentioned above, as a measure of privacy, state data does not include a numeric value for any variable where less than 10 students contributed data. This led to missing data. For example, the percent of districts who reported that their student population included students of Black/African American ethnicity was missing for 13 out of 216 or ~6% of districts. FIML procedures to handle missing data estimation ensure that in the final analysis where we include covariates, the estimation is not biased.

As a stringent test of the effects of STEMscopes, we include multiple covariates in all analyses including baseline 2019 CAST passing rate, district size, percent of economically disadvantaged students, percent of Black/African American students, percent of Asian students, and the percent of Latino/Hispanic students. Of note, in the 5th grade analysis we could not include the percent of White/Caucasian students, the percent of Latino/Hispanic students and the percent of ELLs in the same model as these covariates were highly correlated (r’s > 0.70), nor could we include these in the 8th grade analysis (r’s >0.80) because it would introduce multicollinearity to the model (but please note although we present models with percent Latino/Hispanic included in the results section as this variable has largest difference between the control group and the treatment group, we ran models that separately included percent White/Caucasian and/or percent ELL respectively and all models had similar results).