Background

This report details the findings from an efficacy study on the STEMscopes Science 8th grade curriculum in Oregon. We used a post-facto quasi-experimental design (QED) that utilizes a matched control group to evaluate the potential associations between STEMscopes Science and science achievement for 8th graders in middle schools across Oregon. The evidence in this study is consistent with the “Every Student Succeeds Act’s (ESSA) Tier Two evidence." QEDs with matched samples attempt to overcome the hurdle of “non-random” assignment.  

In addition to examining the potential relationship between STEMscopes Science implementation and science achievement for all 8th grade students in the study, we also considered achievement in several sub-groups of students. Specifically, previous results in the field (Morgan, Farkas, Hillemeier, & Maczuga, 2016) suggest there are science achievement gaps among students who are considered minority groups relative to their non-minority peers, as well as achievement gaps related to whether a student is receiving special education services or not. However, past STEMscopes reports suggest that STEMscopes has a positive impact, often regardless of sub-population membership. 

Thus, within the current report we considered: 1) potential group differences in science achievement (operationalized by the percent of students in a given school who pass their science standardized test) for STEMscopes versus non-STEMscopes schools in middle schools across Oregon. Overall, we hypothesized that even with the more stringent matched control group QED design, schools that purchased and used STEMscopes during the 2021-2022 school year would include a higher percent of students, on average, that pass on the 2022 Oregon Statewide Assessment System (OSAS) Science Assessment than schools that did not purchase and use STEMscopes (i.e., “non-STEMscopes schools”). 2) We also anticipated significant associations between STEMscopes Science and school 8th grade passing rates on the OSAS science test among subgroups of students. 

Results

To examine the effectiveness of STEMscopes Science in increasing Oregon schools 8th grade passing rates on the OSAS Science assessment, we conducted multi-level multiple regression analyses with 136 matched schools from districts across Oregon. Our first analysis focused on predicting 2022 OSAS 8th grade school passing rates (outcome) with a binary variable indicating whether a school was a STEMscopes school or non-STEMscopes school, and covariates (see methods), while accounting for the fact that schools are clustered within districts. Results were significant (non-STEMscopes M = 26.95, STEMscopes M = 30.93, b = 3.98 p < 0.05, ES = 0.22). See Figure 1 for a comparison of rates between STEMscopes schools and non-STEMscopes schools and Table 1 for model parameters. In addition, there were also significant findings among sub-groups of students such that: among students classified as receiving special education services, special education students in STEMscopes schools made greater gains than special education students in non-STEMscopes schools (non-STEMscopes M = 8.09, STEMscopes M = 11.11, b = 3.02, p<.05, ES = 0.24).  There was also a significant finding among Hispanic/Latinx students (non-STEMscopes M = 14.57, STEMscopes M = 17.81, b =3.24, p<.05, ES =0.28). These findings suggest that STEMscopes is also effective across sub-populations of students, as well as when all students are considered. 

Methods

In this section, details are provided about study procedures including the data sources, variables used, and participating schools.

Data sources

Data for this study came from two sources. First, schools that purchased and used STEMscopes for 8th grade in the 2021 - 2022 school year were identified through the STEMscopes analytics platform. Within the analytics reports, we used the number of 8th grade scopes (or units) accessed as a metric of use. Any middle school that had accessed 8th grade units was included in the current study.   

Second, school demographic data and school performance on the 2018-2019 OSAS were accessed through the Oregon Department of Education website. We used the 2018-2019 OSAS school summary data for 8th grade and focused on the school level “meets and exceeds” on the OSAS mathematics test as a baseline measure of academic achievement. Specifically, the state of Oregon created 4 proficiency benchmarks in all academic content and identifies students as “Below,” “Near,” “Meets” and “Exceeds” standards. The percentage of students who are classified as meeting or exceeding the grade level standard is used by the state as the school passing rate. We use the 2018-2019 math school passing rate for 8th grade because the state of Oregon asked for a testing waiver during the 2020-2021 school year related to the Covid-19 global pandemic. Due to this, substantially fewer students were tested. Since the math and science components of the OSAS assessment correlate highly (r = 0.78 in 2018-2019 5th grade), this is an appropriate way to ensure that there were no baseline differences across STEMscopes and non-STEMscopes matched schools in prior academic achievement (notably a subset of the STEMscopes schools were already using STEMscopes prior to the 2018-2019 year and thus using the science assessment would not give a true baseline).

We also downloaded 2021-2022 school enrollment data, including race/ethnicity percentages data, student group data, including percentage data for students considered economically disadvantaged, English Language Learners (ELL), and students who were eligible to receive special education services. We focused on these covariates as previous research has indicated they are associated with science achievement; however, numerous schools were missing ELL information (over 50%), thus this variable was not ultimately used in analyses due to too much missing data. The percent economical disadvantaged and eligible for special education as well as the race/ethnicity, total enrollment variables, and whether the school was a public charter school or not, were used to match STEMscopes and non-STEMscopes schools (see participants section below for details on matching). Once matching was complete and baseline analyses were conducted (see baseline equivalence), we downloaded the 2022 OSAS Science assessment student group file at the school level. We used the percent of students in each school who “met and/or exceeded” grade level standard (passing rate) on the spring 2022 OSAS as the outcome variable, with different outcome variables representing all students and different sub-populations.  

Missing Data
If a variable (e.g., program category/sub-population) had less than 10 students, the data was removed by the state in order to protect student privacy under the Family Educational Rights and Privacy Act (FERPA). This led to missingness by design in both covariate and outcome variables. We handled this missing data in two ways. For any covariate variable used to match schools, we used multiple imputation by chained equations via R-studio’s “MICE” package with the “CART” imputation method (see Van Buuren and Groothuis-Oudshoorn, 2011). We use MICE procedures during this step to ensure complete data for matching procedures via R-Studio's “Match-it” package, although few covariates included missing data (math passing rate [13%], percent of economically disadvantaged students [4%], and percent of special education students [16%]). Once data were matched, in all final analyses, we used R-Studio’s “Lavaan” package, which uses full information maximum likelihood procedures to handle missing outcome data. Many of the student sub-populations had extensive missing data. We only ran analyses among sub-population data that included over 50% of the sample. This excludes most of the race/ethnicity variables (see Table 2).  

Participants
In the 2021-2022 school year, the overall number of Oregon middle schools that purchased and used STEMscopes (in any capacity) for 8th grade was 68 out of 405 possible schools (~16.8%). As mentioned previously, to match schools, we used R-Studio’s “Match-it” package with the “nearest neighbor” method. We included 10 school-level variables to match data: 5 race/ethnicity variables representing the percent of the school population that fit each category: White/Caucasian, Hispanic/Latinx, Black/African American, Asian, and two or more races/ethnicities; as well as the total number of students enrolled in a school, the percent of students that qualified for special education, the percent of economically disadvantaged students for a given school and a binary variable indicating whether or not the school was a public charter school. Finally, schools were matched based on ‘baseline’ academic achievement as indexed by the OSAS 2018-2019 8th grade math passing rate for a given school. All 68 STEMscopes schools were matched, resulting in a final sample size of 136 schools. 

Baseline Equivalence

For all covariate variables (the variables used for matching), including baseline academic achievement, there were no significant differences between matched groups (see Table 2). However, the What Works Clearinghouse (WWC) standards require that baseline differences with a Hedge’s G effect size greater than 0.05 must be controlled for statistically. Following the advice of Stuart, 2010, we included all covariates that were not collinear in the final analyses as a complementary approach to matching and a more stringent test of effects. Several race/ethnicity variables were correlated at or above -/+0.70. Specifically, the percent of students that were Black/African American was highly correlated with the percent of students who reported being Asian (r = 0.90). The percent of students that were Hispanic/Latinx was highly correlated with the percent of students who were White/Causcasian (r = -0.93). Finally, the percent of economically disadvantaged students was highly correlated with the percent of students eligible for special education services (r = 0.80). With this in mind, we present models above excluding these highly correlated variables (however, we re-ran models that separately included each of race/ethnicity and special education percent variables along with the other not collinear covariates, and the pattern of findings was the same). Inclusion of these covariates satisfies the WWC standard, as several variables had effect sizes greater than 0.05.  

Work Cited

Brydges, C. R. (2019). Effect size guidelines, sample size calculations, and statistical power in gerontology. Innovation in aging, 3(4). 
Fraser-Abder, P., Atwater, M., & Lee, O. (2006). Research in urban science education: An essential journey. Journal of Research in Science Teaching, 43(7), 599–606. 
Morgan, P.L., Farkas, G., Hillemeier, M. M., and Maczuga, S. (2016). Science achievement gaps begin very early, persist, and are largely explained by modifiable factors. Educational Researcher, 45, 18-35. 
Stuart, E. A. (2010). Matching methods for causal inference: A review and a look forward. Statistical science: a review journal of the Institute of Mathematical Statistics, 25, 1.