The following report includes results from a ESSA tier 2 post-facto quasi-experimental design main effects efficacy study comparing STEMscopes and non-STEMscopes districts on the 2020-2021 Mississippi high school end-of-course (EOC) exam in Biology and 2021 ACT science scores. Districts were identified as STEMscopes districts if they had a subscription to STEMscopes for the students in the tested grades (i.e., high school) and showed any usage of STEMscopes in that grade based on analytics data. The state of Mississippi creates benchmarks for proficiency in Biology and identifies students within 5 levels (minimal, basic, passing, proficient, and advanced). The percentage of students who were proficient or advanced in science was defined as the district's proficiency rate. The ACT science scores are from 0 - 36 and represent the district’s average score in the spring of 2021. The 2020-2021 school year also occurred during the global Covid-19 pandemic, with many students receiving virtual instruction.

The state average passing rate for all Mississippi school districts that include 5th grade scores (N = 140) was 47.4%, and the average ACT score was 17.6. Of these districts, 26 districts (19%) used the STEMscopes science curriculum, and the other 114 districts used either a district-created or another Biology curriculum. Before evaluating group differences between the STEMscopes and non-STEMscopes districts, we tested whether districts differed in background characteristics to ensure we were making an equitable comparison. Specifically, we tested for differences in previous levels of district biology achievement (based on the 2019 EOC passing rate) and prior 2020 ACT scores, percent average daily attendance in a district (percent of students present each day school was in session), percent of students tested by the district on the EOC as well as district size and graduation rates, percent of experienced teachers in a district (teachers with 4 or more years teaching experience), the percent of students served from different racial/ethnic backgrounds, and the percent of students in a district receiving special education support. As can be seen in Table 1, there were no significant differences between STEMscopes and non-STEMscopes districts when doing these comparisons; of particular importance, we found STEMscopes and non-STEMscopes baseline biology achievement in 2019 did not significantly differ. However, we only present our full model, statistically adjusted for covariates using full information maximum likelihood to account for missing covariate data. Including covariates accounts for their impact on science achievement. Please note the percent of Black/African American students and the percent of White/Caucasian students in a district are highly correlated r = -0.97, p < .01 indicating that as percent of African American students increases, the percent of White Caucasian students decreases and vice versa. Although both had significant differences, they could not be included in the model at the same time due to this multicollinearity (strong relatedness). We present below the model, including the percent Black/African American students covariate. Results were similar with percent White/Caucasian students included in the model. The below table demonstrates that STEMscopes and non-STEMscopes districts were similar to each other in aggregate background characteristics.

Follow-up Analysis on Elementary Results

We used two multiple regression analyses to calculate potential differences in the EOC Biology passing rates, and the ACT Science scores as predicted by districts being a STEMscopes or non-STEMscopes district. We also accounted for (included) 2018-2019 science passing rates on 2020 ACT scores (depending on analysis) as well as the percent experienced teachers by district, district size, students tested and graduation rate, and demographic information of students (e.g., race/ethnicity). By accounting for other important variables in the model, we provide a stringent test of the effect of STEMscopes. Put another way, this provides a weighted effect. Variables were standardized to aid interpretation. Results are presented in Tables 2 and 3 as well as Figure 1 below. The results indicate that once other important variables are accounted for, the districts that used STEMscopes have significantly higher overall EOC Biology proficiency rates compared to non-STEMscopes districts. Specifically, there is a weighted increase of 4.1% more high school students meeting the Mississippi Biology passing benchmark for STEMscopes versus non-STEMscopes districts. Although results were not significant for the ACT, they “trend” in the right direction. The “p-value” in Table 2 tells us how confident we feel about whether an association is true and trustworthy versus possibly occurring by chance. The typical p-value used by researchers is p ≤ .05 which means we are 95% confident that the association is not by chance. In this case, we are only 92% confident that the association is not by chance. In the case of the ACT, STEMscopes districts, on average, had ACT scores that were ⅓ of a point higher with non-STEMscopes districts having a weighted average of 16.93, while STEMscopes districts had an average of 17.26 points on average.