The following report includes results comparing STEMscopes and non-STEMscopes districts on the science component of the 2020-2021 Next Generation Science Assessment (NGSA) for 5th grade. Districts were identified as STEMscopes districts if they had a subscription to STEMscopes for the students in the tested grades (i.e., 5th grade) and showed usage of STEMscopes based on analytic data. The state of Rhode Island creates proficiency benchmarks in science and identifies students as “beginning to meet expectations,” “approaching expectations,” “meeting expectations,” and “exceeds expectations.” The percent of students in each of these categories is used to determine the district’s achievement in science. The percentage of students who “meet and exceed” expectations is used by the state as the district passing rate. The 2020-2021 school year also occurred during the global Covid-19 pandemic. Many Rhode Island students began the school year remotely, or used a “hybrid” model with both in-person and remote learning instruction. Also, it is important to note that even though districts tested students on the NGSA in 2021, they received a federal waiver such that not as many students per district tested as in previous years.

Elementary School NGSA Results

The state average passing rate for all Rhode Island districts that include 5th grade (N = 48) was 30.4%. Of these districts, eight districts (16.7%) used the STEMscopes Science curriculum and 40 districts (83.3%) used either a district-created science curriculum or purchased a different science curriculum. Before evaluating group differences between the STEMscopes and non-STEMscopes districts, we tested whether districts differed in background characteristics to ensure we were making an equitable comparison. Specifically, we tested for differences in previous levels of district science achievement (based on the 2019 NGSA passing rate), whether the district was a Charter district, percent of students who received “in-person instruction,” percent of tested students (as this was lower than previous years), and several student demographic variables including percentages of students with different race/ethnicities, percent of economically disadvantaged students, and percent of students classified as English Language Learners. As can be seen in Table 1 (below), there was one statistically significant background difference: We found that fewer African Americans were served in STEMscopes districts than non-STEMscopes districts, thus this variable was included in all analyses. Please note no other differences between districts were found; of particular importance, we found STEMscopes and non-STEMscopes baseline science achievement in 2019 did not significantly differ. Please note we include several covariates in all models to adequately control for important predictors that can affect science achievement. Next, we evaluated correlations between the potential baseline predictors. We found that several predictors were very highly correlated (with correlations greater than 0.80). These could not be included in the model together (predictors that are too highly correlated, can bias model results). These included: economic disadvantage and baseline 2019 science achievement, Latino/Hispanic students and percent of students who spoke English as a second language; percent of Latino/Hispanic students and whether the district was a Charter, and the percent of White/Caucasian students was highly correlated with the 2019 passing rate and percent Latino/Hispanic students as well. With this in mind, we excluded economic disadvantage, English language learners, percent of White/Caucasian students, and whether a district was a Charter. We included baseline science and percent Latino/Hispanic (along with the other above variable) as this allowed us to capture variance related to demographics in the fewest number of variables and without too high of correlations among predictors.

Analysis on Elementary Results

We used multiple regression analysis to calculate potential differences in the NGSA district level passing rates between STEMscopes and non-STEMscopes districts, controlling for other important variables. This provides a stringent test of the effect of STEMscopes. Or, put another way, this provides a weighted effect. Initial results, although numerically favoring STEMscopes districts with an estimated 1.63 percent increase in science passing rate compared to non-STEMscopes districts, were not significant (p = 0.65). However, following Rhode Island’s example when they evaluated math and reading scores (see here), we added an “interaction” term— that is, we tested whether the effect of STEMscopes varied based on the percent of students in a district who received in—person instruction. Results are presented in Table 2 and Figure 1 at right. The results indicated a significant interaction. The means the “effect of STEMscopes” varies based on the percent of students in a district that were in—person. Specifically, if you look at figure 1, non-STEMscopes students had similar science achievement levels across different rates of “percent in—person” (between ~33 - 37% passing). However, for STEMscopes districts, we see a marked change in achievement depending on the percent of students who learned in person within a district. Specifically, the average district included 53% of students with in—person instruction. If we estimate the effect of STEMscopes at this point, again, results numerically favor STEMscopes (with a 2.73% increase in the number of students passing compared to non-STEMscopes districts), but are not significant. Yet if we look at districts with 85% of students learning in—person, STEMscopes is estimated to increase the district passing rate by 7.22% compared to non-STEMscopes districts with 85% of in—person students, and this is a significant difference (p<.05). This data suggests STEMscopes is an effective in—person curriculum.

FIGURE 1 WEIGHTED 5TH GRADE SCIENCE PROFICIENCY FOR STEMSCOPES AND NON-STEMSCOPES SCIENCE DISTRICTS ACROSS DIFFERENT DISTRICT “IN-PERSON INSTRUCTION” PERCENTAGES

 

Conclusion

STEMscopes is an effective science curriculum for in-person instruction. In Rhode Island, if students were mostly learning in-person (defined at 85% in-person) AND were a STEMscopes district, the average passing rate was 42.66% compared to 36.81% in non-STEMscopes districts. These findings provide a stringent test of STEMscopes effectiveness as several background covariates that are known to affect science achievement were included. We interpret data cautiously given the effects of the global Covid-19 pandemic on instruction (as this was the reason that not all students were “in-person” and the reason that the number of students tested was lower than usual within districts). We also interpret these findings optimistically because STEMscopes was designed to be an in-person, hands-on science curriculum. When students are learning in the classroom, these findings indicate that using the STEMscopes Science curriculum increases the number of students in a district that “meet and exceed” Science learning expectations compared to not using the STEMscopes Science.