Background

The current report focuses on the efficacy of the STEMscopes Science 5th grade curriculum. We use a post-facto quasi-experimental design (QED) that utilizes a matched control group to evaluate the potential associations between STEMscopes Science and science achievement for 5th graders in Dallas ISD. This type of evidence is consistent with the “Every Student Succeeds Act’s (ESSA) Tier Two evidence. QEDs with matched samples attempt to overcome the hurdle of “non-random” assignment.

In addition to examining the potential relationship between STEMscopes Science implementation and science achievement for all students, we also consider achievement in several sub-groups of students. Specifically, previous results in the field (Morgan, Farkas, Hillemeier, & Maczuga, 2016) suggest science achievement gaps among students who are considered minority groups relative to their non-minority peers, as well as achievement gaps related to socio-economic status. However, past STEMscopes reports suggest that STEMscopes may have a stronger association with standardized test passing scores among minorities (particularly black/African American students and Latino/Hispanic students) as well as students considered economically disadvantaged, and English language learners (ELL). We evaluated the association between STEMscopes Science and district passing rates among several sub-groups of students: females & males, African Americans, Latino/Hispanic students, ELL and non-ELL students, and economically disadvantaged students.

Thus, within the current report we consider: 1) potential group differences in science achievement (operationalized by the percent of students in a given school who pass their science standardized test) for STEMscopes versus non-STEMscopes schools. Overall, we hypothesize that even with the more stringent matched control group QED design, schools that purchased and used STEMscopes during the 2021-2022 school year will include a higher percent of students, on average, that pass on the 2022 State of Texas Assessments of Academic Readiness (STAAR) than schools that did not purchase STEMscopes (i.e., “non-STEMscopes schools”). 2) We also anticipate significant associations between STEMscopes Science and school 5th grade passing rates on the STAAR among subgroups of students. 

Results

To examine the effectiveness of STEMscopes Science to increase STAAR district 5th grade passing rates, we conducted multiple regression analyses with 88 matched Dallas schools. Our first analysis focused on predicting 2022 STAAR 5th grade school passing rates (outcome) with a binary variable indicating whether a district was a STEMscopes district or non-STEMscopes districts, and covariates (see methods). Results were significant (non-STEMscopes M = 56.67, STEMscopes M = 62.87, b = 6.20, p < 0.05, ES = 0.36), see Figure 1 for comparison of rates between STEMscopes schools and non-STEMscopes schools and Table 1 for all model parameters.

In addition, there were also significant findings among sub-groups of students such that: among economically disadvantaged students, there was a positive association between STEMscopes Science and STAAR passing rates (non-STEMscopes M = 54.48, STEMscopes M = 60.80, b = 6.32, p = .05, ES = 0.34). We saw a similar pattern for English Language Learners such that there were significant school average passing rate differences between STEMscopes and non-STEMscopes schools in the 2022 STAAR passing rate for ELL students (non-STEMscopes M = 56.23, STEMscopes M = 64.34, b = 8.11, p < .05, ES = 0.40), but interestingly differences were not significant between schools (although numerically positive with a small effect) for non-ELL students (non-STEMscopes M = 56.21, STEMscopes M = 61.77, b = 5.56, p=0.13, ES = 0.25).

There was also a significant finding among African American students (non-STEMscopes M = 38.26, STEMscopes M = 51.33, b =13.07, p<.05, ES =0.40), but findings were “trend” for Hispanic/Latinx students (non-STEMscopes M = 57.94, STEMscopes M = 63.81, b = 5.87, p=0.06, ES = 0.30). The “p-value” tells us how confident we feel about whether an association is true and trustworthy versus possibly occurring by chance. The typical p-value used by researchers is p ≤ .05 which means we are 95% confident that the association is not by chance. In the case of Hispanic/Latinx students, we are only 94% confident that the association is not by chance. That said, this also offers some evidence that African American students in Dallas ISD schools that use STEMscopes are closing academic achievement gaps, at least relative to their Hispanic/Latinx peers. Finally there was a significant finding among females (non-STEMscopes M = 54.99, STEMscopes M = 62.25, b = 7.26, p<.05, ES = 0.36), but a trend level finding for males (non-STEMscopes M = 58.10, STEMscopes M = 63.67, b = 5.57, p=0.06, ES = 0.32). 

Methods

In this section, we provide details about study procedures including the data sources, variables used, and participating districts. 

Data sources
Data for this study came from two sources. First, schools that purchased and used STEMscopes for 5th grade in the 2021 - 2022 school year were identified through the STEMscopes analytics platform. Within the analytics reports, we used the number of 5th grade scopes accessed as a metric of use. We were interested specifically in schools that used STEMscopes Science in a full curriculum capacity. We defined a school as fully using the curriculum if they accessed at least 14 of the 18 available science units (called scopes) for 5th grade (~80%) of the curriculum.

Second, school demographic data and school performance on the State of Texas Assessments of Academic Readiness (STAAR) were accessed through the Texas Education Agency website. We used the 2020 - 2021 STAAR campus data file for 4th grade and focused on the school level “approaches grade level percent” on the STAAR mathematics test as a baseline measure of academic achievement. Specifically, the state of Texas creates proficiency benchmarks in all academic content and identifies students as not proficient, approaching grade-level proficiency, meeting grade-level proficiency, and mastering grade-level proficiency. The percentage of students who approach grade-level performance is used by the state as the district passing rate. We use the 2020-2021 math school passing rate for 4th grade because a STAAR science test is not administered in 4th grade. We wanted to ensure it was (approximately) the same students contributing scores to a school’s passing rate. Since the math and science components of the STAAR correlate highly (r = 0.85 in 2020-2021 5th grade), this is an appropriate way to ensure that there were no baseline differences across STEMscopes and non-STEMscopes matched schools in prior academic achievement.

We also downloaded 2021-2022 school year school enrollment data including race/ethnicity count data, student program data including count data for students considered economically disadvantaged, emergent English learners, students who were eligible to receive special education services, as well as gifted and talented program students. We focused on these covariates as previous research has indicated they are associated with science achievement. All count data was then converted to percentage data by school (e.g., number of economically disadvantaged students/ total number of students in a school). These variables were used to match STEMscopes and non-STEMscopes schools (see participants section below for details on matching). Once matching was complete and baseline analyses were conducted (see baseline equivalence), we downloaded the 2022 STAAR campus data file. We used the percent of students in each school who “approached” grade level standard (passing rate) on the spring 2022 STAAR as the outcome variable. This file also includes sub-population percent passing scores by school.

Participants
In the 2021-2022 school year, the overall number of Dallas schools that purchased and used STEMscopes (in any capacity) for 5th grade was 70 out of 156 in-person schools (~45%). Of these 70 schools, 44 met the criterion of school-wide usage of at least 14/18 5th grade science units (scopes) and were eligible for the study.

As mentioned previously, to match districts based on the data available from TEA website we used R-Studio’s “Match-it” package with the “nearest neighbor” method. We included 11 school level variables to match data: 5 race/ethnicity variables representing the percent of the school population that fit each category: White/Caucasian, Hispanic/Latinx, Black/African American, Asian, and Two or more races/ethnicities; as well as the total number of students enrolled in a school, the percent of students that qualified for special education or gifted and talented programs, the percent of English language learners, and the percent of economically disadvantaged students. Finally, schools were matched based on ‘baseline’ academic achievement as indexed by the STAAR 2021 4th grade math passing rate for a given school. All 44 STEMscopes schools were matched, resulting in a final sample size of 88 Dallas ISD schools. 

Missing Data
If a variable (e.g., program category/sub-population) has less than 10 students, the data is removed by the state and “<10” is included instead in order to protect student privacy under the Family Educational Rights and Privacy Act (FERPA). This leads to missingness by design in both covariate and outcome variables. We handled this missing data in two ways. For any covariate variable used to match schools, we used multiple imputation by chained equations via R-studio’s “MICE” package with the “CART” imputation method (see Van Buuren and Groothuis-Oudshoorn, 2011). We use MICE procedures during this step to ensure complete data for matching procedures via R-Studio's “Match-it” package. Once data were matched, in all final analyses we used R-Studio’s “Lavaan” package which uses full information maximum likelihood procedures to handle missing outcome data. 

Baseline Equivalence
For all covariate variables (the variables used for matching) including baseline academic achievement, there are no significant differences between matched groups (see Table 2). However, the What Works Clearinghouse (WWC) standards require that baseline differences with a Hedge’s G effect size greater than 0.05 must be controlled for statistically. Following the advice of Stuart, 2010, we include all covariates that were not collinear in the final analyses as a complementary approach to matching, and a more stringent test of effects. Several race/ethnicity variables were correlated above -0.80. Specifically, the percent of students that were Black/African American was highly negatively correlated with the percent of students who reported being Hispanic/Latinx, as well as the percent of students that was Hispanic/Latinx was highly negatively correlated with the percent of students who reported being multi-racial. With this in mind, we present models above with only the variable representing the percent of Hispanic/Latinx students included in the model (however, we re-ran models that included Black/African American and Multi-racial percent variable and the pattern of findings was the same). Inclusion of these covariates satisfies the WWC standard as several variables had effect sizes greater than or equal to 0.05. 

Method continued

Planned analyses

Analyses were conducted with R-studio’s Lavaan structural equation modeling package because this package includes estimation with full information maximum likelihood (FIML) to handle missing data. FIML procedures to handle missing data estimation ensure that in the final analysis the estimation is not biased. Our main variable of interest was the 2022 5th grade science passing rate (i.e., percent approaching grade level). We also include sub-population analyses for male and female students, ELL and non-ELL students, economically disadvantaged students, and for two racial/ethnic categories (Black/African American and Hispanic/Latinx) as all other racial/ethnicity percent passing variables as well as the non-economically disadvantaged variable included too much missingness to be considered (e.g., greater than 50% missing).