Executive Summary

For third and fourth grade teachers (n = 6) and their students (n = 151) in Galveston ISD who used the STEMscopes Math curriculum in school year 2020-2021, we saw significant increases in their standardized math achievement test scores related to 1) teacher reported weekly use of the STEMscopes Math curriculum, and 2) teacher reported increases in student’s enjoyment of using STEMscopes Math. Usage and enjoyment were also significant predictors of the benchmark “student approaches grade level” math achievement, and usage was a significant predictor of the benchmark “student meets grade level” for 4th grade, but not 3rd grade students. Overall, results indicate that using the program more and increases in student enjoyment of STEMscopes Math across the year play a key role in student math achievement. 

Background

A critical part of educational product evaluation is understanding how product usage and enjoyment may impact product effectiveness. One of the most important predictors of whether a curriculum is effective relates to how often teachers use a product such that if they are only using materials once in a while, it is not possible for the curriculum to fully support learning (achievement outcomes) as intended. Although many factors relate to why or why not a teacher uses a curriculum regularly, student enjoyment is often cited as an important factor among teachers. Student enjoyment also can increase student achievement scores

This study focused on STEMscopes Math curriculum usage in 3rd and 4th grade, as developed by Accelerate Learning Inc. and implemented by teachers in Galveston Independent School District (GISD). The STEMscopes Math curriculum follows a 5E + IA model and includes hands-on and virtual curricular supports to help teachers implement math learning. This study represents an early phase research study designed to test the impacts of STEMscopes Math Curriculum usage and increases in teacher reported student enjoyment of STEMscopes Math on student achievement. We expected that students in classrooms where teachers reported using STEMscopes more regularly, and students who increasingly enjoyed using STEMscopes Math would have higher STAAR standardized math scores and would be more likely to pass math benchmarks compared to students who did not increase their enjoyment across the school year, or students in classrooms who used STEMscopes Math less often.

Participants

Third (n = 3) and fourth grade (n = 3) teachers (2 male and 4 female) and 151 third (n = 76) and fourth (n = 75) grade students across five GISD elementary schools participated in the study and used STEMscopes Math throughout the 2020-2021 school year. The 2020-2021 school year included unique challenges due to the Covid-19 pandemic. GISD focused on a hybrid model with the first six weeks of school following a remote virtual learning model and the rest of the school year as in-person instruction for most students. 

During the school year, individual students often had to quarantine and were supported with virtual instruction while they were out.

Study Design

The purpose of this study was to examine whether varying use of STEMscopes Math by teachers, and increases in student enjoyment of STEMscopes Math predicted Math achievement. We evaluated teacher reported usage at the end of year (EOY; March, 2021) and changes in student enjoyment from December 2020 to EOY utilizing teacher survey responses as the main data source. We focus on EOY usage because 4 out of 6 teachers’ usage was consistent across the year (two reported increased usage, thus we focus analyses on their highest usage), and we focus on change in student enjoyment as we felt this change may be important to capture given the tumultuous nature of a pandemic school year. Student math achievement data came from GISD and was indicated by middle of year (December) scores on a district created and administered curriculum based assessment (CBA), and scores on the Texas STAAR standardized assessment (May, 2021).

We utilized the STAAR standard scores and STAAR benchmarks as our main outcomes. Standard scores in our data range from 1150 - 2024 and are vertically aligned across grades. This means that even though third and fourth graders have slightly different STAAR math tests with different math questions, all of the scores have been converted to the same scale. Thus a third grader who gets the same raw score as a fourth grader will have a lower standard score to account for the grade level differences. Scaling data across grades onto one scale makes it possible to compare scores and include both third and fourth graders into one analysis. Nevertheless, we include grade as a covariate in all analyses as we know fourth graders will (generally) have higher scores than third graders by design. We include one other covariate in all analyses: the district CBA assessment (as an indicator of student’s previous math abilities). This is to ensure that potential associations between usage and enjoyment are not due to these other factors that vary among students.

Finally, we also evaluated associations between teacher usage and increased student enjoyment on whether a student met STAAR benchmarks (please note these are also scaled by grade and can be combined). We focused on two levels: whether a student: “approaches grade level” and “meets grade level.” Unfortunately (and likely due to pandemic effects), too few students achieved the “mastery” benchmark on the STAAR math assessment this year (n = 22 or ~15%); thus it could not be appropriately analysed.

Analysis And Results

We used multiple regression to evaluate the associations between teacher usage and increases in student enjoyment with student math achievement. We used logistic regression to evaluate the associations between the two independent variables and STAAR benchmarks. Logistic regression is a special regression analysis that focuses on and accurately estimates associations between variables when the outcome variable is binary (in this case “pass/did not pass”). With logistic analyses “odd-ratios” can be easily calculated, that is, we can estimate “how many times more likely” a student is to pass the benchmark based on our predictors.

Descriptives. As can be seen in Table 1, half of teachers reported using STEMscopes Math “as a supplement here and there” and the other half reported using it 1-2 times a week. Teachers rated student enjoyment from 0-100, reporting a mean of 46.83 early in the year and 61.07 at EOY. The mean increase was 14.24 across the year (SD = 6.30). Every teacher reported an increase, with increases ranging 5-21 points. The mean of the STAAR standard scores was 1452 (SD = 165.98). Approximately 58% of students met the benchmark “approaches grade level,” while approximately 32% of students “met grade level” standards. 

STAAR scale score results. Results are in Table 2 and show that teacher usage was a significant predictor of student STAAR math achievement, b = 69.34, p < .05. A one unit category change in teacher reported usage (e.g., from “supplement” to “1-2 days a week”) is associated with a predicted student STAAR scale score increase of ~69 points. Increases in teacher reported student enjoyment of STEMscopes Math also was a significant predictor of STAAR math achievement, b = 53.83, p < .05, with a 10 unit increase in student enjoyment associated with a predicted ~54 point increase in student STAAR scaled scores.

STAAR benchmark results. Similar to results with the full range of STAAR standard scores, usage was a significant predictor of the “approaches grade level” benchmark (b = 0.99, OR = 2.70, p < .05). In this case, “OR = 2.70” represents an “odds-ratio” and can be interpreted: when teachers reported using STEMscopes 1-2 times a week (compared to as a supplement), students were 2.70 times more likely to pass the “approaches grade level” benchmark. Increases in student enjoyment of STEMscopes was also a significant predictor of the “approaches grade level” benchmark (b = 0.82, OR = 2.28, p < .05). For every 10 unit increase in student enjoyment, students were 2.28 times more likely to pass the benchmark.

However, the pattern was different for the “meets grade level” benchmark. Interestingly neither usage (b = 0.65, p = .17), nor enjoyment (b = 0.77, p = .13) were significant predictors of the “meets grade level” benchmark. We will note that among the covariates, grade was a significant predictor (b = 1.35, OR = 3.87, p < .05). This is interesting because the benchmarks are scaled similarly across the STAAR (with students needing to score above ~70% correct on their grade level test to be considered “meeting grade level”); thus we should not have seen a grade level difference based solely on the fact that students are older/ more advanced. With this in mind, we considered whether other predictors may interact or act differently by grade level. To test this, we separated data by grade and re-ran analyses. In third grade (and similar to the main analyses), neither use or enjoyment were significant predictors of the “meets grade level” benchmark. However, in 4th grade, usage was a significant predictor (b = 3.59, OR = 3.61, p < .01). Students in classes that used STEMscopes 1-2 times weekly were 3.61 times more likely to “meet grade level” compared to 4th students in classes that used STEMscopes math as a supplement.