The largest differences in average percentages of students across performance levels occurred at the C Level and the F Level. Figure 2 illustrates the difference in mean percentage of students scoring at a C Level on the Algebra EOC in MN schools compared to non-MN schools. Each of these results is significant with the exception of the percent of Hispanic/Latinx students scoring at the C Level.
Figure 3 illustrates the differences in mean percentage of students scoring at an F Level on the Algebra EOC in MN schools compared to non-MN schools.
 |
 |
| Figure 2 | Figure 3 |
Data Sources
To conduct this study, information was required on Algebra EOC performance at the grade or school level as well as school video usage data. South Carolina provides EOC performance at the school level, but does not provide EOC performance by individual grade level, so analyses were conducted using the school level rather than grade level as a unit of analysis. School-level 2022-2023 Algebra EOC performance data, as well as demographic data, were obtained through the South Carolina Department of Education website for the purposes of conducting the main analyses. For purposes of matching MN schools and non-MN schools, 2021-2022 Algebra EOC school-level performance data were also required.
Estimating prior mathematics performance for this study is complicated by several factors. There were no middle schools in the Math Nation dataset that reported using videos, so the schools included in this study are all high schools. High school students can take the Algebra EOC in any grade (9-12), but since the EOC data are not disaggregated by grade, it is impossible to determine for any school how many students took the Algebra EOC in any specific grade. Thus, using students’ mathematics performance in a previous school grade is not feasible since there is a number of different grades, and information about student performance by grade level is not available. Even if it were assumed that a sizeable majority of high students taking the Algebra EOC are 9th-grade students, estimating those 9th-grade students’ previous mathematics performance would require utilizing their 8th-grade SC Ready math score. This would entail mapping which middle schools feed into the matched high schools included in the analyses for this study, and this information was also not available. Given these complexities, each high school’s previous year’s (2021-2022) Algebra EOC performance was used as a measure of that school’s previous mathematics standing.
In addition to school-level Algebra EOC data for 2021-2022 and 2022-2023, usage data were provided for Math Nation. These data included primary usage information (number of videos played and number of logins) for each school by grade level as well as secondary usage variables (for instance, Test Yourself questions). Because Math Nation data were reported by grade level, the number of videos played across all grade levels were added, and this sum was used for school-level analyses. The same process was used for number of logins. The distribution of videos played per school (or logins) was used, along with MN internal information about true user characteristics, to determine the minimum number of videos played for MN schools.
For the purposes of the main analyses for this study, a school was considered an MN school if that school reported a minimum of five videos played. For the secondary analysis of performance across MN usage levels, the numbers of videos played and logins for each school were converted, using school headcount data retrieved from the South Carolina Department of Education website to a count of videos played or logins per student. The distribution of videos and logins per student for MN schools was then used to construct equal-sized bins of video or logins per student.
Missing Data
As is common across states, South Carolina does not include values for data where a very small number of students contributed data. To account for this missing data in the covariates, multiple imputation by chained equations (MICE) was employed. All analyses were conducted in R-studio and for missing data, the ‘mice’ package was used, with 5 imputations and 20 iterations per imputation. Previous mathematics performance, race/ethnicity percentages, number of testers, and percent of students in poverty were used in the matching process.
Participants
In the 2022-2023 academic year, there were 74 schools that played at least five MN videos out of 287 public/charter high schools in South Carolina. 73 of these schools were matched (see below) and were included in all analyses.
Matching
The following seven demographic and achievement variables were included to construct a matched set of schools: 2021-2022 Algebra EOC score means, number of EOC test takers, percent of students in poverty, and percent of students across race/ethnicity categories of Black/African American, Hispanic/Latinx, White, Two or more race/ethnicities. In R-studio, the “Match-It” package with Mahalanobis Distance matching was used to construct a set of matched schools. A multivariate distance of 0.25 standard deviations was chosen because the ratio of MN schools to non-MN schools is relatively high in this study. This method resulted in 73 MN schools being matched, for a total of 146 sample schools. Analyses conducted after this matching confirmed no significant differences between matched groups on any of the covariate variables.
Quantitative Analyses
There are a number of options for conducting SEM; for this study, the Lavaan structural equation modeling package in R-studio was selected because of its capacity to handle missing data in procedural ways that minimize impact on final results.
