This report explores the impact of Math Nation on 2022-2023 Algebra End-of-Course (EOC) performance in 146 high schools. We used structural equation modeling regression analysis to compare Algebra EOC scores in Math Nation and non-Math Nation schools. The main finding is that Math Nation schools had a higher percentage of students scoring at a Level C or higher on the Algebra EOC (34.98%) compared to non-Math Nation schools (30.52%). This difference was statistically significant. We also examined sub-groups of students. Math Nation schools had significantly higher percentages of Black/African American, White, and students in poverty scoring at Level C compared to non-Math Nation schools. Overall, the most noticeable differences between Math Nation and non-Math Nation schools are observed at Levels C and F. Figures 1 and 2 illustrate these differences visually, showing significant distinctions in percentages of students scoring at Levels C and F, respectively.
|
|
 |
|
Figure 1. |
Figure 2. |
This study had two main data sources. First, Math Nation provided usage data for the 2022-2023 school year to identify Math Nation schools. Data included primary usage information (number of videos played and number of logins) for each school by grade level, as well as secondary usage variables (e.g., Test Yourself questions completed). For the purposes of this study, a school was considered a Math Nation if they reported a minimum of 5 videos watched per student.
We also used publicly available data from the South Carolina Department of Education website. Specifically, we used 2022-2023 Algebra EOC performance data as our outcome variable. We also used 2021-2022 Algebra EOC performance data, enrollment data, and demographic data to match Math Nation and non-Math Nation schools.
As is common across states, South Carolina does not include values for data where less than 10 students contributed data. To account for the missing data, we used the multiple imputation by chained equations (MICE) package in R to account for missing data. MICE is a statistical method used to handle missing data in a way that provides more accurate and reliable results. MICE estimates missing values based on the information available in the dataset. To do this, MICE creates multiple estimations for each variable. The idea is that each variable helps predict missing values for the others, creating a chain of predictions. It’s a bit like trying out different combinations to find the best fit, making the dataset more complete and reliable for analysis.
Next, we used propensity score matching when identifying Math Nation and non-Math Nation schools. Propensity score matching is a method used in research to make sure that when comparing two groups (i.e., Math Nation vs. non-Math Nation schools), they are as similar as possible. Similarity is important to ensure a fair comparison; we want to make sure that the only real difference between the groups is the factor that we are specifically interested in. In other words, we want to ensure that any difference between Math Nation and non-Math Nation schools is due to Math Nation itself rather than pre-existing differences between the two groups.
After creating matched groups, we used a Structural Equation Model (SEM) regression to determine if there were significant differences in the EOC scores of schools that were used. A SEM regression is a statistical method to understand and analyze relationships among different variables in a complex system. SEM allows us to investigate the relationship between these variables by creating a visual model that represents the connections between the variables. Each variable is like a puzzle piece, and SEM helps us put the pieces together to see the bigger picture.
In an SEM regression, the model not only looks at direct relationships between variables, but also considered indirect relationships. SEM regression is a tool that helps researchers explore and uncover the complex connections and interactions between different factors. It provides a more comprehensive view of various elements in a system that influence each other.
In summary, this approach allows us to focus on the impact of Math Nation on Algebra EOC performance and determine if there were significant differences in performance across sub-groups of students in Math Nation and non-Math Nation schools.
