Introduction

In the 2016-2017 school year Math Nation (MN), an interactive online math learning platform, was piloted in 31 school districts as a supplemental math resource. The following year, MN was available to all public schools in the state and as of the 2021-2022 school year, MN was being utilized in 118 (86%) of Mississippi’s 137 school districts.

Previous studies have indicated that MN (formerly Algebra Nation) was effective at increasing student math achievement on the Mississippi Academic Assessment Program (MAAP). This study aimed to extend those findings by testing the efficacy of MN in middle schools and high schools across the state of Mississippi.

The following summary presents the key findings and recommendations from the comprehensive evaluation of the efficacy of MN in improving Mississippi students’ math scores on the MAAP.

Methodology
Statistical analyses were conducted using IBM Statistical Package for the Social Sciences (SPSS) software (version 28.0.0.0). Quantitative analyses included cluster analyses, comparative analysis, and linear regression. Cluster analysis was used to classify schools with similar characteristics and MN usage levels into comparable groups. Comparative analyses were used to compare school math proficiency and growth scores of MN schools with scores of non-MN schools. Regression analyses were used to examine the relationships between variables and control for confounding factors—variables that are not the focus of the study but could still influence the outcome of test scores or MN usage.

Key Findings
These findings indicate that MN had a positive impact on students’ performance on MAAP tests for the 2021-2022 school year. Results of the study indicated a statistically significant improvement in the MAAP test scores for students who actively engaged with MN.

• On average, schools utilizing MN demonstrated 13% higher math proficiency, math growth, and math growth lowest 25% compared to their non-MN peers.
• Math growth for students who scored in the lowest 25% on the previous year’s MAAP assessment, was significantly higher for students in MN schools.
• The strength of the correlations between math proficiency, math growth, and math lowest 25% scores and MN usage variables was stronger in schools with the highest MN usage.
• Videos Watched, Video Watched Minutes, Test Yourself Questions completed and Test Yourself Questions correct had the strongest positive relationships to MAAP scores.
• The MAAP scores for Black or African American students in MN schools consistently had high effect sizes (e.g., > .700) compared to other subgroups in both MN and control schools.
• MN schools located in urban areas within neighborhoods that predominantly consist of households with incomes below the federal poverty threshold and with a school accountability rating of ‘C’, had significantly higher mean math growth and math growth lowest 25% scores and the highest effect sizes compared to all other schools in the study.
• Math proficiency mean scores were significantly higher for Black or African American and White students in urban schools in areas with household incomes above the federal poverty threshold and a school accountability rating of D.

Methodology

Data Collection

Data sources
The study included all Mississippi middle and high schools that participate in the MAAP. School and grade level MN usage data for the 2020-2021 and 2021-2022 school years were provided by Accelerated Learning. These data included login information and the number of times students engaged with various MN learning resources (Table 1). School-level demographic and math assessment data for the 2021-2022 school year were downloaded from Mississippi Department of Education (MDE) website (Table 2). School geographic location, poverty data, and additional demographic data were retrieved from U.S. Department of Education (Table 3).

Preprocessing of Data
All datasets were transformed, cleaned, and organized to make them suitable for analysis. Data from multiple sources had to be integrated to ensure consistency and unity between datasets. The values of variables such as poverty rates, school accountability ratings, and usage calculations were normalized to similar ranges so they would have comparable magnitudes and preserve the relationships between data points. Also, some categorical variables were converted into a numerical format to make them suitable for analyses. To mitigate the impact of outliers, two step transformation to normality was used to normalize MN usage variables.

Schools with less than 10 unique logins (n = 64) were deleted from the MN usage dataset. All data within each dataset were assigned an MN usage variable of 1 for MN schools and 0 for control schools.