Executive Summary

The following report includes results from an ESSA tier 2 post-facto quasi-experimental design main effects efficacy study comparing matched Math Nation and non-Math Nation schools on the 2021-2022 Florida Standards Mathematics Assessment (FSMA) for 8th grade. Schools were identified as Math Nation schools if their district had a subscription to Math Nation for the students in the tested grades (i.e., 8th grade) and the within school usage was, on average 10 videos per student or more. Math Nation schools were then matched to schools that did not use Math Nation (or did not use it consistently) based on previous year math passing rates and several school demographics. Multilevel models were used to test the effect of Math Nation program use on the 2022 Florida Standard Math Assessment school percent passing rate. Findings indicated that Math Nation was an effective program to stimulate student gains in mathematics as evidenced by more students passing in schools that consistently used Math Nation than matched non-Math Nation schools. This resulted in an estimated additional 1,600 8th grade students passing the Florida Standards Math Assessment, aiding in Florida schools recovery from the pandemic.

Background

Mathematics achievement is one of the best predictors of students’ later successes. However, the global COVID-19 pandemic hit math achievement hard, with students across the United States demonstrating lower levels of math related proficiency after nearly a year and a half of disrupted math instruction. For example, the percent of 8th grade students passing Florida’s Standards Math Assessment dipped from 46% in 2019 to 37% in 2021. As the 2021-2022 school year approached, many schools were looking for ways to recover. Math Nation offered Florida schools that chance at recovery. By offering students nearly constant access to online videos explaining math concepts and to “test yourself” knowledge questions, Math Nation offered an opportunity for schools to help students catch up via additional home opportunities to explore, learn, and test mathematical concepts. That said, the efficacy of the Math Nation product has largely remained untested in a general sample of 8th grade students. The purpose of the current post-facto quasi experimental study was to evaluate whether schools that used Math Nation consistently outperformed schools that did not use Math Nation (or did not use it consistently) when it came to the percent of students passing their end of 8th grade standardized math achievement test. We hypothesized that schools that used Math Nation would have higher percentages of students passing on the 2022 Florida Standards Math Assessment than matched non-Math Nation schools. Significantly higher passing rates for the Math Nation schools may indicate that not only are schools on the road to recovery – Math Nation is an effective tool to aid that recovery.

Methods

In this section, we provide details about study procedures including the data sources, variables used, and participating schools.

Data sources

Data for this study came from two sources. First, schools that used Math Nation for 8th grade in the 2021 - 2022 school year were identified through the Math Nation analytics platform. We used the “videos watched” metric as a “usage” indicator.

Second, school demographic data and school performance on the Florida Standards Math Assessment (FSMA) was accessed through the Florida Department of Education website. We used the 2020-2021 school proficiency rate (the schools’ previous year proficiency rate) as a baseline measure of math achievement. The 2020-2021 FSMA school proficiency rate was drawn from 7th grade students to ensure the same students tested in 8th grade in 2021-2022 contributed the baseline math assessment data. We also downloaded school enrollment data: using the number of students enrolled in 8th grade, student ‘lunch status’ data (i.e., school percent of students eligible for free and reduced lunch) as an indicator of student economic disadvantage, race/ethnicity count data for 8th grade, and information regarding school type (e.g., regular, charter, alternative, virtual). For the current study we focused only on regular and public charter schools. Data were cleaned such that race/ethnicity counts were converted to percentages (number of students in a given racial category/ total number of students). These variables were used to match Math Nation and non-Math Nation schools (see participants section below for details on matching). Once matching was complete and baseline analyses were conducted (see participants), we downloaded the 2022 school proficiency rate, as measured by the percent of students who were proficient in math based on their performance on the spring 2022 FSMA, and this was used as the outcome variable.

Participants

In the 2021-2022 school year, the overall number of Florida middle schools that used Math Nation for 8th grade was 768 (this includes regular public schools, public and private charters). As mentioned above, we further defined “Math Nation school” by only including schools where 8th grade students, on average, viewed at least 10 videos per student and schools had a potential non-Math Nation school with similar baseline mathematics scores and school demographics. Any schools with less usage than 10 videos per student were placed into the pool of eligible control group schools along with schools that did not use Math Nation at all. Once we had identified Math Nation and non-Math Nation schools, we matched schools based on data available from the Florida Department of Education. Schools were first characterized as “regular public school” or “public charter.” All other schools including alternative, private, and virtual schools were removed from the data for the current efficacy study. Next, we matched schools as closely as possible across eight categories: their 2021 percent meeting the FSMA benchmark score, school size, the percent of students that were classified as economically disadvantaged (i.e., free and reduced lunch), and the percent of 8th grade students in the school across five race/ ethnicity categories (i.e., Asian, Black/African American, Hispanic/Latinx, Two or more races, and White/Caucasian). Our final sample included 300 matched schools. These schools served a range of sizes and percentages of minority and economically disadvantaged students, please see Table 1 below for details.

Results

Baseline Equivalence. All baseline analyses were conducted in R-studio. For all covariate variables (the variables used for matching) including baseline mathematics performance, there are no significant differences between groups (see Table 1). Likewise, the What Works Clearinghouse suggests a stringent effect size (we use Hedge’s G) for baseline variables of less than 0.05 (if they are not to be included in the final analysis). In our case-only two covariates fail to meet this guideline (percent Asian and Percent White/ Caucasian students), thus these would need to be in the analyses to statistically adjust for potential baseline differences. Below we include analyses without covariates except for baseline math performance which is always included and analyses with all covariates as a stringent test of the effects of Math Nation.

Main Effects

To examine the effectiveness of Math Nation to increase FMSA school passing rates, we conducted multi-level multiple regression analyses with schools (n = 300) nested in districts (n = 66). These analyses were conducted in Mplus with a full information maximum likelihood robust (MLR) estimator because there was extensive missingness in the demographic race and ethnicity data. Specifically, as a measure of privacy, state data does not include a numeric value for any variable where less than 10 students contributed data. This led to missing data (by design— this means we know what caused the missingness). For example, the percent of students who reported Asian ethnicity variable was missing for 161 schools (54%). Mplus’ MLR estimator is one of the best available tools to account for the impact of missingness on analysis estimation. This ensures that in the final analysis where we include covariates, the estimation is not biased.

As the first step in the multi-level analyses, we included the 2022 8th grade school FMSA percent passing scores as the dependent variable alone, to estimate the amount of between district variation in schools’ scores that was associated with what district a school was in. We would expect schools in the same districts to have more similar scores then schools in different districts (i.e., there is an effect of “district”). If variation is close to zero then what district a school is located in does not have any effect. As variation moves closer to 1.00, then the effect of the district on school FSMA passing scores is greater. In the case of the current analysis, the between district variation was 0.20. This confirms we need to include the district level in our analyses as this number is not close to 0.00. In all subsequent analyses, district is used as a “clustering” variable, to account for this effect. Next, we added a binary variable indicating whether the school was a Math Nation school or a non-Math Nation school as a predictor of the 2022 FSMA passing rate. We also added the 2021 7th grade FMSA passing rate as a predictor to account for baseline school math passing rates. As can be seen in Table 2, consistently using the Math Nation program resulted in a significant increase in schools’ average passing rate of 4.65 percent compared to schools that did not use Math Nation or did not use it consistently. This is equivalent to a Hedge’s G effect size of 0.21.

The final analytic step was to add all covariate variables to the model. As can be seen in Table 3 and Figure 1, even while controlling for the effects of covariates that could potentially impact FSMA passing rates, consistently using the Math Nation resulted in a significant weighted increase in schools’ average FSMA passing rate of 4.85 percent compared to schools that did not use Math Nation or did not use it consistently. This is equivalent to a Hedge’s G effect size of 0.22. We estimate this difference is associated with approximately 1,600 more students meeting the passing rate benchmark in the Math Nation schools included in the sample.