Executive Summary

The following report includes results from an ESSA tier 2 post-facto quasi-experimental design main effects efficacy study comparing matched Math Nation and non-Math Nation schools on the 2021-2022 Florida Standards Mathematics Assessment (FSMA) for 6th grade. Schools were identified as Math Nation schools if their district had a subscription to Math Nation for the students in the tested grades (i.e., 6th grade) and the within school usage was, on average 10 videos per student or more. Math Nation schools were then matched to schools that did not use Math Nation (or did not use it consistently) with matching based on previous year math passing rates and several school demographics. Multilevel models were used to test the effect of Math Nation program use on the 2022 Florida Standard Math Assessment school percent passing rate. Findings indicated that Math Nation was an effective program to stimulate student gains in mathematics as evidenced by more students passing in schools that consistently used Math Nation than matched non-Math Nation schools. This resulted in an estimated additional 887 6th grade students passing the Florida Standards Math Assessment.

Background

Mathematics achievement is one of the best predictors of students’ later successes. However, despite the importance of mathematics knowledge, students in the United States have not been keeping up with peers in other industrialized countries during recent years, with the United States ranked 22nd internationally on the 2018 Programme for International Assessment (PISA, 2018). With this in mind, much attention has been given to what types of mathematical instruction and supplements to instruction could potentially boost student enjoyment and knowledge of mathematical concepts. The Math Nation Program is a supplemental program that gives students 24 hour access to high quality mathematics materials including online instructional videos to help supplement what they are learning in their mathematics classrooms. Although prior studies demonstrate Math Nation’s effectiveness improving high school student’s Algebra competencies, the current study focuses on evaluating Math Nation’s effectiveness in supporting 6th grade mathematics learning. The purpose of the current post-facto quasi experimental study was to evaluate whether schools that used Math Nation consistently outperformed schools that did not use Math Nation (or did not use it consistently) when it came to the percent of students passing their end of 6th grade standardized math achievement test. We hypothesized that schools that used Math Nation would have higher percentages of students passing on the 2022 Florida Standards Math Assessment than matched non-Math Nation schools. Significantly higher passing rates for the Math Nation schools may indicate that Math Nation is an effective tool to boost US students’ understanding of mathematical concepts.

Methods

In this section, we provide details about study procedures including the data sources, variables used, and participating schools.

Data sources

Data for this study came from two sources. First, schools that used Math Nation for 6th grade in the 2021 - 2022 school year were identified through the Math Nation analytics platform. We used the “videos watched” metric as a “usage” indicator. Second, school demographic data and school performance on the Florida Standards Math Assessment (FSMA) was accessed through the Florida Department of Education website. We used the 2020-2021 school proficiency rate (the schools’ previous year proficiency rate) as a baseline measure of math achievement. We also downloaded school enrollment data: using the number of students enrolled in 6th grade, student ‘lunch status’ data (i.e., school percent of students eligible for free and reduced lunch) as an indicator of student economic disadvantage, race/ethnicity count data for 6th grade, and information regarding school type (e.g., regular, charter, alternative, virtual). For the current study we focused only on regular and public charter schools. Data were cleaned such that race/ethnicity counts were converted to percentages (number of students in a given racial category/ total number of students). These variables were used to match Math Nation and non-Math Nation schools (see participants section below for details on matching). Once matching was complete and baseline analyses were conducted (see participants), we downloaded the 2022 school proficiency rate, as measured by the percent of students who were proficient in math based on their performance on the spring 2022 FSMA, and this was used as the outcome variable.

Participants

In the 2021-2022 school year, the overall number of Florida schools that included sixth grade and that used Math Nation for 6th grade was 960 (this includes regular public schools, public and private charters). As mentioned above, we further defined “Math Nation school” by only including schools where 6th grade students, on average, viewed at least 10 videos per student and schools had a potential nonMath Nation school with similar baseline mathematics scores and school demographics. Any schools with less usage than 10 videos per student were placed into the pool of eligible control group schools along with schools that did not use Math Nation at all. Once we had identified Math Nation and non-Math Nation schools, we matched schools based on data available from the Florida Department of Education. Schools were first characterized as “regular public school” or “public charter.” All other schools including alternative, private, and virtual schools were removed from the data for the current efficacy study. Next, we matched schools as closely as possible across eight categories: their 2021 percent meeting the FSMA benchmark score, school size, the percent of students that were classified as economically disadvantaged (i.e., free and reduced lunch), and the percent of 6th grade students in the school across five race/ ethnicity categories (i.e., Asian, Black/African American, Hispanic/Latinx, Two or more races, and White/Caucasian). Our final sample included 490 matched schools. These schools served a range of sizes and percentages of minority and economically disadvantaged students, please see Table 1 below for details.

Results

Baseline Equivalence. All baseline analyses were conducted in R-studio. For all covariate variables (the variables used for matching) including baseline mathematics performance, there are no significant differences between groups (see Table 1). However, the What Works Clearinghouse suggests a stringent effect size (we use Hedge’s G) for baseline variables of less than 0.05 (if they are not to be included in the final analysis). In our case, two covariates failed to meet this guideline (percent Asian and percent African American students), thus these would need to be in the analyses to statistically adjust for potential baseline differences. Below we present analyses that always include baseline 2021 math performance (even though there was not a significant baseline difference, the inclusion of this variable ensures that prior mathematical ability is stringently controlled for within the final models) and percent of students identified as of Asian or African American descent.

Main Effects

To examine the effectiveness of Math Nation to increase FMSA school passing rates, we conducted multi-level multiple regression analyses with schools (n = 490) nested in districts (n = 70). These analyses were conducted in Mplus with a full information maximum likelihood robust (MLR) estimator because there was extensive missingness in the demographic race and ethnicity data. Specifically, as a measure of privacy, state data does not include a numeric value for any variable where less than 10 students contributed data. This led to missing data (by design — this means we know what caused the missingness). For example, the percent of students who reported Asian ethnicity variable was missing for 278 schools (57%). Mplus’ MLR estimator is one of the best available tools to account for the impact of missingness on analysis estimation. This ensures that in the final analysis where we include covariates, the estimation is not biased.

As the first step in the multi-level analyses, we included the 2022 6th grade school FMSA percent passing scores as the dependent variable alone to estimate the amount of between district variation in schools’ scores that was associated with what district a school was in. We would expect schools in the same districts to have more similar scores then schools in different districts (i.e., there is an effect of “district”). If variation is close to 0.00 then what district a school is located in does not have an effect. As variation moves closer to 1.00 the effect of the district on school FSMA passing scores is greater. The between district variation was 0.13. This confirms we need to include the district level in our analyses as this number is not close to 0.00. In all analyses, district was used as a “clustering” variable, to account for this effect. Next, we added a binary variable indicating whether the school was a Math Nation school or a non-Math Nation school as a predictor of the 2022 FSMA passing rate. We also added percent of students in a school identified as Asian and African American descent to the model. As can be seen in Table 2 and Figure 1, consistently using the Math Nation program resulted in a significant weighted increase in schools’ average FSMA passing rate of 1.63 percent compared to schools that did not use Math Nation or did not use it consistently. We estimate this difference is associated with approximately 887 more students meeting the passing rate benchmark in the Math Nation schools included in the sample.