What was this study about?

The study looked at whether Florida students did better in Algebra 1 after their schools used the Math Nation program. The big question was: Does using Math Nation help students perform better on the state Algebra End-of-Course (EOC) exam?

How did the researchers figure this out?

Let’s say you’re trying to figure out if a workout program helps people get stronger. But almost everyone in the workout group is already athletic, while the comparison group is mostly beginners. If you just compare their results directly, it’s not fair—because they didn’t start from the same place.

That’s the situation we had here. Many more schools used Math Nation (1,153) than didn’t (430), and they were different in important ways. For example, schools that chose to use Math Nation often had better scores the year before, more stability, or fewer students missing school. Just comparing raw scores would be like comparing a varsity team to a junior varsity team—it wouldn’t tell us much about the actual impact of the program.

Instead, we used a fairer, more careful method called weighted propensity score regression:

Step 1: Level the playing field
We looked at dozens of facts about each school—things like:

• How students performed in math the year before
• How many students were economically disadvantaged
• How many students missed lots of school
• How stable the school was (did it have high staff turnover or not?)
• Other characteristics like race/ethnicity and school size

Then, we used that information to estimate the likelihood that each school would use Math Nation. This is kind of like giving each school a “score” based on how much they look like a typical Math Nation school.

Step 2: Weight the data
Next, we used those scores to give each school a weight. By doing this, schools that were underrepresented in the sample (like comparison schools) counted more in the analysis, and schools that were overrepresented (like many Math Nation schools) counted a bit less. This process is called Inverse Probability of Treatment Weighting (IPTW).

This doesn’t mean we left out any data or downplayed the impact of Math Nation; it just made sure every school’s experience counted fairly in the comparison. Think of it like adjusting the volume on each school’s voice in a big conversation: quiet schools get turned up, loud ones get turned down—so everyone is heard equally.

If we had just looked at the average test scores for Math Nation schools and compared them to non-Math Nation schools, we would have gotten a misleading picture. That’s because those schools started in very different places. It would be like giving credit to the workout program for results that were really due to the participants being fitter to begin with.

Instead, by using this weighted method, we were able to create a “what-if” scenario:

What if schools that didn’t use Math Nation were just like the ones that did—would their students have performed the same?

This approach lets us make a fair comparison and say with more confidence whether Math Nation was actually helping students.

What did the researchers find?

Once we leveled the playing field, we found that schools using Math Nation outperformed non-Math Nation schools in several important ways:

• Schools using Math Nation had about 8% more students scoring at the highest level (Level 5) on the Algebra test.
• Fewer students got a Level 3 score (which is average). This suggests that some students may have moved from average to high scores.
• In schools that used Math Nation more, there were signs that fewer students were stuck in the lowest scoring levels, though these results weren’t always strong enough to be certain.

To put it simply: The more a school used Math Nation, the more students seemed to do better on the Algebra EOC test — especially those who reached the top level.

Why does this matter?

This study suggests that how much a school uses a learning tool like Math Nation can make a difference. It's not just having access to it — students need to actually use it regularly. When they do, it may help push them toward better performance, even if they were doing okay before.

How reliable are these results?

The researchers checked the results in multiple ways — like double-checking a recipe by trying it with slightly different ingredients. We looked at different levels of Math Nation usage—from light to heavy. At every level, students who used Math Nation more tended to perform better. The pattern was clear: more use = more improvement. This builds confidence that the improvements we saw were linked to the Math Nation program and not just random chance.

The Big Takeaway

Just like going to the gym doesn’t help unless you actually work out, simply having Math Nation isn’t enough. But when students use it regularly, it can really help them succeed — especially in getting top scores in Algebra. Helping more students reach top performance in algebra can change the path of their education. It opens the door to more advanced math, better test scores, and future opportunities. Math Nation is a helpful tool to get them there, especially when used consistently.