Research Shows Gains in ST Math Use

Technology is making its way into many classrooms across the country, easing the way for schools to use educational software as one part of instruction. While the proliferation of ed-tech products and software programs can seem unending, the research backing up the claims made by companies has yet to catch up. This environment makes it challenging for school leaders to make informed decisions about products that have a big impact on how teachers teach and students learn.

A study completed by the independent, not-for-profit research and development agency WestEd is an effort by the MIND Research Institute to document if its program, ST Math, makes a measurable difference on student test scores. The institute commissioned the study to compare the state test scores of 19,980 second-to-fifth-graders in California, who had access to ST Math’s program in the 2010-2011 school year, with a comparison group of students who did not have the program. All teachers who used ST Math got the same training and teacher support from the ST Math staff on how to use the software. They also received help on how to integrate the technology for use in face-to-face teaching.

Researchers found that 65 percent of students who had access to ST Math scored proficient or advanced on the California Standards Test, as compared with 62.5 percent in the comparison group. That difference is statistically significant, but in real terms doesn’t appear to be a very big effect.

However, not all classrooms that used ST Math took a comprehensive approach. For grades that “fully implemented the program,” meaning at least 85 percent of the students used the program to cover at least 50 percent of the content for the class, the difference was greater. These schools invested time in program adoption, rather than just having access to the program and using it intermittently. For that group, 68 percent of students scored proficient.

“The biggest limitation is the fact that we’re doing a matched comparison study,” said John Rice, senior research associate at WestEd. It’s a quasi-experimental research setup -- not the gold standard randomized control study by any means -- but the best that could be done in this case. Researchers adjusted for factors like race, socioeconomics and starting levels of proficiency, to make the data as even as possible, but comparing grade-level cohorts in this manner is inherently opaque.

“The grades that implemented ST Math selected themselves to do this, so there may have been other factors that influenced improvement besides the program,” Rice said. Still, the results indicate that teachers who invested more heavily in the program saw better results on tests. Rice said a longitudinal study that looks at individual student growth would give more detailed information about the program’s effect.

While tests are only one measure of learning, they are often the most public indicator of a school’s performance. For this study, WestEd researchers excluded high-performing schools from the sample, hoping to take a deeper look at how the program affects underperforming students.

HOW DOES THE PROGRAM WORK?

ST Math features a penguin that guides students through math concepts in a visual way. It doesn’t rely on language to teach math concepts the way a textbook does. “We don’t usually consider the extra cost in complexity of introducing students to math using language,” said Andrew Coulson, chief strategist at the MIND Research Institute, ST Math's publisher. He said that introducing the concepts without the burden of language allows kids to grasp the ideas before learning the vocabulary and symbols that go along with them.

“It has the side benefit that you don’t need to be proficient in English-language skills in order to use the program,” Coulson said. The MIND Research Institute opened its doors as a nonprofit in 1998 as a spinoff of work done by several professors at UC Irvine. Now, a combination of grants, donations and licensing revenue from schools using ST Math helps pay for research, development and even some reduced-cost licenses for schools who can’t afford to pay.

“I hope this is one of what will be many scientific studies coming from the industry,” Coulson said. He hopes to conduct a similar study in several years to see how effective the program is at teaching Common Core State Standards.

“The whole industry needs to grow up and say we’re going to track what we’re doing and report out,” Coulson said. So far, that level of accountability and transparency around outcomes has been rare.