Can Digital Games Boost Students' Test Scores?

In the past few years, educators have been closely watching the evolution of digital games used for learning. With a huge influx of products -- whether they're individual apps for tablets or an entire suite of software -- the market is already big and continues to grow, with entire game-based schools cropping up across the country.

There's no question students are interested in digital games -- 97 percent of kids play them -- but what educators and industry watchers want to know is whether playing those games can actually improve student achievement.

A new SRI study released today suggests they do -- at least in the subjects of science, math, engineering, and technology. According to the report, which is an analysis of 77 peer-reviewed journal articles of students K-16 studying STEM subjects, "when digital games were compared to other instruction conditions without digital games, there was a moderate to strong effect in favor of digital games in terms of broad cognitive competencies."

More specifically, "students at the median in the control group (no games) could have been raised 12 percent in cognitive learning outcomes if they had received the digital game."

Another way to explain it: "For a student sitting in the median who doesn’t have a game, his or her learning achievement would have increased by 12 percent if he or she had that game," said Ed Dieterle, Senior Program Officer for Research, Measurement, and Evaluation for the Bill and Melinda Gates Foundation, which funded the SRI report.

Simulations have an even bigger impact, according to this analysis. When considering simulations -- taking a phenomena, process, or behavior and coding it into something that can be manipulated and studied -- improvement index jumped to 25 percent, meaning students who used simulations could have increased their learning outcomes by that amount.

Which begs the question, how do we define learning outcomes? According to Stacey Childress, deputy director of education at the Gates Foundation, learning outcomes can be defined in a few ways: progress toward mastery of a particular set of content and skill objectives in areas such as math and literacy; demonstration of complex skills like collaboration and critical thinking; and improvement in what researchers call "non-cognitive" skills such as persistence and grit.

"With learning games, it’s important to understand which kinds of outcomes they are designed to improve and whether or not students are actually making progress on those dimensions," Childress said.

The Gates Foundation has made huge investments in the educational gaming world. Last year, the foundation launched the Games Learning and Assessment Lab (GlassLab), which was tasked with prototyping and developing games and formative assessments. The work is being conducted by the Institute of Play, the Educational Testing Service (ETS), Pearson, Inc., Electronic Arts (EA), and the Entertainment Software Association (ESA). GlassLab recently released, SimCityEdu, which integrates assessments aligned with Common Core State Standards. The educational version uses the same code as the commercial game, but with the addition of using students’ choices during challenges as a method of assessment, though not all education experts agree that assessment should be built into games.

The foundation has also invested in The Center for Game Science and the Radix Endeavor at MIT with the intent to develop games that embed valid assessment measures.

For this analysis, SRI considered reports from a gamut of sources in those 77 studies -- going as far back as a 1992 study from The Journal of Educational Research looking at the effects of computer simulations and problem-solving approaches on high school students, to a 2006 study in the journal Interactive Learning Environments using just-in-time information to support scientific discovery learning in a computer-based simulation.

Other studies examined include one from 2011 that compares different versions of a game in terms of the degree to which the learning mechanics and goals are integrated directly into the central game mechanics (intrinsic design) versus separating the learning mechanics and goals from the central game mechanic (extrinsic design); and another from 2012 that compares different approaches to socially organizing players within a game in terms of collaboration and competition to maximize learning. The games within each study were developed specifically for research purposes, and thus are not as elaborate as some commercial titles like SimCity, but are solid examples of learning games, according to Dieterle.*

"This is the first big study to hit the pause button for a second and reach back in time and extract everything we could from what previous researchers have done with the intent of using that information to inform us about the field going forward," Dieterle said.

FUTURE OF GAMES IN CLASSROOMS IS NOW

If digital games were rare in the past, that's no longer the case. According to a recent teacher survey conducted by PBS, 43 percent of classroom computing goes to playing educational digital games. And in one study undertaken last year by the Joan Ganz Cooney Center, surveying 505 teachers, the majority of teachers reported that games increase motivation and make it easier to personalize learning.

But experiences and perceptions around games are still very much a mixed bag, depending on whom you ask. Some educators are skeptical that digital games are the answer. They question whether games provide enough context and depth that come from hands-on experiences.

“Imagine the difference between a student who’s playing an online math video game and a student who’s sitting in a small group with a teacher, working out problems and receiving immediate, individualized feedback and guidance,” said St. Louis-based fifth-grade teacher Jenny Kavanaugh in a recent interview. “There is no comparison.”

But Childress points out that the issue is more nuanced.

"The games and learning space is still in an exploratory, R&D phase. We shouldn’t frame games, or any other instructional support, as 'the answer,'" she said. "All of us working in education should be skeptical about any innovation that doesn’t aim to produce evidence of its effectiveness. The SRI results are a strong start in the direction of solid evidence."

And Childress does not see the use of digital games as an either/or scenario -- either teachers or digital games.

"We should be careful not to view learning technologies as a replacement for deep teacher and student interactions. We see effective technology supports as enabling the opposite," she said.

Digital games can be a part of a holistic plan that challenges students with things like "quests" and "missions," when paired with tactics like spending targeted time with students in small groups or individually to help them address areas where they need help, she said.

For educators who aren't sure where to start, or how to find ways to integrate digital games into the current model, Childress said teachers' own network can be a great resource. For instance, one of the Gates-funded organizations, Playful Learning, focuses on creating a national network that offers teachers workshops on using games in the classroom.

For their part, game developers should incorporate ways to help educators do their jobs better, as is the case with other industries that have embraced technologies. "The best product developers deeply understand who they are designing for and the use case they are targeting, and offer the kind of implementation supports professionals need to integrate new tools into their daily work," Childress said.

If Quest to Learn, the entirely game-based schools in New York and Chicago, are any indication of whether games can be successful learning tools, the potential seems bright. According to CNN, the school's New York test scores, "an admittedly conventional metric, show the Quest kids have outperformed peers in the New York City school system in each of the last three years, in both English Language Arts and Math, according to data provided by the school," with the only exception being the 2010 math scores.

But not every school can be a Quest to Learn, with dedicated funding for games. Finding the funds to finance digital games is one of the main obstacles, in fact. In the Cooney Center survey, 51 percent of teachers said that cost of digital games was the primary obstacle to integrating them into class, and only 17 percent of those surveyed said the school spent $100 or more on games.

To that end, Childress said there are a number of free resources available on the web, and that the foundation has funded 17 game development projects over the last three years, a number of which are free or available at reduced cost to districts serving students in low-income neighborhoods.