Recovering in reality with augmented reality games

Evan Carter
The Detroit News

Ann Arbor — Like many 7-year-olds, John “Jack” Haas-Pueblo enjoys playing games on smartphones and tablets.

But while many other kids play these games just for fun, Jack also uses them as a part of his rehabilitation from a brain aneurysm in February.

With a steadying hand from certified occupational therapist assistant Donna Thompson, left, and physical therapist Leah Hagamen, right, Jack Haas-Pueblo, 7, guides his finger along the touch screen of a smart phone using a mobile app called Spellbound that helps him interact with a children's book during his therapy session at C.S. Mott Children's Hospital at the University of Michigan, August 8, 2016, in Ann Arbor.

Physical and occupational therapists at C.S. Mott Children’s Hospital at the University of Michigan, where Jack is staying, use augmented reality — or AR — games such as “Pokemon Go” and an Ann Arbor-developed app, SpellBound, to help kids practice fine motor skills while playing as they recover from injuries and illnesses.

During a recent session, Jack’s physical therapist Leah Hagamen and his occupational therapist Donna Thompson were reading the book “Albert the Confused Manatee” with the SpellBound app so Jack could practice touching specific spots in the app on a smartphone with his right index finger.

“Whoa,” Jack exclaimed after the app told him the beluga whale on his screen was one of the biggest animals in the world.

Jack was taken to the emergency room at Bronson Methodist Hospital in Kalamazoo on Feb. 20 with what his mother, Rachael Haas, 42, thought was the flu. While in the emergency room, doctors discovered he had suffered a brain aneurysm caused by an arteriovenous malformation rupture. The resulting hemorrhage began pushing his brain stem into his spinal column.

He underwent two major surgeries within 24 hours. He survived both but fell into a coma for a couple of days, according to his mother. Even after he woke up, he was in what doctors call a “locked in” state — he had mental cognition but couldn’t express himself to the outside world.

His physicians in the hospital thought Jack had little chance of recovery and encouraged his mother to take him off life support, she said.

Jack, who once enjoyed playing baseball and football, couldn’t walk, talk or even hold up his head. But his mother refused to give up and pursued treatment elsewhere.

Jack Haas-Pueblo, 7, can't hide his joy when he got to bounce on an exercise ball while using a touch pad to control a Star Wars BB-8 remote control robot on the floor of the physical therapy room at C.S. Mott Children's Hospital at the University of Michigan in Ann Arbor.

Progress at last

In May, after stints in three other Michigan hospitals, Jack was transferred to Mott. Since then, he can raise his head, grasp objects and talk in full sentences.

“His legs are really strong, he just doesn’t have the coordination yet,” Haas said. “The last couple of weeks, he started to talk like he used to, to the point were the only time he stops is when he’s eating and sleeping.

“They’ve done a lot of tests, and a lot of the cognition is intact, it’s just that the bridge between the brain and the body is trying to heal itself,” she said.

While his physical and occupational therapists take him through traditional exercises like practicing walking and sitting straight up on an exercise ball, Jack does much of his therapy work while he practices sitting and standing up straight in front of the table where he plays AR games like “Pokemon Go” and SpellBound.

According to hospital officials, Mott has been using AR technology for three years — long before “Pokemon Go” or SpellBound were released. In addition to rehabilitation, the hospital has used AR technology to calm children before they go into surgery.

During the therapy session, Hagamen, Thompson and rehabilitation engineer Jamie Mayo worked on Jack’s head and back posture as well as using his right hand — he’s less confident using his right hand than his left — while Jack played games on Mayo’s Samsung Galaxy S7.

Even though ataxia has taken the accuracy from his body movements, it was obvious that Jack had the energy and enthusiasm of a typical 7-year-old as he threw Pokeballs and interacted with confused manatees on the smartphone screen. And while he followed his therapists’ instructions, he was most intent on having fun.

“When you do pediatric therapy, you have to play games,” Thompson said. “Kids don’t care about your goals.”

As a part of his therapy, many of the doctors assigned to work with Jack use tablet games to keep him engaged and having fun while completing stretches and motor skill exercises. For example, Thompson puts an iPad a little bit above Jack’s field of view when she wants him to practice looking up.

At one point, after some effort and concentration, Jack slid two special SpellBound cards within the view of Mayo’s smartphone camera, making an AR tiger and elephant pop up on the device’s screen; the tiger growled and the elephant trumpeted. Jack kept tapping the screen to hear the tiger growl again.

It often took Jack several attempts to get his wandering right pointer to go where he wanted it to go, but in the end he was always successful — at some points with a little help from Hagamen or Thompson.

Whenever Jack completed a task, his therapists cheered and he flashed a big, toothy smile.

Helping kids with AR games

SpellBound is the first app created by Christina York and her startup company, ALTality. York began her company in November 2014 because she believed in the potential of AR 18 months before the “Pokemon Go” phenomenon made the technology mainstream.

“AR has fascinated me, and I think there are a lot of problems which augmented reality can help solve,” York said.

With the success she has found with SpellBound, York is focusing on creating technology for pediatric hospitals to help distract, entertain and engage patients.

According to Travis Linderman, the director of operations for entrepreneurial services for Ann Arbor Spark, the co-working space and “economic engine” ALTality is based in, York’s outfit is part of a whole group of companies creating AR technology in Ann Arbor.

“It’s positioning Ann Arbor to be one of the players in this in the next 10 years,” Linderman said. “There’s definitely people in Detroit collaborating on this (also).”

Having companies in Michigan that specialize in AR technology could be important to the state’s economy in coming years as the research and advisory firm Gartner forcasts that in 2016, 1.9 million head-mounted displays will be sold (AR and virtual reality), with that number skyrocketing to 39.9 million units sold in 2020.

But even with its potential to play a part in Michigan’s economic future, Jack only cares that SpellBound allows him to play with animated tigers and elephants.

“I thought they were real good,” he said.

(313) 222-2613

Twitter: @evancarter_94