Little Jim Harbaugh would leave the house wearing his baseball hat and carrying a baseball glove.

"Where you going?" his father, Jack Harbaugh, would ask.

"Doubleheader today against the Indians," the 6-year-old would say. "Pitching both ends of the doubleheader."

"You're pitching both?" Jack asked, pretending to be surprised.

"Yeah, I'm pitching both."

And Jim would head out to a nearby parking lot, throwing the ball against the wall, as he pitched his imaginary doubleheader.

"He had one of the great imaginations," said Jack Harbaugh, 75, a longtime football coach whose career included a stint from 1973-79 under Bo Schembechler at Michigan. "He was a great daydreamer."

Jim would return home hours later, his hat askew.

"How did it go?" his father asked. "You're tired out, huh?"

"Won both ends," Jim told him.

"I think he was undefeated," Jack said, laughing. "I don't think he ever came home with his head down."

It is Father's Day on Sunday, and Jack will spend time with his son, Jim, Michigan's new football coach, back in Ann Arbor, a place that brought the family so much joy and strong memories. Their families will celebrate, but both men will tell you, though, that every day is Father's Day.

Jack Harbaugh has countless stories about both his sons, John and Jim, separated by 15 months, and so close in age they shared an undeniable competitiveness and played a lot of imagined — and real — games together. For good measure, they even faced each other in a "real" Super Bowl in 2013.

But long before their successes, the boys fashioned a basket from a hanger, used thread to make a net, wore undershirts they taped with numbers and played for hours, always outscoring the basketball stars of the day. Jack and Jackie also have a daughter, their youngest, Joani. Like her famous football-coaching brothers and parents, she is happily tied to the world of coaching as the wife of Indiana basketball coach Tom Crean.

For Jim Harbaugh, being in Ann Arbor now takes him back to his roots, and with his father who can share the old memories and the new ones.

"He's somebody that makes me laugh easily," Jim Harbaugh said. "He makes other people laugh. He's enjoyable to be around. He's not the person who brings doom and gloom. As far back as I can remember, I can think of him giving me advice and telling me what to do and learned very early that most all of the time if I did exactly the way he said it, things worked out very well."

Dad, wife are 'go to' people

Jim Harbaugh said his wife, Sarah, is his trusted confidante, whose opinion and advice, like that of his father's, takes precedence over any other. He calls them his "go-to" people for consultation about pretty much anything.

The two men have said their relationship has evolved over the years, and while Jack always will be the father and Jim the son, they have built a friendship in part because of the uniqueness of their careers as football coaches. Jack has become an adviser and a wall against which Jim can bounce ideas.

Perhaps most importantly, they share a mutual respect and an unwavering trust.

"I wish I had known them (John and Jim) before I started coaching," Jack said. "I watch and marvel at the passion and love they have for the game. I marvel at how caring they are for the players. They raise the players around them and the coaches around them. Sometimes I shake my head and think, 'Golly, maybe I would have been a better coach if I could have known them then.' "

Jim Harbaugh, 51, knows what a great coaching and life resource his father is.

"If I've got a football issue, if I've got any issue that I find myself racking my brain, giving me a headache trying to figure out the best thing to do, I call him and ask him," Jim Harbaugh said. "The conversation goes like this with him, 'Back when I was at Western Kentucky, a similar situation came up and this is what we did.' Two minutes later, I got the best advice to use, and my migraine leaves me.

"Nobody is 100 percent right on all their decisions, but I find when I get his advice, together we're scoring in the high 90s."

But the core of the father-son relationship is pure, not to be overanalyzed as anything more than the sharing of a strong, impenetrable bond.

"Above everything else, it's love," Jim Harbaugh said. "I love my dad. And just as good, he loves me back. There's no question, it is a relationship of unconditional love, whatever that truly means. What I do know is I love him and I love everything about him, love being around him, and he loves me back that way."

While their relationship has broadened to friendship, Jim said it is and always will be about being a father and son.

"I don't compare this to other father-son relationships," Jim Harbaugh said. "What has he done for me? How do you measure it? There's so much that you can't measure it. You can't put a percentage on it. He believed in me from the beginning.

"I love watching him with my kids, watching how he parents. I do a lot of the same things ... He's a wonderful teacher. I like going to ball games with him — love going to a baseball game with him, I like going to (see) 'Judge Judy' with him, and watching sports with him. If I'm doing something, it's 900 times more fun if I'm doing it with him, whatever we're doing."

Jack Harbaugh laughs when asked about going to baseball games with Jim.

It is always, he said, an adventure.

"He brings a glove, and he will insist on sitting somewhere where there are no obstructions, so when the ball is hit, there's no screen it can go into," Jack Harbaugh said. "If it's hit at his section, he wants a clean shot at it. He claims he has 19 balls he's gotten at the games and he claims they aren't batting practice balls, but ones he got during the games at some point during the nine innings.

"The ball comes, and no matter who's around him, it's everyone for themselves. He will dive for the ball. He will get on the concrete and crawl on his hands and knees to retrieve the ball. We did this at San Francisco last year at the Giants game. A couple people behind us weren't as aggressive for the ball, and we had to apologize that (Jim) works on a different frequency."

Lives coming full circle

Jack and Jackie Harbaugh have their Wisconsin home up for sale and plan at some point to move to Ann Arbor, in some ways seeing their lives together — they met in a freshman biology class at Bowling Green — coming full circle back to a place where they had so much fun.

The story often has been told of the Harbaugh kids running around Michigan football practices and occasionally meeting with consequences, courtesy of the players. These were formative years for the Harbaugh boys and where their love for their father's profession began to take hold.

"You'd hear a cry, 'Help, help,' when you'd leave the building," Jack said, mimicking a feeble voice. "They were tied up in a locker or tied to the goal post or tied to the bench press. We just had to make sure we got them and got them home."

Home. It's how Jim Harbaugh described Ann Arbor and Michigan when he was officially announced as the coach. He had played in the NFL for 14 seasons and coached in college at San Diego and Stanford before moving to the 49ers.

Michigan is where he starred as a quarterback and, really, where he grew up.

"The great story for me was when Jim decided (last December) what he was going to do, he was going to come to Michigan," Jack Harbaugh said.

Jack took the phone call at his home not long after he watched Jim coach the 49ers to a victory over the Cardinals in the final regular-season game Dec. 28.

So much had been swirling nationally about Jim Harbaugh and his future after four years with the 49ers. Was he destined for another NFL job? Could he take over his alma mater?

And then Jack got the call.

"He told me, 'I made up my mind. I'm going to the University of Michigan,' " Jack Harbaugh said.

Father asked son his reasons for reaching this decision.

"He said, 'You know, I'm thinking the seven years I spent in Ann Arbor walking from our house to St. Francis and coming to the (Michigan football) building and the relationships with the coaches and with Bo, you know what, I've got these three little ones, and I'm anxious they share the childhood I had and have those experiences,' " Jack Harbaugh said. "He thought so much of that experience, he'd want this kids to enjoy that."

Jim Harbaugh knows what taking the job at Michigan means to his father — and family.

"Something I would strive to do is give my dad joy," Jim said. "That's what he's done for me. I've always had a strong desire to make my dad proud of me. Hopefully I will do that every day. That's a powerful motivator. For my mom and my dad, for my wife and kids, those I associate with, I have a strong desire to make them proud.

"He's one of my best friends, my dad."