Swisher wanted nothing more than to chat up the legendary figure about his days on the sidelines, but he let the moment pass, too scared to say anything.
Friday night, Swisher found himself standing at the plate in Yankee Stadium with a chance to win the Yankees' first game since Steinbrenner passed away on Tuesday.
The moment didn't pass him by.
With two outs and the score tied in the bottom of the ninth inning, Swisher singled off Lance Cormier to give the Yankees a 5-4 walk-off win against the Rays and a proper sendoff for their owner.
"The agenda today was win -- that's what Mr. Steinbrenner wanted us to do," Swisher said. "That's all he ever wanted to do. On a day like this, when we celebrate his life, gotta take him out on a 'W.'"
The Rays took a 4-3 lead into the bottom of the eighth inning, inserting setup man Joaquin Benoit to bridge the game to closer Rafael Soriano.
Swisher led off against Benoit and tied the game with a home run into the right-field stands.
"There was a lot today about Yankees-Rays, first place-second place right out of the break," Swisher said. "I think we need to put all that aside today, because today was Mr. Steinbrenner's day, regardless of the situation, regardless of anything. We went out there and we played that game as best we could for him today."
After Curtis Granderson singled to start the ninth, Brett Gardner walked and Derek Jeter came up with one out and a chance to win the game.
Jeter, of course, has been a centerpiece on five Yankees World Series Championship teams since 1996 and had a close relationship with The Boss.
It would've been fitting for him to get the hit that brought the Yankees a win in Steinbrenner's honor.
"[Jeter] comes to the plate, and I was thinking the same thing in the outfield. I was like, 'Yeah, this is perfect for a night like this,'" Rays center fielder B.J. Upton said.
But Jeter struck out swinging on a pitch in the dirt from Dan Wheeler, leaving it up to Swisher to deliver the hit for Steinbrenner.
"Mr. Steinbrenner loved his Yankees and loved his Buckeyes. And a Buckeye got it," manager Joe Girardi said. "I was thinking how fitting it would have been if Derek got the hit. ... I thought for sure Derek was going to get it done. But George's second love was the Buckeyes."
Swisher dug in and delivered a line drive to right field for his third hit of the night. Granderson never hesitated around third and when catcher Kelly Shoppach couldn't handle the throw from Gabe Kapler, Yankee Stadium erupted as the emotional day came full circle.
From the pregame tributes to Steinbrenner and long-time public address announcer Bob Sheppard to the walkoff single, Swisher was fully immersed in the moment.
"It was because of [Steinbrenner] that I got brought over here, and I thank my lucky stars every day," Swisher said. "This is baseball heaven."
Kyle Maistri is an associate reporter for MLB.com. This story was not subject to the approval of Major League Baseball or its clubs.