"There's no question in my mind that I had to be here," he said. "It was my home away from home. I basically spent most of my adult life here, such great memories."
Williams spent the time away from baseball as a chance to be with his family, sending his oldest child to college. It gave him the opportunity to pursue other interests like music, though Williams said he is staying in shape "just in case."
He continued to keep his distance from Yankee Stadium, saying there was a duel purpose for that. He didn't want to serve as a distraction, but he also stayed away for his own peace of mind.
"I know that it was hard for me, because I missed the game," he said. "I was always trying to keep track of how the guys were doing. The Yankees were always in my heart."
Like many others, Williams has mixed feelings about the closing of the stadium that has housed so many historic moments. He knows that he will have to keep his own memories fresh in his mind, as the ballpark will no longer be around to serve as a reminder.
During his final visit to Yankee Stadium, Williams recalled his first day on the field when he made his Major League debut. For the former center fielder, the last contest seems to come full-circle, as his first game was also against the Orioles. As a young player, Williams said his main focus was to establish himself as a big leaguer. But during his later years with the Yankees, he learned to concentrate more on the history and tradition of the park he called home.
"As I got older, I started worrying less about my career and just started to have more of an appreciation to where I'm playing," he said.
Some of the memories that stick out to Williams come on a more personal level, having nothing to do with baseball. Williams distinctly remembers one day when Paul Simon stood in center field performing a sound check and the sounds of the guitar could be heard throughout the stadium.
Williams took part in a pregame ceremony as part of the celebration of Yankee Stadium on Sunday night, and though he said baseball has taught him never to be surprised, Williams could feel a familiar sensation as the moment drew closer.
"I feel as nervous as I was before a playoff game just to see the reaction of the crowd," he said. "Being a human being and being subject to all the emotions that this occasion will bring, I think it's going to be a very emotional moment for me and I guess for all the people that are really tied up to this building."
Williams was announced last during the introductions and received a rousing, deafening ovation from the fans.
He has heard the plans for the new stadium, though he has not seen the facilities yet. But while some talk has questioned whether Yankee Stadium will hold the same feeling once it moves across 161st Street, Williams said it's not the structure that made the Cathedral the place it is.
"It was more the people than the stadium," he said. "The building was here, and you talk a lot about the magic and the aura but what really made the stadium is the fans. Concrete doesn't talk back to you, chairs don't talk back to you. It's the people that are there, that root for you day in and day out, that's what makes this place magical."