But Alou insists he knew back then that Williams would be something special.
"I knew he was going to be something because of his character, because of his dedication and all the tools," said Alou, now a special assistant with the Giants after 31 Major League seasons as a player and skipper. "He had tools all over him."
On Thursday, Williams had memories all over him.
The immensely popular Yankees center fielder sat in a press-conference table at Major League Baseball's headquarters in Manhattan with three baseball legends who helped shape his early life in Puerto Rico: Alou, who managed him early in his playing days; Minnie Minoso, who his father raved about while Williams was growing up; and Hall of Famer Tony Perez, who had been around Williams since he was a teenager.
This was the day that Williams would share something in common with them, as the Latino Baseball Hall of Fame announced that he would be enshrined as part of its third induction class.
Williams was joined by five other contemporaries (post-1959) selected from a group of 24 finalists: former catcher and current Yankees bench coach Tony Pena (a native of the Dominican Republic), infielder Manny Trillo (Venezuela), third baseman Aurelio Rodriguez (Mexico), outfielder Tony Oliva (Cuba) and outfielder Ben Oglivie (Panama).
Ex-players Ramon Bragana, Angel Castro, Pat Scantlebury, Luis Arroyo, Guayubin Olivo and Alejandro Carrasquel were named as part of the Veterans Committee, and Hall of Fame executive Pat Gillick was given the organization's honorary Tommy Lasorda Award in an event that saw Omar Minaya -- the first Hispanic general manager in the Majors -- serve as emcee.
Williams was an All-Star and a champion in New York City, but it's his Puerto Rican roots that he'll always be most proud of.
That's why this day was so special for him.
"What I'm most proud of is that all of those things [I accomplished], I did it as a Puerto Rican, and I was able to represent my country in a very dignified way," Williams said. "Puerto Rico was where all those dreams started. So I'm very proud to say that I did it as a Puerto Rican, as a Latino baseball player."
Williams, now a successful musician recording his third album, is one of the island's favorite sons. His sterling reputation was formed throughout a successful 16-year stint with a Yankees team that has a very large Puerto Rican fan base.
Williams' Major League career finished with five All-Star Game appearances, four World Series rings, four Gold Gloves, a batting title, 287 home runs, a .297 regular-season batting average and an .850 OPS in 121 playoff games.
His numbers, coupled with his grace and humility, made him an easy choice for the Latino Baseball Hall of Fame's new crop of immortals.
"I saw him playing baseball when he was a youngster in Puerto Rico because I lived there, and he has always been a great guy, a great person and a great player," Perez said. "All those years he played for the Yankees and all those winning seasons and the World Series, that was great. I knew he was going to make it, and I know I want to see him in Cooperstown, too."
Founded in 2009, and run in coordination with MLB and the National Baseball Hall of Fame, the Latino Baseball Hall of Fame has its headquarters in the Dominican Republic. On Feb. 11, inductees will be enshrined at the Altos de Chavon amphitheater in the Casa de Campo resort located in the Dominican city La Romana.
There, they'll join 38 former Latino players and personalities already inducted, including legends like Roberto Clemente, Roberto Alomar, Perez, Minoso, Alou, Juan Marichal, Rod Carew, Orlando Cepeda, Dave Concepcion, Luis Aparicio, Luis Tiant, Edgar Martinez, Andres Galarraga, Camilo Pascual and Fernando Valenzuela.
Members are selected via several rounds of voting that includes input from the MLB Players Association, fans who register at the Hall's website (http://www.salondelafamadelbeisbollatino.com/) and three different channels from each Latin American nation.
Currently in the works is a museum, on which construction is supposed to begin for in the near future.
Said Latino Baseball Hall of Fame founder and president Roberto Weill: "Now in this time of history, where baseball in the Latino world is becoming an ever-present important equation in the world of Major League Baseball, we felt that it was important to not only isolate, but to give excellence at its maximum by the Latino world to the Latino luminaries."
Williams, who played for the Puerto Rican national team during both World Baseball Classics, is honored to be inducted into the Latino Baseball Hall of Fame.
As for Cooperstown?
"Just to be mentioned in that light, it's great," Williams said. "It always gives you a great sense of accomplishment. But at the same time you have no control over that decision and what's going to transpire. My dad always told me, 'You always want to have that goal to get to the Hall of Fame.' But as I'm getting older, I know that my joy of the game, a lot of it comes from the relationships I was able to develop; all the memories and experiences I was able to go through that have made a profound impact on my life. Everything else is just icing on the cake."
Alden Gonzalez is a reporter for MLB.com. Read his columns and his blog, listen to his podcast and follow him on Twitter. This story was not subject to the approval of Major League Baseball or its clubs.