Cesar Ramos started for the Rays a day after the team announced that he is has joined Jake Odorizzi, Erik Bedard, Enny Romero, Alex Colome and Nathan Karns as a candidate for the No. 5 spot in the rotation.
Ramos surrendered one-out singles to Eduardo Nunez and Ichiro Suzuki in the first inning, before former Rays Russ Canzler singled home Nunez. Kelly Johnson, another former Ray, grounded into a forceout to put the Yankees up 2-0 in the first.
Ramos allowed two runs on four hits while striking out two in 2 2/3 innings. Of the 42 pitches he threw, 27 were strikes.
"Felt pretty natural to be back out there [starting]," Ramos said. "The ups and down, I was more curious to see how I would react to it. I felt fine the whole way."
Grant Balfour pitched the fourth for the Rays, and the new closer surrendered a two-run single to Brett Gardner that put the Yankees up, 4-1.
Adam Warren, who is in the hunt for the Yanks' fifth-starter spot along with Michael Pineda and David Phelps, surrendered an opposite-field home run to Zobrist in the first.
Warren allowed one run on four hits, and he struck out two in 2 1/3 innings. He threw 44 pitches, 25 for strikes.
"I thought it was a step in the right direction," Warren said. "I was a little ill about the walks last outing, so I really focused on that. I still want to get in better counts and get ahead of batters a little better, so I'll keep working on that. I still need to refine some things, and that's what Spring Training is for."
Yankees reliever Dellin Betances threw 1 2/3 innings of scoreless relief, but Jennings homered off New York righty Robert Coello with two aboard in the bottom of the fifth to tie the score at 4. Joyce added a solo homer to put the Rays up 5-4.
"Every time I go out there, I'm just trying to prove myself," Betances said. "That's all I can do, just do the best I can, come in and throw strikes and show them I could pitch. Hopefully I get a shot. That's all I try to worry about."
Up next: Masahiro Tanaka will make his first start and second appearance of the spring, as the Yankees take on the Phillies on Thursday at Bright House Field in a 1:05 p.m. ET contest on MLB.TV. Tanaka is scheduled to work three innings. Mark Teixeira and Alfonso Soriano are also expected to see their first spring action for New York, with Derek Jeter also along for the short trip. Kyle Kendrick draws the start for the Phillies.