With Rodriguez replaced in the cleanup spot by the light-hitting utilityman Ramiro Pena, the Yankees were limited to three runs over six innings by right-hander Jake Arrieta, who earned the victory in his Major League debut.
Burnett was handed his first loss in eight career starts at Camden Yards, allowing four runs on eight hits in 6 2/3 innings.
"It was a tough game," said right fielder Nick Swisher. "We had some chances to put some runs on the board and couldn't get it done. That's going to happen. They did a good job taking advantage of the situations that they got."
Fighting to keep afloat without terrific command of his fastball and curveball, Burnett allowed the Orioles to push ahead in the sixth inning.
Luke Scott connected with a curveball for a one-out fly ball and initially slammed down his bat in disgust, but the ball kept carrying, sending Swisher back to the right-field wall.
Swisher couldn't corral it and it kicked away. Scott chugged into third base before scoring the eventual winning run on an Adam Jones double to right-center.
"I was right on the wall, and when it was coming down, I was like, 'Oh, I've got this,' " Swisher said. "The next thing you know, I was looking for the ball. I've got to check the replay and figure out what happened."
"I thought [Scott] popped it up, too," Burnett said. "I guess he's stronger than we all think, you know? I saw his reaction, and the next thing you know he's on third."
Burnett surrendered two quick runs in the first inning, hitting a pair of batters while allowing a Ty Wigginton sacrifice fly and Jones' RBI single. Scott Moore got to Burnett for a game-tying solo home run, his first of the season, in the fifth inning.
"They're a swinging team. They come in swinging," Burnett said. "You've got to get ahead on guys like that because if you're pitching behind, aggressive teams are going to get you."
Pitching into the seventh, Burnett recorded the first two outs before allowing a pair of hits and yielding to lefty Damaso Marte. Burnett walked one -- intentionally -- and struck out five in taking just his third career loss in 15 starts against the Orioles.
"A.J. battled. He didn't have command of his fastball tonight, and he had to go to his curveball and changeup," manager Joe Girardi said. "He did a good job of keeping us in the game. Unfortunately, they scored a couple more runs, but without a lot of command of his fastball, he kept us right in the game."
Having outscored the Orioles, 65-29, in their previous 11 meetings this season, the Yankees had a rare quiet night against Baltimore pitching.
That had a lot to do with Arrieta, who was ranked as the Orioles' No. 4 prospect by Baseball America entering this year and was leading the International League with a .189 opponents batting average at the time of his recall.
"I was just excited, really excited, knowing that I was going to be tested right out of the gate by one of the best teams in all of baseball," Arrieta said. "And to be able to throw well against these guys, it boosts confidence."
Yankees catcher Chad Moeller, who started the year with the Orioles, was happy that Arrieta had a chance at the big league level.
"I figured it would come at some point soon," Moeller said. "He threw the ball well and looked under control out there. He didn't seem frightened; what was going on inside, who knows? But I thought he handled himself well."
Curtis Granderson drove in the red-hot Robinson Cano with a RBI triple in the second inning, and Moeller scored all the way from first base on Derek Jeter's third-inning double. That was followed by Swisher's sacrifice fly, which gave the Yankees a one-run advantage at the time.
But after Marcus Thames fanned on Arrieta's final pitch to leave the bases loaded in the sixth inning, Rodriguez's early exit would hurt the Yankees in the seventh.
Will Ohman issued walks to both Swisher and Mark Teixeira with two outs in the frame, bringing up the cleanup spot. Girardi called back Pena and sent up Francisco Cervelli as a pinch-hitter, but Cervelli struck out swinging to kill the threat, offering a reminder of how much the Yankees might miss A-Rod if the injury lingers.
"I think that's everybody's biggest concern," Swisher said. "I think we've done a great job of playing with guys injured and stuff like that. If that's the case, we'll have to step it up."