It was Small's third start of the season, and his shortest. He has not made it past the fifth inning in any of those games and his ERA is 9.67.
"You're only as good as your pitching, and we didn't pitch well enough to expect better results today," manager Joe Torre said.
Torre said Small had trouble with his location and couldn't keep the ball down in the strike zone. The Orioles made Small pay for those mistakes.
"I've got to work down," Small said. "I didn't work down, and you saw the results."
Small's spot in the rotation comes up again Friday, but it might not be him on the mound. It could be Shawn Chacon, who made a rehab start for Double-A Trenton on Sunday, and threw 72 pitches in five innings.
Depending on what the full report on Chacon is, who's missed 16 games with a left leg hematoma, he might be activated and placed back in the rotation.
With the loss, the Yankees finished with a 5-2 mark on their road trip and trail the Red Sox, against whom they begin a four-game series on Monday, by a half-game.
"We're anxious to get home and play," Torre said. "We had a good road trip. We wished it could be a touch better, but we played hard, worked hard."
Javy Lopez hit two home runs for Baltimore, while Kevin Millar and Brandon Fahey added solo shots. It was the most home runs allowed by New York in a game this season, and the first time the Yankees have given up two long balls in an inning.
Derek Jeter left the game in the sixth inning with a bruised right thumb after being hit by a pitch. X-rays were negative and Jeter is listed as day-to-day.
Torre said he took Jeter out of the game because it was the safest thing to do since the game wasn't close at that point, and the manager was hopeful Jeter would be ready for Monday's game.
With Jeter out, the Yankees lineup shifted all over the field, as Miguel Cairo moved from third base to shortstop, Andy Phillips went from first to third and Jorge Posada changed from catcher to first base.
Alex Rodriguez started for the first time in the series after being out with the stomach flu. He went 0-for-4 with a walk and said he is still feeling weak.
Rodriguez should be almost at full strength Monday and will probably play the field. Jason Giambi, who has missed the past two games with a viral stomach disorder, could be ready to play Monday, but he would likely not be able to play first base, Torre said.
Closer Mariano Rivera will be ready Monday after missing the last four games due to back spasms.