Swisher was in the Yankees' original lineup, but manager Joe Girardi elected to revise his batting order about an hour before game time.
Having battled the knee issue since Aug. 24 in Toronto, when he fouled a ball off his knee, Swisher had said he saw some improvement but re-aggravated the injury during the Bombers' last homestand.
"I don't like going out there, running around with a limp all over the place," Swisher said. "I can't do that. It's going to start affecting other parts of my body."
Swisher said that he went back to square one when he spun and fell during an at-bat on Wednesday, a contest that later saw him hit a game-winning, walk-off home run to sink the Orioles -- the Yankees' only win in their last seven games.
Yankees manager Joe Girardi said that Swisher has no further tests scheduled, but Swisher said that he does not feel comfortable playing with the knee as is.
"I can't go chasing after a ball in the gap with one leg," Swisher said. "That's not going to do anybody much good. I thought it would get better a lot faster than it has."
Greg Golson was inserted in right field, batting ninth, and went hitless in three at-bats. Alex Rodriguez also received a scheduled off-day, spelled at third base by Eduardo Nunez.