Swisher was in New York's original lineup for the contest against the Blue Jays, but went through batting practice before being removed from the order.
"It's still a little sore, and with a built in off-day tomorrow, it's the smart thing to do," Swisher said. "With two full days of rest, it gives me the opportunity to rest the leg."
Swisher suffered the injury fouling a ball off himself in the seventh inning of Tuesday's 11-5 victory at Toronto. Swisher stayed in for two more pitches before being removed, leaving a two-strike count for pinch-hitter Brett Gardner, who struck out.
Yankees manager Joe Girardi said before the game that he was going to try to use Swisher, hoping he'd get through BP in playable condition.
"He said he's sore, but when you get running around and you get some adrenaline, sometimes things change," Girardi said. "It's not like we're dealing with a pulled muscle."
Center fielder Curtis Granderson moved to the No. 2 hole with Swisher nixed, while Gardner took over in left field, batting eighth. Austin Kearns switched from left field to right field.