Swisher, who had started 41 of the Yankees' first 44 games, sat out Monday's series opener against the Blue Jays. Chris Dickerson started in his place in right field.
For manager Joe Girardi, Swisher's struggles -- especially from the left side of the plate -- triggered the decision to give the right fielder an extended rest.
"His left-handed swing has been a struggle," Girardi said of the switch-hitter. "I figured it would be a good time to give him a couple of days to just take a deep breath. He's important to us."
Swisher is hitting .214 with a .330 on-base percentage and two home runs in 176 plate appearances. As a left-handed batter, those numbers drop to .170 with a .305 OBP, though both homers have come from that side of the plate.
"I don't know, I'm not really a numbers guy," Swisher said. "I don't really check it out. Obviously, left-handed I don't know if I'm doing that well. But hey, so what? I know I can hit left-handed, I've done it, I've proved it year in and year out. So I'm not really worried about it."
For his career, Swisher is a .251 hitter with a .356 OBP, but his power has been glaringly absent this season. The owner of a lifetime .462 slugging percentage, he has posted a .303 mark this season.
"Obviously, when you're struggling as a player, it bothers you," Girardi said. "I don't care who you are, it bothers you. His personality has been upbeat. For the most part, he's been the same guy, but I know it bothers him."
Swisher did nothing to hurt his reputation during a pregame meeting with reporters, but when asked if he agreed with Girardi's decision, he called the point moot.
"Not my call, man," he said. "I stand behind my manager. Sometimes, other people know what's best for you rather than you. I've kind of relaxed a little bit. Made sure I got all my stuff in, all my work in. [I'll] just take a couple of days and get back in there tomorrow."
With the Yankees set to face lefty Ricky Romero on Tuesday, Swisher will likely return to the lineup.
"I feel pretty good, man. I feel like I'm making good contact," Swisher said. "Like Yogi [Berra once] said, 'I'm not in a slump, I just ain't getting no hits.' I guess I should just take a page out of his book."