Gardner has been on the 15-day disabled list with a right elbow injury he sustained in an April 17 game against the Twins. Swisher has been sidelined with a left hamstring injury since he tweaked it on Sunday against the Tigers.
Gardner is eligible to come off the disabled list on Thursday, but because Girardi wants the left fielder to play in a rehab game or two, the skipper said it seems more unlikely that Gardner will be back in the lineup for Thursday's series opener in Kansas City.
"He's supposed to do more stuff today," Girardi said on Tuesday. "We're going to see where he's at after today. We're starting to get to the point where Thursday might be a little doubtful, just because he's going to have to go play a game or two."
The Yankees have been using reserve outfielders Andruw Jones and Raul Ibanez with the two starters out, and Eduardo Nunez earned his second start in left field in as many days on Tuesday.
As for Swisher, Girardi said the Yankees are also going with a conservative timetable to wait and see how his body responds to running the bases and other baseball activity.
"If you don't do anything for two or three days, and then you go out and run the bases, it's kind of hard to declare him ready," Girardi said. "One of the reasons that we give somewhat of a conservative timetable sometimes on guys is, if all of a sudden you say he's going to be ready in three days or four days and they're not ready, then red flags go up everywhere. ... I believe we'll have him back Tuesday. If we get him back sooner, would I be happy? Absolutely."