Brad Williams was named as Gillespie's replacement for the triangular one-day series with India and New Zealand.
"I pulled up sore this morning following a light bowl... and I came to the decision that I shouldn't go to India," Gillespie said.
"It's not fair on anyone if I went on tour and wasn't 100 per cent fit."
Gillespie strained a side muscle in last week's first test against Zimbabwe and was ordered to miss the second Test, which started on Friday.
The injury-prone paceman was included in a 14-man squad to go to India but only on condition he passed a fitness test.
"It's fair to say that
Gillespie is the third Australian player to pull of the tour, following injured paceman Glenn McGrath and middle-order batsman Darren Lehmann.
The world champions leave for India on October 22. The tournament will be held from October 23 to November 18.
Australia - Ricky Ponting (captain), Adam Gilchrist, Michael Bevan, Andy Bichel, Nathan Bracken, Michael Clarke, Ian Harvey, Matthew Hayden, Brad Hogg, Brett Lee, Jimmy Maher, Damien Martyn, Andrew Symonds, Brad Williams