He made the remarks during a reception in honour of the Prime Minister's XI cricket match Monday.
Mathieson, a men's health ambassador, was encouraging the crowd to get prostate checks.
He then suggested that men should look for a small, female, Asian doctor to perform the examination.
"We can get a blood test for it, but the digital examination is the only true way to get a correct reading on your prostate, so make sure you go and do that, and perhaps look for a small, Asian, female doctor is probably the best way," he said.
Mathieson later admitted it was a joke in poor taste.
“My comments last night were trying to raise awareness about prostate cancer and the need for men to have regular checks and the importance of early detection," he was quoted as saying in the local media.
"It was meant as a joke and on reflection I accept it was in poor taste. I apologise for any offence caused," he said.
Meanwhile, Prime Minister Gillard defended her partner, saying Mathieson worked hard to raise awareness of prostate cancer, but he did the right thing by apologising.
"Obviously there are various ways of getting that message across, but he's acknowledged that the joke cracked last night was in poor taste," she said.
The remark generated criticism and Australian media suggested that the Opposition Leader Tony Abbott called on the prime minister to respond to Mathieson's remark.
"After everything the prime minister has said on these sorts of subjects, I do think that she does have to deal with this personally," he said.
The head of the Men's Shed movement, David Helmers, said Mathieson's joke was "inappropriate", but paid tribute to his work for the organisation.
Chief executive of the Prostate Cancer Foundation Anthony Lowe described Mathieson's joke as "a bit unfortunate".
"Tim, by the way, has been a great supporter of men's health -- both the Men's Shed movement and also (for) prostate cancer for a long period of time - and we're grateful for his support," Lowe said.
Image: Australia's PM Julia Gillard and her partner Tim Mathieson in Honululu
Photograph: Jason Redd/Reuters
Obama No 1, Manmohan No 19 on top tweeting leaders list
Aussie FM channel defends itself on prank call
US school shooting shocks the world
Aus to begin uranium sale to India after negotiations