Newcastle United's Dyer denied any involvement in an alleged gang rape in a central London hotel in October. No charges were brought following the allegations.
"The Grosvenor House scandal set me back," Dyer said in an interview at England's training camp in Sardinia.
"I was nowhere near (the room)... When I was named in that, I was devastated.
"There were times when the (Newcastle) chairman and manager would have to get someone to bring me in for training because I just didn't want to train.
"The newspapers are so powerful, people were believing them. I remember going in for a meal in Newcastle and a fan coming up and saying 'I'm going to rip your head off. How can you do that to a girl.'
"I was devastated, honestly, but I put a brave face on it.
"Every away game I was dreading it because fans were calling me a rapist ... It's not nice to be called something like that when I wasn't
"That was hard. That set me back and in the second half of the season I had a hamstring problem."
It did not help Dyer, who has 20 caps, that he had a reputation as a playboy and for enjoying the trappings of a footballer's lifestyle.
"It is annoying when people say I am the leader of the brat-pack. It's not true, especially this season. I haven't gone out as much as I used to. That's something I have always been labelled with," Dyer said.
"But one of the problems this season with that Grosvenor House thing going on, I did lose a bit of my cockiness on the pitch and my youthful exuberance, which are strengths," he said.
"People say I am cocky because I have two cars and a diamond watch but that means 90 percent of footballers are cocky as well.
"Everyone has got them. We are in a fortunate position. We can afford these things."
Dyer, an exciting right-sided midfielder, is unlikely to make the starting line-up for England in Group B against Switzerland, champions France and Croatia.