Their 1-0 win over the Algerians on Sunday was their young nation's first World Cup victory. Their only previous appearance at the finals, in 2002, ended in three defeats.
"That means a lot to me and to everybody in the team," captain Robert Koren, who scored the goal, said after the match.
"For such a small country, to go to the World Cup is already a big achievement, but for us it's a big thing today because we got our first three points.
Koren, who has just helped his club West Bromwich Albion win promotion to the English Premier league, said he also took heart from Saturday's 1-1 draw between England and the United States, in which neither team looked outstanding.
"Both teams put a lot of energy into the game but there were a lot of mistakes from both teams too," he said. "We are really confident after these three points today and we will give them a good game."
First up are the Americans, in Johannesburg on June 18.
"Our minds are already on the U.S.," Koren said. "There is no time to celebrate too much."
The Slovenians will be hoping for at least a point from that match before they face England in what should be their most challenging game of the first round.
"England are a really strong team but you can be sure we will give them a good game and we are not afraid of them."