Jankovic saved a match point before pulling off a thrilling 4-6, 6-3, 7-6 win over the world number one to set up a showdown with Spaniard Maria Jose Martinez Sanchez, a 6-4, 6-2 winner over former French Open champion Ana Ivanovic.
Williams had been the favourite to win the Rome title but came unstuck on the red clay against a tenacious Serb who refused to surrender to the world number one.
"I completely choked. I really should have won," the 12-times Grand Slam champion told a news conference.
"I wasn't unlucky. It was all on me. I think I had an apple in my throat."
Williams, playing her first tournament since winning January's Australian Open, twice failed to serve out for the match.
But the moment she will keep replaying in her mind was when she was one point from victory at 5-4 in the third set. Instead of sealing her place in the final, Williams squandered her chance and was left to rue a missed opportunity.
"I should have won. I was making errors on the key points," said Williams, who will be aiming to win her second major of the year when the French Open starts later this month.
"I feel I might show up tomorrow and play. I don't feel I lost. I don't think it's set in."
The victory handed twice champion Jankovic her second win over a Williams sister in as many days after she dropped just one game against fourth seeded Venus on Thursday.
CLAYCOURT MACHINE
"Beating two Williams sisters in two days is amazing. I'm really proud of myself," beamed Jankovic.
Her win had set the scene for a possible all-Serbian final but surprise package Martinez Sanchez played spoilsport by whipping 2008 Roland Garros winner Ivanovic.
The contest between Jankovic and Williams got off to a scrappy start, with the players trading breaks twice amid a raft of double faults and unforced errors.
When things settled down, Williams unleashed three cracking winners to put Jankovic under pressure on serve in the eighth game.
The seventh seed held out on that occasion but Williams grabbed the break she needed in the last game of the set after her opponent mishit a shot in the decisive point.
The Serb came back strongly, giving the American no peace on her serve and breaking twice to claim the second set.
Jankovic played catch-up throughout the final set but she refused to lie down.
She broke twice to stay in the match before winning five consecutive points to come back from 5-2 down in the tiebreak and stay on course for her third Italian Open title.
Despite the defeat, Williams said her form this week had given her hope of winning a second major this year when the French Open begins later this month.
"I'm sliding like a machine," she said when asked about her claycourt game. "I feel very good."
French Open could leave Roland Garros
Mauresmo falls in Rome opener
Serena Williams retains her Australian Open title
Venus and Serena Williams win US doubles crown
Serena Williams fined $175,000 for US Open rant