The debacle that pollsters faced in the Bihar assembly elections is in part being blamed on the small sample size and lack of advancement in field surveys, says Sahil Makkar
A senior research associate went on leave and a junior failed to check the result template in his absence before announcing the predications. This was the explanation Today's Chanakya offered for getting the Bihar assembly elections wrong in its exit poll survey.
The market research firm, which has been in the poll forecast business for the last 18 years and attracted nationwide attention after its accurate prediction of the 2014 Lok Sabha elections, had forecast 155 seats (with a margin of 11) for the Bharatiya Janata Party and its allies and 83 seats (with a margin of nine seats) for the grand alliance in the Bihar polls.
The actual results, however, were completely the opposite - the grand alliance won 178 seats, while the BJP-led National Democratic Alliance could notch up only 58.
"A simple computer template coding for marking the alliances got interchanged at our end. Due to this, our seat numbers remained the same but the respective alliances got interchanged," Today's Chanakya said in its apology.
Others too went horribly wrong with their exit poll surveys. And Bihar was not the first time this has happened. Year after year, pollsters have failed to gauge the mood of voters in the country.
Amazingly incorrect
Way back in 2004, every pollster had predicted that the BJP would win the general elections and return to power. The results, which led to the United Progressive Alliance government in New Delhi, left everyone baffled.
Jai Mrug, political commentator and psephologist, says there have been cases of overestimating the BJP's performance in recent times. This was evident in the recent Maharashtra and Jharkhand assembly elections where most pollsters predicted large number of seats for the party and it fell short of a simple majority.
One reason could be the small sample size. But then how big a sample is sufficient? Hansa Research, which works with NDTV, failed to predict the mood of the Bihar voters despite having conducted "computer aided personal interviews" of over 75,000 people to "ensure better data quality": it gave the National Democratic Alliance 120-130 seats and the grand alliance 105-115 seats. The exit poll results were nowhere close.
Yogendra Yadav, who quit the business of psephology to pursue a political career, says pollsters usually get the data right but they err when they are asked to forecast seats, which is based on field surveys and vote-to-seat conversion ratio. The last advancement in these areas were done 30 and 20 years ago, respectively, he says. "This industry needs dire investment in research and development."
Yadav believes that poll forecasters should only give the vote shares and avoid predicting the number of seats a party might win -as they do in the West.
"But the television channels say percentage point doesn't dish out enough masala to their viewers." That news television is highly competitive adds to the pressure on channels to spice up things.
At the same time, the news channels don't have the expertise and capacity to scientifically crunch the numbers, so they depend on market research firms.
Other than Today's Chanakya and NDTV's Prannoy Roy, no one has offered an apology to their viewers for the Bihar fiasco.
"The last time we made an error as big as this was about 32 years ago when NT Rama Rao won a landslide. Since then we've never made such a big error. This time, like before, we tried our best with a large sample size and we went to every single constituency," Roy said in his apology.
"But the data from the fieldwork agency, normally very reliable, was incorrect and this happens. We are looking into why it went wrong."
Roy and other editors didn't respond to a query asking if they would desist from engaging pollsters in the future.
The race against time
The Bihar elections saw another inexplicable mistake by some television news channels: many within an hour of counting claimed that the NDA was forming the government in Bihar. Roy apologised for this mistake too, and blamed Nielsen, the research agency, for it. Nielsen denies the charge that its data was wrong.
"As is our process, on result day, our staff at the various counting centres collects the latest updates as and when they are announced by Election Commission officials. These updates are then relayed to our hub in Delhi to be shared with all the subscribing channels. The same process was followed on November 8 as well, and the data provided to all channels was identical," says a Nielsen spokesperson.
"Nielsen is only a conduit in the whole process. Therefore, it would be incorrect to say that data provided was wrong," the person adds.
Roy's rivals in two leading television channels disagree with him -- they believe that the data from Nielsen was correct. "We have no reason to believe that the data was incorrect. We kept telling our viewers that these are early trends and the whole picture is yet to emerge. We never jumped the gun," says the input head of a television news channel.
Both the news channels relied on their respective reporters who had visited Bihar. "Exit polls got this one wrong. Journalists got it dead right mostly. Many of us openly said on air that Bihar Chief Minister Nitish Kumar was well ahead," says the second person.
On counting day, Nielsen provides the same data to all television channels by deputing its personnel at each counting booth. It is faster by 20-25 minutes than the data uploaded on the website of the Election Commission. It is very likely that television channels ended up extrapolating the early trends backed by the results of their exit poll surveys and announced the NDA as the winner.
Insiders and experts say there should be transparency in disclosing the data and television channels should also review their processes.
"Exit polls, if scientifically done, are still useful to understand voter behaviour. Maybe all channels and pollsters need to come together and find a better way to create a more transparent polling methodology and collection of data, maybe even take a temporary moratorium till this is achieved," says Rajdeep Sardesai of the India Today group.
The matter is now before the News Broadcasters Association for review.