Even with the best methodology there is a limitation to what opinion poll/exit polls can achieve.
When done correctly, the polls should give the broad story, points out Rajeeva Karandikar, director, Chennai Mathematical Institute.
As the Bihar counting was going on and by Tuesday evening, it was becoming clear that the unfolding story was far from what was dominating most TV channels the preceding two days -- that Tejashwi Yadav was winning Bihar elections hands down.
Several friends called me that evening and the next day asking me as to why all the exit polls flopped? Of course, I was not involved in any opinion or exit poll for Bihar this year, but had been following the same keenly.
Well, firstly, not all exit polls flopped.
Among the mainstream TV channels, the ABP News-C-Voter exit poll predicted that the National Democratic Alliance will get 104-128 seats while the Grand Alliance led by the Rashtriya Janata Dal was likely to win 108-131.
But the range was a bit too wide for a house with 243 seats. The actual numbers were 125 for NDA and 111 for the Grand Alliance, which were within the predicted range.
Perhaps the same results were also telecast on Times Now as point estimates -- 116 for NDA and 120 for Grand Alliance.
While C-Voter did convey that it was a neck to neck fight, the prediction was that it was advantage Tejashwi Yadav, while in the end it was advantage Nitish.
The Republic TV-Jan Ki Baat exit poll gave slightly less seats for NDA: 91-117, and more seats for the Grand Alliance, 118-138. This was off the mark, but not too far.
But two other polls were out late that evening: The India Today-Axis poll predicted 150 seats for the Grand Alliance and 80 seats for the NDA; News18-Chanakya projected 180 for the Grand Alliance and 50 for NDA.
In public memory, both Chanakya as well as Axis had accurately predicted the 2019 seats for the NDA -- about 350 seats.
Chanakya had also predicted the 2014 Lok Sabha seats for the NDA correctly.
Perhaps because of that, most channels including the channels that had engaged their own exit polls started focusing on the Poll of Polls and giving a huge victory to the RJD and allies.
Indeed, all the discussions on TV channels were assuming that the outcome was a forgone conclusion, and analysts started discussing why Nitish lost and so on.
So first and foremost, what was wrong was the excessive hype created by the electronic media about opinion polls, including THE exit polls, as if it is the truth, the whole truth and nothing but the truth.
Calling politicians to the show and insisting that they should respond to the exit poll results does not add any value -- those who are predicted to lose can keep questioning surveys and those who are predicted to win can grin and say that of course we know this.
All the time on the show is spent on this bickering and there is no time to discuss methodology etc, even when the analyst is willing to share the information.
Indeed, polls such as the ones by CSDS, where I was involved, had so much more insight which could never be shared on air. The CSDS team used to write all this and put it on their Web site or publish it in a newspaper the next day or EPW subsequently.
The fact is that even with the best methodology there is a limitation to what opinion poll/exit polls can achieve.
When done correctly, the polls should give the broad story -- like the picture captured by C-Voter and Jan Ki Baat polls, that it is a very close election.
Since the details of methodology used for the data collection and analysis, or even the vote share estimated by them, is not available, it is very difficult to see as to where the Axis and Chanakya polls went wrong.
Here is a general comment that applies to all exit polls. The pre-election polls done, say, weeks ahead of the polling day have two limitations as far as the power to predict the final outcome of the elections is co,ncerned.
First, the opinion poll can at best measure the opinion of all the voters, while what matters is the voters who actually go and vote on the polling day, which is about 60%-70% of all eligible voters.
The second reason is that the best a poll can measure is the opinion at the time the poll is conducted, and we know (from discussions as well as from surveys) that public opinion is very volatile as the election approaches.
To give one concrete data point, in 1998 we had the funds to undertake the following exercise -- a nationwide opinion poll just days before the first phase.
The poll was in three phases, and we were able to talk to the same respondents a day after they had voted in their respective phase.
So the gap was about six days for constituencies that were in the first phase, 12 days for the second phase and 18 days in the third phase, so the average gap was about 12 days.
In the pre-election poll the question was, if election is held tomorrow whom will you vote for?
And then in the day-after poll, the question was, whom did you vote for yesterday?
And during this period of 6 to 18 days, about 30% respondents had changed their preference.
Let me add that the estimated vote shares in the day-after poll, which we got just before the counting started, were almost on the dot.
The maximum churn was in the third phase.
Thus, any pre-election poll should not be taken as predicting an election outcome.
Exit poll addresses both these issues -- it seeks responses only from those voters who have just voted.
However, now another issue enters the picture. In this environment, when the enumerator is standing outside the election booth, it is not possible to choose the respondents from a given list.
Thus, proper randomisation is not possible. At best, only the booths where respondents are to be interviewed can be chosen via randomisation.
The choice of whom to interview at a booth has to be left to the investigator. This often introduces bias in the sample.
Given that more investigators are males, along with the fact that in India people are hesitant to talk to a stranger of the opposite gender, it is expected that the sample in an exit poll is biased against women voters.
Analysts do try to correct for this, but in this way other biases may be entering.
During the period I worked with CSDS, we preferred the day after poll, or post-poll where we could do a door to door poll and choose respondents following proper methodology, and our experience with predictions based on post-poll had been very good.
Of course, in a poll which is this close (the gap between NDA votes and the Grand Alliance votes is miniscule, less than 0.05% votes), it is impossible to get very precise seat estimates.
But error of the magnitude seen in the report by Axis or Chanakya is not explainable by error margin. It shows that there were severe shortcomings in those surveys.
