Why couldn't tech predict the US election results?

 Machine learning and AI 

Such sentiment analysis, however, comes with a heavy workload and also requires mathematical models. "There are three ways to make improved predictions – a better model, better data, and more data," says Jeremy Perlman, VP Europe for Trifacta, which helps RBS, Santander and PepsiCo analyse data. "The problem is that data created on social media and the web is expanding at a ridiculous rate, so machine learning will be critical to making better predictions at massive scale.”

Jeremy Perlman, VP Europe for Trifacta

Jeremy Perlman, VP Europe for Trifacta

Since computing power is increasingly exponential with the birth of super-computing in the cloud, the need to analyse more and more data shouldn't be a major hurdle. "Computational devices can very effectively, with high precision and rapidly, gather millions of tweets, posts or similar and run sentiment analysis – to understand likes and dislikes," says Jepson. "This, together with data from polls, will increase the precision of predictions." 

During the EU Referendum vote in the UK, cognitive technology company Expert System and the University of Aberdeen analysed a sample of 5,000 tweets collected on June 20-21 to uncover voting intentions. It found that 64.75% of tweets from the UK were inclined to leave the EU. That overstated the final result by over 10%, which leads to an obvious conclusion; voters who preferred to leave the EU may merely have been more active on Twitter.

"For it to be possible for AI to predict an election outcome, one would need to analyse every available data variable that could affect a conscious human decision – including the weather forecast and historical data – in order to predict what the crowd will do," says Dmitry Bagrov, MD UK of DataArt. "Any predictive platform would require access to every possible data variable and the capability to process this mountain of information."

Could AI and machine learning be used by pollsters?

Could AI and machine learning be used by pollsters?

'Shy voter' problem 

What if people lie to pollsters, or decide to vote when they usually don't bother? The former feeds false data into models, and the latter comes without any data. "Pollsters have to contend with an increasing phenomenon known as ‘shy voters’; these are voters who don’t want to say who they’ll be voting for out of fear and/or embarrassment," says Andrew Cameron-Webb, founder of AI-based social media management platform WeLikeIt.

He thinks shy voters were a big factor in the Scottish independence and EU referendums as well as in Trump's success. "Pollsters need a more complete solution, a system that can track millions of separate data points on Twitter, Facebook, Google and YouTube, constantly monitoring public engagement and excitement about the candidates." He calls it a 'system that never sleeps’.

If social media can predict elections, why bother counting votes?

If social media can predict elections, why bother counting votes?

Small margins 

Although he was one of many pollsters that got it wrong, Nate Silver underlined just how difficult it is to forecast elections in his post-election blog by investigating what would have happened if just one out of every 100 voters shifted from Trump to Clinton. "That would have produced a net shift of 2 percentage points in Clinton’s direction," he writes, "giving her a total of 307 electoral votes … and she’d have won the popular vote by 3 to 4 percentage points, right where the final national polls had the race and in line with Obama’s margin of victory in 2012." 

The lines are very fine in voter forecasting, but the use of sentiment analysis to constantly monitor the change in public mood could result in terrifying tech-led constitutional change. For if social media can be used to predict elections, why not use it to automatically elect a presidential candidate, or decide a referendum? Or even to justify contentious decisions when in office, using it as an ongoing and refreshed mandate?

"We’ll soon be capable of determining the result of an election with extremely high precision and accuracy," says Jepson, who wonders if an election result could be produced just by sentiment analysis of social media. We use them to choose what news we read and what music we listen to, so why not elect politicians by algorithm? "Who knows, maybe the election process itself will become irrelevant in the future."