New year predictions may fail but our fondness for making them never does

One January a government department made a five-year plan. By December all of their predictions for the first 12 months had gone horribly wrong. Were they embarrassed? Did they give up forecasting? No. They made another five-year plan, with another set of predictions for each 12 months, and started all over again.

New year is when people make most predictions. I have a prediction about the predictions everybody else will make over the next few days. It is that most predictions for 2019 will be wrong but this will not deter anybody from making predictions about 2020.

Why do predictions go wrong?

The first reason is that people simply project present trends into the future. Even supposedly sophisticated people do this.

Academics at Oxford University predicted that in fewer than 150 years the 100 metres world record for women will be faster than for men. They plotted historical records on a graph and projected trend lines into the future. The lines cross in 2156.

Why stop there? Projecting the trend lines even further into the future suggests a time will come when the 100m world record for both men and women is less than zero – that athletes will finish a race before they start.

Sometimes a trend could not continue. Often it could but still does not.

After half the Premier League season Watford have 27 points and Crystal Palace have 19 points. If current trends persist then in the second half of the season Watford will gain 27 points or more and Palace will gain 19 points or fewer. Over the second half of the season Watford will beat Palace by at least eight points. Experience tells me they probably will not.

The second reason predictions fail is that we know what has happened up until now but we may not know why it happened. We think we do but we could be wrong.

Bring together two people with diametrically opposed core beliefs. Give them the same facts. Each will be able to put those facts into a story that is consistent with their core beliefs. Both stories will make sense. There is a difference between a story making sense and it being right.

In my experience the best-sounding explanation for anything is less likely to be true than almost everyone realises.

So let us think again about Watford and Palace. The best-sounding explanation for a points difference over half a season is that one team are better than the other, full stop. Usually, though, a big difference in points between non-elite Premier League teams over the first half of a season is not caused primarily by a big difference in how well they played – or if it is then in the second half of the season there is not such a big difference in how well they play.

Looking back at past Premier League seasons I see that teams who gained 27 points in the first half of a season averaged 23 in the second half while teams who gained 19 points in the first half of a season averaged 22 in the second half. Over the first half of the season eight points separated them. Over the second half the average difference was one point.

In the second half of this season, I think, the most likely outcome is that Watford will beat Palace by one point.

Every reason why predictions go wrong can be gathered under one general heading. Unforeseen circumstances. Why has one of my predictions gone wrong? My prediction failed because of unforeseen circumstances. Something happened that I did not expect to happen. How could I have seen that coming?

This might sound like another way of saying my prediction failed because my prediction failed. And in one sense it is. But in another sense it tells us more.

Most predictions are wrong. So there must be lots of unforeseen circumstances. Much more happens than anybody can foresee when they look at present trends, and even with hindsight they might not know how or why those trends developed. Happy new year, whatever it brings.

Doubt is better than certainty

Suppose somebody wanted to make fewer bad predictions. What should they do?

All predictions that fail do so in one way or another because of unforeseen circumstances. As I have just said, there are lots of failed predictions, which must mean there are lots of unforeseen circumstances.

So be less confident that you know why present trends emerged, and less confident that they will persist. Philip Tetlock of the University of Pennsylvania has spent more than 30 years studying predictions, good and bad. He says the best predictions come from people in doubt.

Nate Silver is famous for accurate predictions of US elections. He broadly agrees. “I’ve spent a lot of my life studying probability and uncertainty,” he said. “Cover these topics for long enough and you’ll come to a fairly clear conclusion: when it comes to making predictions, the world usually needs less certainty, not more… Most experts – including most journalists – make overconfident forecasts.”

Joan Baez sang a song with the line: “Every day that passes I’m sure about a little bit less.” That is the way to go.

Read every day for no-nonsense previews and expert sports betting tips

Today's top sports betting stories

Follow us on Twitter @racingpostsport

Like us on Facebook RacingPostSport