What happens to Premier League surprise teams and where do they come from?

West Ham’s best season in the top division was when they finished third in 1985-86. The next season they finished 15th. Four times West Ham have finished sixth – in 1926-27, 1958-59, 1972-73 and last season. After those previous sixth-place finishes, what did they do? They finished 17th, 14th and 18th.

When a team have a good season nearly everyone thinks that next season they will do as well or better. More often than not they do worse. It is right to be pleased for a team when they have a good season, and to wish them equal or greater success next season. More often than not, however, those wishes are unfulfilled.

West Ham’s average finishing position in the top division is 13th. After unusually good seasons they had average seasons – actually a bit worse than average.

Scrolling through the channel listings on my television one night I saw a programme called Ten Things You Need To Know About The Future. I tuned in. The first words I heard were: “If present trends continue…” I tuned out.

Those words may have preceded more bad predictions than any other. When did this trend begin? It must have started at some point. Before this trend there was another, different trend. If anyone at that time had made a prediction about present trends continuing it would have been wrong. That trend ended. Will this trend not?

For West Ham this season there will be an extra competition. By finishing sixth in the Premier League last season they have qualified for the Europa League.

I studied the Premier League results of teams playing in the Europa League. I did so for the 17 seasons from 2004-05 to 2020-21, during all of which there was the same format for the Europa League. It has changed this season but not much.

There was a difference between teams who were used to playing in Europe and those who were not – between big clubs with big budgets and big squads and others. The first group comprised the Big Six: Arsenal, Chelsea, Liverpool, Manchester City, Manchester United and Tottenham. Everyone else was in the second group. I call the first group elite and the second group non-elite.

Non-elite teams playing in the Europa League finished an average of three places lower in the Premier League than they had the season before. Whether this drop was due to the Europa League is debatable.

Non-elite teams qualified for the Europa League with an average Premier League position of seventh. If you look at all teams who finish seventh in the Premier League you will find that in the next season on average they finish 11th – four places lower.

Only a few non-elite teams get into Europe. Most seasons most of them finish seventh or lower in the Premier League. Those who finish 17th or higher are not relegated and play in the Premier League again the next season. Typically the team who finished seventh had 20 points more than the team who finished 17th.

This makes it sound as though they are much better. The evidence of what happens next suggests they are not – that most of the differences in a season are due not so much to enduring disparities in skill as to ephemeral fluctuations in luck.

If you picked pairs of teams at random from those who finished a Premier League season between seventh and 17th, how often would the team who finished higher that season also finish higher in the next season?

If this season told us everything about next season the answer would be 100 per cent. If this season told us nothing about next season the answer would be 50 per cent. The actual answer is 60 per cent. For such teams, this season is closer to telling us nothing about next season than it is to telling us everything. Much closer.

Where did I get that figure of 60 per cent? I examined every possible pair of teams who finished a Premier League season between seventh and 17th and counted how often the team who finished higher in that season also finished higher in the next season. I did this over 25 pairs of seasons, starting with 1995-96 and 1996-97, ending with 2019-20 and 2020-21. For non-elite Premier League teams, I realised, the future tends to be the past jumbled up.

When I was a teenager I used to lie in bed at night trying to listen to Radio Luxembourg. It was a waste of time. I had a transistor radio. No matter how delicately I turned the dial I could never get much signal – just a bit of signal with a lot more noisy interference, high-pitched whistling.

Trying to listen to Radio Luxembourg it was painfully obvious that I was hearing only a little of the tunes I wanted and a lot of the tuneless atmospherics I did not want. Looking at football results it is not easy to make out what is signal and what is noise – how much is attributable to skill, which should be reproduced, and how much to luck, which should not. The evidence suggests that in the results of non-elite Premier League teams there is less signal than noise.

I started by talking about what happened to West Ham after an unusually good season in the top division. What had happened before?

Before finishing third West Ham had finished 16th. Before finishing sixth they had finished, in descending order, 14th, 16th, 18th (when that place was not relegated) and first in the division below. Abnormally good seasons were preceded, as they would be succeeded, by normal ones – strictly speaking, slightly worse than normal. West Ham shot up the table last season. Probably somebody else will this season.

Today's top sports betting stories

Follow us on Twitter @racingpostsport