Are preseason college football polls stupid? I know that's a mean word, but just how much do coaches and sports writers know about teams before the season starts?
The preseason AP and Coaches polls have a long tradition and help fans deal with the hot month of August before games start. But most people scoff at the notion that these polls can accurately evaluate teams.
Matt Hinton, a college football writer at Grantland, has questioned the usefulness of preseason polls here at Football Study Hall. He found that the preseason AP poll has no ability to predict final AP poll.
So screws the preseason polls. Let's give these coaches and sports writers a few weeks to watch games. Then we can find value in these polls for determining team strength.
However, this conventional wisdom has it backwards. The preseason polls are remarkable predictors of team strength. Later polls lose this predictive power.
Let me show you.
How well do preseason polls predict the winner of bowl games?
To test the predictive power of a ranking, we can ask how often the higher ranked team wins a bowl game. With no home field advantage for either team at these neutral sites, the higher ranked team should win.
For the preseason polls, I rank teams beyond the top 25 based on the number of points. For example, this gives 44 ranked teams in the 2015 preseason AP poll, as California, Western Kentucky and West Virginia each earned 1 point.
Ranked teams are predicted to beat unranked teams, while no prediction is made for two unranked teams. From 2005 to 2014, college football gave us a reasonable sample size of 339 games to evaluate these polls.
The preseason AP poll predicted 58.8% of bowl game winners (154-108 with no prediction in 77 games). The preseason Coaches poll did even better at 59.9% (163-109 with no prediction in 67 games).
To put this in perspective, consider the team favored by the closing line in the gambling markets. Todd Beck's data at The Prediction Tracker gives an accuracy of 61.5% (208-130 with no prediction in one game).
The preseason polls are remarkable predictors of team strength. Not bad for a bunch of sports writers and coaches that haven't seen a single play of football.
The accuracy of polls later in the season
We can also ask about the predictive power of these polls later in the season. For each season, I took the final poll before bowl season. The pollsters have had a regular season and conference championship games to refine their opinion.
The final pre-bowl AP poll has predicted 56.0% of bowl game winners (130-102 with no prediction in 107 games). The pre-bowl Coaches poll does worse with an accuracy of 55.4% (129-104 with no prediction in 106 games).
The predictive power of the pre-bowl polls is much less than the preseason polls. This visual shows the results:
The pre-bowl polls also make a prediction in fewer games than the preseason polls. For example, the preseason AP poll made a prediction on 77.3% of bowl games. This drops to 68.4% by bowl season.
This implies that pollsters are dropping good teams from the rankings that make and win bowl games. We'll see an example from 2014 later.
The wisdom of crowds
The predictive power of the preseason polls almost certainly comes from the wisdom of crowds. No one pollster can make a perfect preseason poll, but the collection of many ballots cancels out the effect of small errors made by each one.
However, this wisdom of crowds goes to shit once the season starts. A team usually drops in the polls after a loss no matter the opponent or margin of victory.
I'm not blaming the pollsters for these choices. Fox Sports writer Stewart Mandel passed off his ballot in the AP poll to a colleague out of frustration. He said "I found it to be largely thankless and futile. For one thing, at that time more and more big games were moving to primetime, so I'd often find myself getting out of a stadium at 2 am and having to then do my ballot."
Let's look at two bowl games that show the power of the preseason polls.
Alabama v Notre Dame in 2012
Let's go back to the 2012 and the national title game between Alabama and Notre Dame.
Alabama had a typical Nick Saban year. They lost one game to Texas A&M and Johnny Manziel but still won the SEC and made the title game.
Notre Dame had a magical season, as they went 12-0 during the regular season. However, they did struggle at times. The Fighting Irish needed some generous officiating to beat Stanford in overtime and went 5-0 in games decided by less than a touchdown.
While some thought Notre Dame had a chance against Alabama, the preseason polls thought different. Alabama began the season 2nd in the both the AP and Coaches poll. Notre Dame started 24th and 26th in the Coaches and AP poll, respectively.
Alabama showed their talent gap over Notre Dame by beating them 42-14.
South Carolina v Miami in 2014
For another example, consider the 2014 Independence Bowl between South Carolina and Miami. South Carolina's defense had struggled all season. In contrast, Miami had strong efficiency numbers on both offense and defense that made them a 3 point favorite.
However, South Carolina started the season 9th in both the Coaches and AP poll. They went 6-6 and dropped from both by the beginning of bowl season.
Miami got votes in both preseason polls but couldn't crack the top 25. They also didn't make either pre-bowl poll with their 6-6 record.
South Carolina beat Miami 24-21.
Predictions
It's surprising that preseason polls have more predictive power than late season polls. I would have never thought to do this study had I not read that Nate Silver uses the preseason AP college basketball poll in his tournament predictions.
Since the preseason polls predict bowl games so well, they can also help during the season. I now incorporate this data with my preseason model in The Power Rank's college football predictions.
Over the past two seasons, my preseason model has predicted the winner in 71.2% (1066-431) of games between FBS teams in 2013 and 2014. With data from the preseason polls, this accuracy increases to 72.1% (1080-417).
I save these predictions for members of The Power Rank, but you can get a sample of them in my free email newsletter. To learn more, click here.
Ed Feng has a Ph.D in Chemical Engineering from Stanford and started The Power Rank. He's still angry about Stanford's loss to Notre Dame in 2012.
Loading comments...