Apparent Talent, Team Quality and Coaching Effects

As a longtime (albeit recovering) recruiting addict, I have often wondered just how closely the talent accumulated by a football program is reflected in the results on the field. I certainly knew that there was some general correlation between strong recruiting and team strength, after all USC and Florida are usually pretty strong teams, whereas Eastern Michigan and Idaho are, well, Eastern Michigan and Idaho. But I wanted to really look at how much teams’ accumulated talent, per the recruiting services, explained their success (or lack thereof) on the field.

In order to study this question I downloaded and tabulated recruiting data for all FBS teams from 2002 through 2012. Ideally I would include multiple sources of recruiting data and calculate a consensus recruiting rating, but given how labor intensive that step was I utilized only the ratings from Scout.com to start with. Once I had the data all arrayed and scrubbed, I calculated average star ratings for each program each year. I decided to utilize average star ratings rather than class rank because class rankings are heavily influenced by the number of commits in a class. Since I would be looking at a four year average of the talent, I wanted to just look at the "pure talent" without regard to lumpiness in the size of the individual classes. I calculated two measures of what I am calling Apparent Talent; the first was a simple four year moving average of class star averages. The second measure was one I felt was slightly more realistic: a weighted average that gives more value to classes three and four years ago, to account for the fact that upperclassmen usually start and play more than younger players.

I used FO's F/+ rating as an objective measure of team quality. I felt this would be a more appropriate measure than something like winning percentage, since I definitely wanted to adjust for team SOS. I also conducted analyses using S&P+, FEI and Sagarin’s Predictor rating. They all looked similar, give or take an outlier here or there. Since F/+ has the highest correlation with winning percentage of all of those advanced metrics, I decided to use it as the primary proxy for quality.

Finally, I limited my analysis to only programs from FBS Automatic Qualifier conferences, for a couple of reasons. First, I wanted to pare the data down to a manageable size. Second, and equally important, when I really inspected the recruiting data and thought about it I became concerned that the quality of the recruiting ratings for players that went to most non-AQ schools was probably inferior. It is a simple matter of economics, really. The recruiting services get several sets of eyes on the players who end up going to schools in which there is a lot of interest (i.e. schools from the AQ "power" conferences), and therefore a lot of paying subscribers, but likely expend less effort evaluating PSAs destined for college football’s relative backwaters.

A Definite Correlation

Not surprisingly, there was a definite and fairly strong correlation between Apparent Talent and team quality (as measured by F/+). The correlation coefficient between the simple 4 year Average star rating and the Predictor was 0.51. Interestingly enough, the correlation with my age-weighted model was actually slightly lower, at 0.49. This was based on sample of 466 observations: all seasons for teams in the ACC, Big East, Big 10, Big XII, Pac-10/12, and SEC from 2006 through 2012.

In order to further explore the relationship between Apparent Talent and Team Quality I calculated departures from average on both metrics for every squad in the sample and plotted them on a scatter plot. I also ran a simple linear regression. I thought utilizing departures from average would make interpretation of the data more intuitive. For instance, it stands to reason that a team that is 40% better than average in terms of Apparent Talent ought to be significantly better than average in terms of Team Quality, but how much better?

As it turns out the leverage of apparent talent on team quality is a lot less than a 1:1 ratio. In fact, based on the regression, the leverage (or Beta, or slope of the regression line) is approximately 0.46. Simply stated, at team with 40% better than average talent (for an FBS AQ school) would be expected to be 18% better than the average FBS AQ team in terms of F/+ rating.

What Good is This?

That’s a really fair question. For one thing, it fairly conclusively dispels a notion still held by some college football neo-Luddites. Namely, that "recruiting rankings don’t matter". They pretty clearly do.

For the real stats nerds, here is the regression output of a simple linear regression of F/+ rating for each FBS AQ team from 2006-2012, versus the simple arithmetic average star rating for the previous 4 recruiting classes, per the Scout database:

Yeah, but Other than Proving the Sky is Blue, What Good is This?

This is the part that is really interesting. While this model has some predictive power, it is still very simple and crude and probably not something I would want to use to say…put futures bets down on teams at a Vegas sports book. For instance, it isn’t as predictive as a simple naïve forecast that a team would have the exact same F/+ rating in a given season as it had in the previous one. And how could it be? That sort of naïve forecast has the advantage of capturing much of the same talent information (since the 4 year average recruiting ratings don’t move that much in a single year), while also capturing a whole host of other factors…like how those players are actually doing playing the game of football.

But the fact this model doesn’t capture those things actually makes it pretty useful for other purposes. Consider this simple conceptual model of team quality:

Team Quality (F/+Rating) = Apparent Talent + Coaching Effect + "Noise"

College football is a dreadfully complicated endeavor to model statistically, as you are doubtless aware. By using F/+ Rating we have already largely factored out a lot of the random in-game stuff, like bad bounces that result in one point losses, untimely misses by a walk-on field goal kicker, and so on. In this theoretical model, the "noise" factor includes a whole host of stuff ranging from key players being suspended, scandals turning the mojo around a program decidedly poor, to a whole cohort of players turning out to be grossly overrated. As anybody who follows college football knows, this stuff happens and can definitely affect how a team performs on the field, but from a statistical perspective it is truly unknowable, random and therefore can’t be considered.

The remaining piece of the puzzle is what I am calling "Coaching Effect", but refers not only to game prep and game-day coaching, but scheme, S&C program, and talent evaluation--coaches that routinely have better results than recruiting rankings indicate they should are probably doing a better job of evaluating the available talent than the ratings services are doing.

It stands to reason that over time, strong coaching staffs ought to consistently and measurably outperform (or at least perform in line) with the available talent on hand and weaker staffs will underachieve (sometimes spectacularly). Using the Apparent Talent metric and the simple model described above, we can statistically measure how well a staff performs, both in single season and over time. Of course even good coaches can have an outlier year to the downside and even lousy coaches are fortunate once in a while (cough, cough…Gene Chizik), but consistent deviation from what is predicted by the model ought to be statistically unlikely enough that it should indicate something about coaching performance. And as with most data of this kind, the really large outliers are very interesting.

The methodology was pretty simple. I calculated expected F/+ rating for each season for all AQ teams from 2006 to 2012, based on the Apparent Talent (four year star average), then compared it to the final actual F/+ for each team. I then normalized the deviations from the expected values using the standard deviation of the differences (Std Delta).

For example, the 2008 Washington Huskies were an approximately average team in terms of talent, with an Apparent Talent rating of 2.85. Based on that, they would have been expected to be an approximately average team; the model yielded a projected F/+ of 214.0. Instead, the team went 0-12 with a final F/+ rating of 143.3, just over 33%--or 2.77 standard deviations—worse than the model predicted.

The Best & Worst Coaching Effects on F/+ (AQ Teams, 2006-2012)