Apparent Talent, Team Quality & Coaching Effects: Part Deux

This is the follow up to the article I wrote for Football Study Hall that attempted to relate Team Quality (as measured by FO’s F/+ rating) to recruiting rankings. I chose to focus only on teams in the Automatic Qualifier "power" conferences, for a variety of reasons.

For the purposes of the initial analysis I utilized a simple four year arithmetic average of Scout.com’s average star ranking for each class, as of NSD. I chose to call the measure of recruiting rankings that I used Apparent Talent, because recruiting rankings are imperfect and subjective and also because I did not attempt to control for players that didn’t qualify, transfers, medical retirements, etc..

To summarize, Apparent Talent is a useful predictor of team quality, to a statistically significant degree, but it falls well short of being a truly robust predictive model. The reasons that it doesn’t do a great job of predicting team quality are myriad, but I have a strong hypothesis that is well supported by the data that an awful lot of the deviation between what could be expected based on Apparent Talent and what shows up on Saturdays in the Fall comes down to the impact of coaching staffs on the outcomes. I termed this Coaching Effect.

Some coaching staffs do a great job of taking the talent on hand and producing results well above what could reasonably be expected (i.e. what is predicted by linear regression). Other staffs do the complete opposite. A larger group does neither reliably, at least not to a statistically significant degree.

Refinement to the Analysis

Since the first article, I have downloaded and tabulated recruiting class ranks from a second source: Rivals.com. I re-ran the entire process using Rival’s rankings and compared them to the results from Scout’s. In aggregate, there was no difference worth discussing. The conclusion to draw there is that there isn’t a measurable difference in the "accuracy" of the two services rankings in aggregate, for the purposes of this analysis anyway. At the micro level, there were a few meaningful differences, so going forward I will use a composite of the two rankings to calculate Apparent Talent.

"I was told there would be no math."

I will be the first to admit that I am not a statistician. My unspectacular grades in my undergrad and graduate stats classes are proof positive of that fact. I do utilize lots of numbers and some statistical analyses in my professional life, but I am seldom the person actually designing the stat studies. A few extremely quant-savvy and helpful readers pointed out some of the pitfalls in the regression analysis I was doing. But I have to admit, when concepts like serial correlation and heteroscedasticity enter the fray, I am rapidly getting out of my depth.

Several other readers pointed out that the r-squared value of the statistical relationship between Apparent Talent and Team Quality didn’t exactly knock their socks off. I can recall multiple stats professors and quant professionals telling me not to get too wrapped up in r-squared because while it is intuitively appealing it isn’t the silver bullet of stats interpretation that most non-nerds think it is. They instead extolled the virtues of t-tests, z-scores, and P-values and…zzzzzzz.

Anyway, I wanted to get away from the heavy duty regression-centric discussion and couch the relationship in terms that most of us can relate to better. In order to do this, I ranked all 466 team seasons from AQ conferences in terms of Team Quality and then looked at the distribution. Lo and behold, the distribution was extremely close to a normal distribution that even those among us who struggled with Stats 101 ought to be familiar with:

In this case, instead of using just F/+ as the proxy for Team Quality, I decided to be a bit more democratic and inclusive and used an equally weighted composite of FO’s F/+, FO’s S&P+, and Jeff Sagarin’s Predictor ranking. I call this three-headed bastard Quality Composite. You will recall also that I am now using a composite of Scout and Rivals four year average star rankings as my Apparent Talent measure.