What changes have I made?
I’m including this blurb in each of these posts, so if you read it previously, feel free to scroll down to the rankings.
First, with a better chance to analyze which statistical factors are most consistent from the beginning of the season to the end, I made some slight tweaks in the weighting of each statistical factor (the short version: efficiency carries even more weight now). I also worked marginal efficiency and marginal explosiveness into the equation.
Then, I implemented the changes I made during 2018 for previous years. From each week’s rankings post:
- I changed the garbage time definition. S&P+ stops counting the major stats once the game has entered garbage time. Previously, that was when a game ceased to be within 27 points in the first quarter, 24 in the second, 21 in the third, and 16 in the fourth. Now I have expanded it: garbage time adjustments don’t begin until a game is outside of 43 points in the first quarter, 37 in the second, 27 in the third, and 21 in the fourth. That change came because of a piece I wrote about game states at Football Study Hall.
- Preseason projections will remain in the formulas all season. Fans hate this — it’s the biggest complaint I’ve heard regarding ESPN’s FPI formulas. Instinctively, I hate it, too. But here’s the thing: it makes projections more accurate. Our sample size for determining quality in a given season is tiny, and incorporating projection factors found in the preseason rankings decreases the overall error in projections. (For previous years, from before I actually released any sort of preseason projections, I found the most predictive success by keeping a layer of five-year history within the ratings. It’s a small percentage, but it’s in there.)
- To counteract this conservative change, I’m also making S&P+ more reactive to results, especially early in the season. If I’m admitting that S&P+ needs previous-year performances to make it better, I’m also going to admit that S&P+ doesn’t know everything it needs to early in a season, and it’s going to react a bit more to actual results. Basically, I’ve added a step to the the rankings process: after the rankings are determined, I go back and project previous games based on those ratings, and I adjust the ratings based on how much the ratings fit (or don’t fit) those results. The adjustment isn’t enormous, and it diminishes dramatically as the season unfolds.
One more recent change had the most impact, however: I made S&P+ more reactive to conferences as well. It’s similar to step 3: after the rankings are determined, I project previous games based on those ratings, and I track each conference’s average performance versus projection. For the top conference, I found that by the end of the season it was aiming low by two or three points per game per team. For the bottom conference, it was the reverse.
By shifting each team’s rating based on this conference average, and by increasing the weight of said adjustment as the season progresses, it adds basically improves against-the-spread improvement by about one percent per season and cuts the average absolute error by somewhere between 0.2 and 0.3 points per game. That doesn’t seem like much, but look at the Prediction Tracker results and note how much of a difference 1% and 0.3 points per game could make to your projective ranking there. It’s pretty big.
It does, however, mean a fundamental shift in how mid-major teams are judged. Not to spoil the suspense, but look at the difference this adjustment made in some 2018 rankings:
- Fresno State: originally ninth, now 16th
- UCF: originally eighth, now 18th
- Utah State: originally 19th, now 21st
- Appalachian State: originally 11th, now 29th
It’s a pretty harsh adjustment, though it both makes the numbers better and perhaps passes the eye test a bit more. So we’re going with it.
Wait, so you’re including previous years of history in each season’s ratings? How could that possibly be right?
Back when I mentioned at the beginning of 2018 that I’d be using priors in the rankings all year, I had an interesting conversation with some readers on Twitter about what happens at the end of the year. Would I be removing the priors for the year-end ratings so that a) we’d be evaluating teams’ performance based only on what happened in that year, and b) the “recent history” prior wouldn’t then carry influence right into the next year’s projections (since they would include the priors that were included in the previous year’s rankings)?
It was a legitimate question, one to which I don’t think there’s a right answer. To me, it simply comes down to this question: when you’re looking back at previous seasons’ ratings, what are you looking for?
To me, it’s usually to get a sense of who might beat whom, right? I understand the draw of a “this year only” evaluative look, but S&P+ is intended to be a predictive measure, and I’ve decided to include these priors because they make the predictions better. In that sense, removing those priors at the end of the season makes it less of a predictive measure (even though obviously any “predictions” made of a team from 2006 in 2019 is theoretical only).
(How will this impact the preseason projections moving forward? The weights of each of the primary projection factors — recent performance, recruiting, and returning production — are always based on updated correlations, meaning that it depends on how strongly each factor predicts the next season’s performance. I’m dumping the new S&P+ ratings into the old engine, and, well, we’ll find out. Maybe the recent performance numbers end up with a lower correlation. We’ll see.)
On with the rankings.
2007 S&P+ rankings
|San Diego State||4-8||-12.6||93||24.4||79||39.2||105||2.2||2|
|San Jose State||5-7||-16.0||103||20.3||99||34.7||88||-1.6||107|
|New Mexico State||3-9||-18.7||107||24.2||80||40.4||110||-2.5||117|
Under Pete Carroll, USC shared the 2003 national title with LSU and won it outright in 2004. Even though we’re supposed to pretend some of those now-vacated wins didn’t happen, we saw them — they happened. And they kept happening for a few more seasons. And while 1.5 titles will forever be impressive, it was close to so much more.
USC ranked second in S&P+ in 2005, second in 2006, first in 2007, and, [spoilers], first in 2008. They were the best or second-best team in the country every year for six straight seasons. But they managed to lose by a combined six points at Oregon State and UCLA in 2006, by a combined eight to Stanford and Oregon in 2007, and by six at Oregon State in 2008. Erase two or three of those losses, and you’ve got a three-, four-, or maybe five-time national champion.
Yes, of course, “erase their losses, and they’d never lose!” isn’t particularly hardcore analysis, but it’s particularly difficult to avoid thinking like that when it comes to 2007. As discussed in the 2007 advanced box scores piece, USC’s post-game win expectancy in the loss to Stanford — the number that is intended to say “based on the stats produced in this game, you could have expected to win this game X percent of the time” — was 99 percent. The Trojans win that game almost every time. Combined with the 28 percent from the Oregon loss, you get the following scenarios based on this post-game probability:
- Beat both Stanford and Oregon: 27.7 percent of the time
- Split the two games: 71.6 percent
- Lose to both Stanford and Oregon: 0.7 percent
Losing both of those games — again, based on the stats actually produced in those games — was nearly impossible, and going 1-1 almost guarantees the Trojans a spot in the BCS title game in this crazy-ass year. Alas.
Stats love Ron Prince
The ratings tweaks produced quite a bit of movement in 2005 when comparing previous rankings to new ones, but there was minimal movement in 2006. 2007? Quite a bit, if only because teams were all sorts of clustered together.
- Teams that saw their rankings rise by 20 or more places from old method to new method: Hawaii (up 30 spots to 35th), Kansas State (up 29 spots to 19th), Arkansas (up 28 spots to 17th), Alabama (up 26 spots to 31st), Northwestern (up 23 spots to 70th), BGSU (up 22 spots to 73rd), Miami (Ohio) (up 21 spots to 89th), Memphis (up 21 spots to 95th)
- Teams that saw their rankings sink by 20 or more places: East Carolina (down 30 spots to 68th), Washington (down 29 spots to 55th)
Hawaii most certainly wasn’t a top-15 team, but having them at 65th always confused me. Thirty-fifth is just right. Meanwhile, Ron Prince’s second K-State team gets some respect for being competitive against a tricky schedule — the Wildcats whomped S&P+ No. 7 Texas in Austin, nearly beat No. 8 Kansas at home, lost by 10 No. 13 Auburn, and nearly beat No. 30 Oklahoma State in Stillwater.
They did fade late, losing their last four games by double digits, but considering the opposition, they’d have probably been pretty close to the top 10 before that fade began. (That fade continued right on into 2008, and Prince was canned.)
Here are your average S&P+ rankings by conference:
- SEC (+15.4 adjusted points per game, up 2.1 from 2006)
- Big 12 (+13.1, up 3.1)
- Pac-10 (+11.6, up 3.1)
- Big East (+11.5, down 0.3)
- Big Ten (+9.5, up 1.8)
- ACC (+6.0, down 0.8)
- Mountain West (+0.6, down 0.4)
- Conference USA (-8.1, down 2.0)
- WAC (-8.8, down 1.5)
- MAC (-13.7, down 2.5)
- Sun Belt (-18.2, up 1.2)
With the way parity struck in 2007, you’d have maybe thought that the power conferences would have seen their averages shrink without as many truly dominant teams. You’d have been incorrect, apparently! Four of six power conferences saw their averages rise, and four of five mid-major conferences fell.
The SEC fared particularly well, with all but one team finishing with a positive rating and the lowest-ranked team (No. 72 Ole Miss) just one spot behind the top team in the Sun Belt (No. 71 Troy). It also had six teams in the overall top 17, plus a seventh (No. 12 Mizzou) that would join the conference a few years later.
The Big 12 surged, too. The bottom of the conference was still dreadful, anchored by No. 98 Baylor (Art Briles would take over in 2008), but six teams ranked 19th or better, including one (the aforementioned KSU) that finished with a losing record. Texas slid in at No. 7, obviously benefiting from the recent-history adjustment that dragged down both Kansas (No. 8) and Mizzou, but the Horns did still win 10 games that year, too.
This was the year the Big 12’s conference identity really took hold, too. The top nine teams in the conference (everyone but Baylor, ISU, and Colorado) all ranked in the Off. S&P+ top 30 (seven ranked in the top 12), but only three ranked in the Def. S&P+ top 40.