Data Drop: Before the Draft Edition

Categories: NFL Draft, Statistics
Comments: No Comments
Published on: April 21, 2013

The 2013 NFL Draft is less than a week away.  All the draft analyzers are putting in their final push to get everything in order for the big day.  In the spirit of getting everything in order, I’ve added a load of new data to the site.  Check the Draft Numbers menu for everything.

I’ve added my entire list for wide receivers and pass receiving tight ends.  I’m a bit scared of putting that up, actually.  There was a time when I said I wasn’t going to do that.  I’m scared because I don’t have a good metric for understanding what these numbers predict.  The numbers are an indication of the pass catching talent of that player separate from the quarterback and offensive scheme.  However, I don’t think the numbers will predict receiving yards or touchdowns at the end of a season.  My major project for the summer will be to connect these numbers to something more meaningful and concrete, like I have with quarterbacks.  Until then, we can just look at the nice pretty list.

One note, the tight end data refers to pass catching ability only.  There are many other factors that make a quality tight end other than being able to catch passes, most notably blocking ability.  The tight end list only ranks tight ends on the single measure of pass catching.

Tweaking the Model

Categories: General Info, Statistics
Comments: No Comments
Published on: April 14, 2013

Apologies for the unexpected hiatus last week.  I had to fly to Seattle for a wedding.  But now I’m back in the Frozen Tundra and ready to tell you about some stats.

I’ve been adjusting the statistical methodology I use in the last couple weeks.  Nothing has changed about how I calculate the Completions Away from Average metric, but I have been changing how I use that metric to predict NFL success.  Be advised that this is another post that is heavy on technical details.  If you are not a technical reader, the take-home point of this post is that the changes will make my predictions better in the long term, but add more uncertainty in the short-term.

Change #1:  Dependent Variable

The DV I’ve been using for predicting is NFL Passer Rating after three years in the league.  I chose three years arbitrarily.  I thought, “Well…most draft prospects have a decent chance of playing after three years in the league, and we don’t want to get too far away in time from college because we can’t account for growth and coaching, so let’s use three years.”  Arbitrarily isn’t the best reason to choose a DV, so let’s use something with a little more meaning.  I’m going to stick with Passer Rating as it’s a reasonably good per-attempt statistic.  The per-attempt part is what I really care about.  Yes, there are problems with it, but I think the good outweighs the bad.

But choosing three years as an arbitrary time-period doesn’t make much sense.  We don’t have to change this much to make it meaningful, but we should change it.  Rather than Passer Rating after three years in the league, I went with Passer Rating after four years in the league.  Four years is much more meaningful because it is the length of all rookie contracts under the new CBA.  Thus, I’m predicting what a player is most likely to do during his rookie contract.  I think that is much more meaningful than what I was doing previously.

Change #2:  Prediction Model

Changing the prediction model solves two problems, which is handy.  When I have two problems, I like being able to solve them both at the same time.  So what are the two problems that need solving?

Problem #1:  Adding Data

The typical method of designing a statistical model is you have some theory, you collect data in a way to test that theory, and you use the data you collected to create a mathematical formula that minimizes the errors between your theory and the data.  In most cases, my statistical model included, the mathematical formula is fairly simple.  We draw a scatterplot of our data, and draw a straight line through the cloud of data points.  Then all we need is our 8th grade math skills to solve for the important values of that line, y = mx + b, where m is the slope of the line and b is where the line crosses the vertical axis (a.k.a. intercept).  We also know x, in this case Career CAA, for each individual player, so we just solve for y, in this case Passer Rating after four years in the league, and we have our prediction.  This process is called regression modeling.

But then what?  New players will always be entering the NFL.  Also, some of the players that I have in the model haven’t been in the league for four years, so their data will change.  But that’s not a problem, right?  We just keep collecting the data and everything should get even better, right?

What I’m about to say might surprise some of you that are not familiar with traditional methods of statistical modeling.  Continually adding data and using traditional regression techniques will actually make your predictions worse.  Regression models hate adding new data.  They want to take the information you give them the first time and do the best they can with it.  When you take the same model, feed in the old data plus some new data, the model is more likely to follow blind alleys and tell you that unimportant things might be useful predictors.  So the realities of the NFL make traditional modeling techniques less useful.

Problem #2:  Distribution of the DV

Traditional regression assumes a normally distributed dependent variable.  For most things, this is fine.  For Passer Rating this is not fine as passer rating isn’t normally distributed.  In a normal distribution, most cases are in the middle with a few cases on the high and low ends.  However, a per-attempt statistic like passer rating isn’t normally distributed because some quarterbacks won’t have many attempts.  For example, let’s look at Ryan Mallet this year.  He had 4 attempts this season, one of which was intercepted.  Now, the passer rating metric just takes what it has and extrapolates from that information.  The number that is Ryan Mallet’s 2012 passer rating assumes he would throw an interception every 4 attempts, which we humans know is not true.  If he was on the field and made 100 attempts, he wouldn’t expect him to throw 25 interceptions.  To our brain, that’s ridiculous.  But the math equation doesn’t know this.  It’s just taking what we gave it and spitting out a result.  So Ryan Mallett’s passer rating is currently 5.  Some players, like Mallett, have really poor numbers.  Other players, like Kirk Cousins, have really good numbers. We shouldn’t believe either number just yet because they haven’t had enough attempts to get a good picture of what they can do when they are on the field.  This is the nature of the beast in the NFL; some quarterbacks have passer ratings that are extreme compared to others.  But regression models don’t like extreme values.  Regression models like everything to be nice and normal.  Extreme values confuse the model and exert undue influence on the final result.

Solution:  Bayesian Robust Linear Regression

I won’t say too much about the Bayesian portion of this.  When I start talking Bayesian methods in class, my students tend to glaze over like fully fed zombies.  Baaaaaayes.  Regardless, the Bayesian portion solves Problem #1.  Bayesian analyses are built to accept new data in ways that traditional regression models are not.

The Robust Linear Regression part is the really cool part.  In this analysis, we can change the assumption about the distribution of our DV.  We can, for example, assume our values are distributed as a t-distribution, which means most of the values are in the middle, but extreme values are expected and dealt with easily.  So, let’s revise the predictions with this analysis.

New Model

We will set some very general prior predictions before we run the analysis.  We will assume that our DV is distributed as a t-distribution.  We will also assume that possible intercepts and slopes are distributed normally.  The analysis will calculate which degrees of freedom by estimating a value called tau.  Tau is conceptually, but not quite the same as degrees of freedom.  It’s a measure of how fat the tails are.  The closer to 0 this is, the better this analysis will do compared to traditional techniques.  If tau is > 30, we shouldn’t see much difference.  We assume possible values of tau follow a gamma distribution beginning at 1.

So what do we find?  Note, we’re using 4-year NFL passer rating as our DV rather than 3-year as in past analyses.


The first question is if we made a wise choice in assuming our dependent variable is t-distributed rather than normally distributed.  Our most credible estimate of tau from these data is 2.56.  Remember, the closer to 30 this is, the less this matters.  Since the most credible estimate is much, much less than 30, we should improve our predictions substantially with this procedure.

Equation of the Line

We’re still trying to find a line that best predicts our outcomes.  Bayesian regression is a little different in that it gives us lots of different lines that are all credible given the data we have.

In the initial estimates, we have a small sadness.  0 remains a credible slope.  This is what I meant above about increasing uncertainty in the short term.  There is a chance that everything I’ve been talking about is not true and this CAA metric isn’t worth anything.  The nice thing about Bayesian analyses is that they put a probability on this chance.  Given the data we have, there is about a 6% chance that I’ve been blowing smoke at you this entire time.  I’m willing to keep going with that, but you can make your own decisions regarding that.

To create the actual equation, I took the most credible intercept and the most credible slope and stuck them in the same equation.  Note that I’m still new to Bayesian estimation techniques and I’m not quite sure if this is kosher or not.  If I find that it’s not, I will revise the model.

New Equation for Prediction

4-Year NFL Passer Rating = 70.5 + 0.143 * (Career NCAA CAA)

I’ve added a new column to my predictions for the 2013 draft class.  Note that this doesn’t change their relative rankings any, just the estimate of their NFL passer rating four years from now.

The Way-Back Machine: JaMarcus Russell

Categories: NCAA FBS, NFL, Statistics
Comments: No Comments
Published on: March 17, 2013

Blizzard permitting, I will be standing in front of an Introductory Psychology class lecturing on the topic of intelligence on Monday.  The major point of the lesson will be that intelligence is a combination of ability, on the one hand, and the skills and motivation to use that ability on the other.  Both influence outcomes on intelligence tests about equally.  It’s not just intelligence we see this pattern for.  Many outcomes we label “talent” or “God-given ability” can be broken down statistically into an inherited, biological component and a skills and motivation component.

Which brings us to JaMarcus Russell.  The narrative about Russell is that he is a classic case of a highly talented individual lacking the skills and motivation to put that talent to use.  Pundits point to his arm strength and success in college as indicators of his talent, while his numerous issues both on and off the field as indicators of his lack of work ethic, poor motivation, etc.  Even now, four years after his last played down of professional football, we seem to be paying attention to his apparent comeback.  We talk of how great an “untapped” talent he still is.  If only he could get the motivation component under control.  But untapped talent tends to remain so.  Without the skills and motivation necessary to unlock such talent, that talent will remain hidden and useless.

So what do the data say?  Was JaMarcus Russell tapping his potential in college before his unfortunate demise in the NFL?  Before we answer that question, some caveats.  JaMarcus Russell was one of the first players I was interested in looking at once I realized I had a useful model for predicting success at the NFL level.  But there are problems.  First, I cannot analyze anything before 2005.  An important piece of data, incomplete pass targets, was not recorded in any form prior to then.  So there are points in JaMarcus Russell’s career that we simply can’t analyze.  Second, the limited amount of data limits our ability to separate JaMarcus Russell’s ability from the impact of play-calling, skills of the typical defense faced, etc., etc., etc.  Essentially, we are limited how well we can separate Russell from program-level factors.  This is an especially difficult problem for evaluating Russell because Les Miles began his tenure at LSU in 2005.  We can’t correct for the influence of Les Miles in these early data.

So once again, data.  The table shows JaMarcus Russell’s Completions Away from Average for the 2005 and 2006 NCAA season as well as totals.  Remember that 0 CAA is average.

2005JaMarcus Russell-11.703
2006JaMarcus Russell12.825

Even though we only have two seasons of data, note the wild inconsistency across these two seasons.  In 2005, he had a poor season but in 2006, he had a pretty good season.  The combination of the two gives us a prediction of his passer rating after 3 years in the NFL of 65.9.  His actual NFL passer rating after 3 years in the league was 65.2, a phenomenal prediction for this model given the limited data it’s working with.

So, was JaMarcus Russell a tremendous college talent that flamed out because of poor work ethic.  Not really.  He did exactly what we should have expected.  He was inconsistent in college and ultimately should have been labeled as an average NCAA quarterback.  Sound familiar?

The bigger question is, how did evaluators get it so wrong with JaMarcus Russell.  I can only speculate about that, but I think evaluators were seduced by what they saw outside of actual games.  JaMarcus Russell can throw a football a long way and is “naturally gifted,” according to Jeff Garcia, so what if he is somewhat mechanically inconsistent and has some sideline issues?  It would appear that, at least in JaMarcus Russell’s case, it matters a great deal.

Attending the Sloan Sports Analytics Conference via Twitter

Categories: Uncategorized
Comments: No Comments
Published on: March 3, 2013

I’ve been participating in a very strange experience lately.  Over the last two days, I’ve been vicariously “attending” the Sloan Sports Analytics Conference via Twitter.  At first I didn’t even mean to.  I just happened to follow enough nerds.  I logged in Friday afternoon and suddenly my Twitter feed was filled with interesting points, great zingers, and summaries of each talk that were surprisingly effective for only being 140 characters long.  I could even ask questions if I knew the right hashtag.  It really was an incredible experience.  Here I am getting a full conference experience – outside of the networking – while sitting halfway across the country correcting student assignments.  Technology, man.  Crazy.

Anyway, the whole experience got me thinking about sports analytics as a field and what I can contribute to it.  Ultimately, I think I can add two pieces of the football analytics puzzle:  1) quantifying interdependent team processes and 2) specifying my methodology for calculating Completions Away from Average.

Part 1:  In Which Jared Plays Under Appreciated Academic

I don’t often jump up and down in my office.  My job as a college professor is fairly low-key.  I don’t have lots of reasons to get really excited during the work day.  On Friday, I did.  There was one panel at Sloan talking about how sports are almost exclusively outcome focused, but the successful teams are the teams that focus more on process.  Basically, they were saying that if a team is following a process that that generally leads to positive outcomes, positive outcomes will follow at a greater than average rate.  You won’t have to rely on luck or poor competition or something else that’s not under the team’s control if the team is process focused.  This piqued my interest.  I spent 10 minutes playing Twitter like a video game, clicking on the new tweets as fast as they pop up.  Everyone was digesting the idea.  There was general agreement that process is important.  In fact, general agreement that process should be the true focus of a well-constructed team.  And then the big question came.  The question that always gets asked when a bunch of data nerds start talking team process.  How do we measure/quantify/work with process?  And now I’m literally out of my chair.  “Ooooh!  Ooooh! I know the answer!  I’m not in Boston, but it’s still cold and snowy around here.  Maybe we can pretend.  Pick me! Pick me!”

Well, of course they didn’t pick me.  We got other answers of “You don’t,”, “Results are the best proxy,” and, the answer that started a whole new round of jumping, “It’s exceptionally difficult in football because the data are interdependent”.  I don’t have a specific problem with any those answers because they are the best generally accepted answers the field has.  But I think I have a better answer.  I tried to engage some of the tweeters in conversation about this topic, but it didn’t work.  The ideas are too big for 140 characters and when I tried to tweet them, I ended up sounding asininely critical.  Why do I think I have a better answer?  To answer that, you need a little background on graduate school.

Let’s imagine that you have earned your master’s degree and want to go for the Ph.D.  Before you get admitted to the program, you have to demonstrate that you have the knowledge and skills necessary to be successful.  There are two general ways that programs will do this, qualifying exams or an area paper.  My program went with the area paper.  An area paper is a project where you go off for a while and find an important, unanswered question you find interesting.  You then read everything ever published regarding that question.  In the end, write a 50-100 page paper describing your answer to that unanswered question.  You then present your paper to three or four tenured professors who try their best to rip down, discredit, or otherwise poopoo on your answer.  If it holds up to a couple hours of questions, you get in.  If it doesn’t, well then you have a problem.

I went off and spent nine months working 12 hours a day, six days a week in my tiny little basement office.  I found the question I thought was interesting, read all that I could about it, and came up with an answer.  In the end, I wrote a paper I titled “Inferring Team Process Using Interdependent Team Data.”  It was good enough to get me admitted to my program and I spent the next two years working the same schedule figuring out a way to test and evaluate my answer.  So you can imagine why I was jumping up and down about the answers about quantifying group process in interdependent teams.  It’s a tough problem.  I took me nine months working nearly 80 hours a week to figure out an answer and another two years working 80 hours a week to figure out how to validate that answer.  But in the end, I think it’s a good answer.

I even got my original idea published as a book chapter.  I had to change the title to make it fit the scope of the book, but the idea is essentially the same.  You can find the book here.  If you read it, you will be part of an exclusive club.  This book is very obscure.  It’s been on the market for a year and Google Scholar doesn’t even know it exists.  When Google can’t find you, you know you’re off the grid.  But there you go.  Inside my chapter in that book, you will find my idea for inferring and quantifying team process when teams are interdependent.  As a short summary, you find out about team process by understanding the distribution of team member expectations.  What does each team member expect will happen as the team completes its task?  Variability in expectations can be used as a proxy for effective (low variability) and ineffective (high variability) team processes.  If everyone is on the same page as it were, things are good.  If everyone is not on the same page, things are bad.  Moreover, there are multiple ways the team could be on the same page or not.  To continue the metaphor, the team members could all be on different pages, or they could all be on the same page but reading different books.  A specific statistical method can be used to figure out in what way the team is or is not on the same page.  There’s a lot more to it than those short sentences, especially on the quantitative end.  If you would like the full argument, read the chapter.  Maybe you can get the book through interlibrary loan or something.

Part 2:  In Which Jared Contemplates Publishing the Methodology

So there was that whole thing.  There was some good to come out of feeling under appreciated.  It got me wrestling with the idea of publishing the methodology I use to calculate my Completions Away from Average metric.  I want you to know that I believe I have a fundamental, scientific duty to publish this methodology.  It’s not scientific to purposefully keep people from verifying, replicating, and questioning my work.  I recognize that publishing the methodology is the quickest route to credibility.

But I’ve been dragging my feet about doing it.  I’m not proud of the reason, but I do feel I have some justification.  You see, I don’t have any job security.  For many reasons, some under my control and some not, I’m not a competitive job candidate for full-time professor jobs.  Instead, I’ve been cobbling together these one-year teaching positions to try and make ends meet.  If you do these non-secure contracts full-time, they let you call yourself a professor, but I don’t have a true professorship in any sense of the word.  My CAA methodology is useful, it’s powerful, and it’s the only thing I’ve ever created outside of academia that I think someone might pay me for.  Which makes me hesitant to publish it.

So what do I do?  I don’t have a job after June, I have something that I think is worth money, but I believe I have a fundamental duty to give it away.  A significant portion of my identity is tied up with being a scientist and educator, but I also enjoy eating.  Basically, the conflict breaks down into whether I think credibility or knowledge is more valuable.  I haven’t answered that question yet, which leads me to drag my feet about publishing. In the meantime, the only way to generate credibility is to take the “FiveThirtyEight approach.” Have a proprietary method, make some forecasts, and wait to see how the results match the predictions.

So, now you know what sort of questions I’m wrestling with right now.  I’d like to thank the Sloan Sports Analytics Conference and Twitter for giving me an excuse to talk about them.  Any advice from the internet would be welcome and appreciated.

The Kaepernick Question

Categories: NFL, Statistics
Comments: No Comments
Published on: November 29, 2012

In one of the stranger decisions we’ve seen this year, Colin Kaepernick is now the starting quarterback in San Francisco while Alex Smith sits on the bench.  I won’t pretend to know all the ins and outs that went into this decision.  What we can know is that it isn’t based on objective data regarding job performance.  From every conceivable measure I can find, win percentage, completion percentage, Completions Away from Average (+15.71 currently), Alex Smith is a top NFL quarterback.

People with better words than me have discussed how Alex Smith has grown and doesn’t deserve to be benched in favor of an unproven commodity during a playoff race.  I won’t continue to go over that ground.  What I might be able to uniquely add to this conversation is a perspective on what we can expect from Colin Kaepernick.  Is his early success simply random chance and small sample sizes or is there something more substantial there?

Well, the initial prediction based on his career NCAA CAA does not look promising.  For his career, Colin had a CAA of -22.32.  This means that over his career, Colin completed 22 fewer passes than we would expect an average NCAA quarterback to complete.   The regression equation predicts that Colin Kaepernick will have a passer rating somewhere near 62.6 by the end of next year.  So far, not a promising start.

There might be something else to consider though.  The sample sizes are far too small to start making a large deal about this, but the trajectory of Colin Kaepernick’s NCAA career is worth mentioning.  Below you will see a graph of Kapernick’s CAA broken down by year.



The reason Colin Kaepernick has such a low career level CAA is largely because of his sophomore season with some addition from his junior season.  His senior season, however, was actually quite successful.  Whether the change is caused by noise in the data or meaningful learning  is a question for another day.  It is worth noting that I’ve run the regression equations using CAA from a player’s final season in the NCAA to predict NFL passer rating and the equation is not significant.

However, if you’re a San Francisco fan and worried about your team’s prospects both at the end of this year and for the future, the improvement Kaepernick showed during his senior season might be a straw you can grasp at.

page 1 of 1
Welcome , today is Wednesday, March 21, 2018