The Kaepernick Question

Categories: NFL, Statistics
Comments: No Comments
Published on: November 29, 2012

In one of the stranger decisions we’ve seen this year, Colin Kaepernick is now the starting quarterback in San Francisco while Alex Smith sits on the bench.  I won’t pretend to know all the ins and outs that went into this decision.  What we can know is that it isn’t based on objective data regarding job performance.  From every conceivable measure I can find, win percentage, completion percentage, Completions Away from Average (+15.71 currently), Alex Smith is a top NFL quarterback.

People with better words than me have discussed how Alex Smith has grown and doesn’t deserve to be benched in favor of an unproven commodity during a playoff race.  I won’t continue to go over that ground.  What I might be able to uniquely add to this conversation is a perspective on what we can expect from Colin Kaepernick.  Is his early success simply random chance and small sample sizes or is there something more substantial there?

Well, the initial prediction based on his career NCAA CAA does not look promising.  For his career, Colin had a CAA of -22.32.  This means that over his career, Colin completed 22 fewer passes than we would expect an average NCAA quarterback to complete.   The regression equation predicts that Colin Kaepernick will have a passer rating somewhere near 62.6 by the end of next year.  So far, not a promising start.

There might be something else to consider though.  The sample sizes are far too small to start making a large deal about this, but the trajectory of Colin Kaepernick’s NCAA career is worth mentioning.  Below you will see a graph of Kapernick’s CAA broken down by year.



The reason Colin Kaepernick has such a low career level CAA is largely because of his sophomore season with some addition from his junior season.  His senior season, however, was actually quite successful.  Whether the change is caused by noise in the data or meaningful learning  is a question for another day.  It is worth noting that I’ve run the regression equations using CAA from a player’s final season in the NCAA to predict NFL passer rating and the equation is not significant.

However, if you’re a San Francisco fan and worried about your team’s prospects both at the end of this year and for the future, the improvement Kaepernick showed during his senior season might be a straw you can grasp at.

“And Why Should the People Listen to You?”

Comments: 1 Comment
Published on: November 26, 2012

Because, unlike other models, my model predicts per attempt NFL statistics using NCAA data.

[Author’s Note:  The non-technical reader is invited to skip this section as it will be filled with statistical jargon and  “academese”.  The technical reader is invited to submit any skepticism in the comments section]

The ultimate test of a quarterback model is its ability to take NCAA data and predict NFL statistics on a per attempt basis.  Predicting per attempt NFL statistics rather than total NFL stats is key here.  Predicting total NFL stats for quarterbacks isn’t difficult.  All one needs to know is draft order.  Players that get drafted higher tend to have higher total stats.  The thing is, the only reason that relationship exists is that players that get drafted higher get more playing time.  More playing time means more opportunities to run up attempts, completions, yardage, and touchdowns.  But if you look at per attempt statistics like Passer Rating, yards per attempt, etc., draft order shows no ability to predict NFL performance.  Quarterbacks drafted in the 6th round are just as likely to be successes as quarterbacks drafted in the 1st or 2nd round.

Our primary metric is Completions Away from Average (CAA; see The Concept for additional information).  The data necessary for calculating CAA has only been collected since 2005.  In addition, two years of historical data are required before estimates become stable.  Therefore, the data set includes NCAA CAA beginning in 2007 and NFL Passer Ratings beginning in 2008.

Total CAA for the quarterbacks’ NCAA careers were used as predictors in a regression equation [Author’s Note:  This will change the projections mentioned a few weeks ago, which were based on data from the 2012 season only.  It will also change our assessment of the 2011 NFL draft class.].  The criterion variable was NFL passer rating for the quarterback during his first three years on the league.  All quarterbacks that played for an FBS team and were drafted between 2008 and 2012 and started at least one NFL game were included in the data set.  Undrafted quarterbacks or quarterbacks that did not start an NFL game were not entered.  This gives us a set of 40 quarterbacks with complete data.  Data from the 2012 season was included up through Week 11.  One multivariate outlier was identified (see graph below), Colin Kaepernick and his crazy awesome game last week.  This data point was deleted for the analysis reported here.  We will revisit this deletion at the end of the season.

The regression table is shown below.  The highlights from the table are as follows.  Career CAA is a significant predictor of NFL passer rating during the first three years in the league, accounting for 12.3% of the variance.  In addition, for every addition CAA a quarterback accumulates during his NCAA career, a quarterback’s three year passer rating is expected to increase by 0.18













Career CAA




12.3% may not seem like much, but there are two reasons to be optimistic about that number.  First, no published analysis exists that shows NCAA statistics being able to predict per attempt statistics in the NFL.  Second, NFL passer rating is another of those statistics that is infected with the results of others.  The quarterback’s offensive system and talent of the receivers also factor into passer rating.  So, finding NCAA Career CAA predicting NFL passer rating three years out is likely getting at something meaningful for an individual quarterback.

So, why should the people listen to me?  Because the information presented here is useful and predictive of future performance at the next level.

The Concept

Now that I’ve thrown down my top NCAA FBS quarterbacks, it would probably help to give some context of how exactly I’m arriving at these player rankings.  For the moment, I will hold back on giving away the entire mathematical formula.  However, an understanding of the assumptions and methods would probably help you interpret everything.  This is especially true since I’m telling you that the Colby Cameron from Louisiana Tech is the best senior quarterback in FBS football, Keith Price from Washington is the best franchise quarterback prospect, and Matt Barkley is going to bust quicker than cheap plastic on Christmas morning.

Let’s begin with an assumption.  The primary job of a quarterback is to complete passes.  Where those passes are thrown, who they are thrown to, and what route the receiver is running are not important.  I realize that this assumption is enough to spark controversy on its own.  Regardless, this is the primary assumption of these analyses.  (Note:  The “who they are thrown to” assumption will be relaxed in future work.  Specifically, it’s probably important if a pass is thrown to a running back or a not-a-running-back as passes to running backs are typically shorter and the target is usually not moving).

Next, we recognize that there is a massive amount of historical data regarding player performance.  We can look at how receiver performance changes when the quarterback changes.  NCAA football is an excellent test case for this as receivers and quarterbacks are forced to turn over at least every four years, if not sooner.  By looking at how performance changes when teams change quarterbacks, when quarterbacks throw to different receivers, etc. we can make a very important estimation.

To understand how the estimation works, let’s look at Colby Cameron of Louisiana Tech.  Given all the historical data regarding receiver performance and quarterback performance, we can make an accurate estimate of how many passes every single quarterback in FBS football would complete given Louisiana Tech’s offensive scheme and stable of receivers.  So, we can estimate how many passes Matt Barkley, Geno Smith or whoever would complete if we suddenly dropped them onto this year’s Louisiana Tech team.  We assume the same plays are run in the same order.  We then look at the distribution of all of these estimations.  We find the average of the estimated number of completions.  We then compare this number to the number of completions Colby Cameron actually has this year.  After all these calculations, we discover that Colby Cameron has completed 26.17 more passes than the average NCAA quarterback would complete given Louisiana Tech’s offensive system and stable of receivers.  This is tops among senior quarterbacks in NCAA FBS football.  It’s not, however, tops in college football.

The top of college football in this category is Keith Price.  He has completed 76.37 more passes this year than we would expect the average NCAA FBS quarterback to complete given Washington’s offensive system and stable of receivers.  This is why I say that Keith Price is the best franchise quarterback prospect available.  Keith Price has topped the NCAA two years running in this category.

What about Matt Barkley?  How can I say that Matt Barkley is going to bust in the NFL?  Given the estimations, Matt Barkley has completed 7.71 fewer passes than an average NCAA FBS quarterback would complete given USC’s offensive system and set of receivers.  According to this analysis, Matt Barkley is absolutely not the #2 quarterback prospect in the 2013 NFL draft.  He is, in fact, a below average NCAA quarterback.  How can we possibly expect that he would be an above average NFL player if he can’t even get above average at the college level?

Hopefully, this at least gives you some context regarding the statements I’m making.  I plan on posting a table with the actual numbers for every quarterback in the coming months so you can see exactly where each quarterback stacks up.  Until then, I have more data to collect and more analyses to run.

Quarterback Talent

I’ve talked before about how I think the draft class of 2012 will be regarded as one of the great classes.  The class has a strange combination of high talent and playing time that is likely to pay off in spades.

The 2013 draft class does not look to be panning out the same way.  There is not the same kind of talent at the top of FBS college football this year compared to 2012.  Not only that, but all the experts are looking in the wrong direction for talent.  We’re likely to see a lot of quarterback busts by players that play for schools in the BCS automatic qualifier conferences.  The really talented quarterbacks are in the smaller conferences this year.  I’ll talk about predicted busts another time, but let’s see who in the senior class is bubbling to the surface through 11 weeks.  Note that these ratings apply only to data from this season.

#1)  Colby Cameron – Louisiana Tech – Louisiana Tech has been a great surprise this year, in no small part due to their quarterback.  He’s put up gaudy numbers and led the Bulldogs (yes I did have to look the mascot up) to a 9-1 record.

#2)  Ryan Aplin – Arkansas State – Ryan has the most solid career data of anyone on this list.  He’s approaching 10,000 yards for his career and completes passes at high rates.  The primary reason people aren’t looking at him is that his team is not known as a high powered passing offense.

#3)  Kawaun Jakes – Western Kentucky – This was an interesting name to pop up.  The Western Kentucky system does not seem like the type of system to create an NFL caliber quarterback.  The only thing that worries me about this pick is that most of his passes are to running backs and tight ends.  These passes tend to be shorter passes to receivers that aren’t moving.  Regardless, the fact that Kawaun has the numbers he has in that offense with those receivers makes him worth a look.

#4)  Collin Klein – Kansas State – The senior quarterback at Kansas State has shown a tremendous ability at the quarterback position this year.  His running ability will likely attract some buyers as well.  The fact that he plays on the currently ranked #1 team in the country won’t hurt his draft prospects either.

Like I said, the quarterback prospects for this class aren’t fabulous, but there is talent there for those willing to look for it.  Of all the potential draftees out there, there is likely only one franchise quarterback among them.  If your team is really in the market for a franchise quarterback, hope and pray that Washington’s Keith Price declares for the draft.  He doesn’t play on a flashy team with a great win-loss record, but that guy is a beast.

A Tale of Two Quarterback Controversies

Categories: NFL, Statistics
Comments: 1 Comment
Published on: November 7, 2012

We’re halfway through the season and quarterback controversies are brewing.  I want to compare and contrast two quarterbacks that have been struggling recently, Michael Vick in Philadelphia and Christian Ponder in Minnesota.

Right off the top, quarterback controversies halfway through the season are wildly overblown.  The differences among NFL quarterbacks are actually very small and not larger than the effects of random chance.   That being said, we have enough data to begin to understand the different reasons these two quarterbacks might be struggling.

Let’s start in Philadelphia.  Michael Vick is taking a lion’s share of the blame for problems in Philadelphia’s passing game.  The question is, how much is Michael Vick to blame for Philadelphia’s problems?  Well, our handy-dandy math equation tells us that Michael Vick is a below average quarterback this year.  However, he’s not that below average.  The model tells us that Michael Vick has failed to complete 3.9 passes that an average quarterback would have completed, less than half a pass per game.

So while Michael Vick is below average, he’s not so below average that a quarterback change is warranted.  He’s certainly not Mark Sanchez below average (who has failed to complete 22 passes an average quarterback would have completed).  Sanchez does have an advantage over Vick in this area though.  Sanchez has the only backup in the league expected to be worse than he is, whereas Vick’s backup, Nick Foles, is projected to be rather good.

Now let’s look at Minnesota.  To say Christian Ponder has struggled the past few weeks is to say Januarys in Vikings Country are a little chilly [Note:  The author is a resident of Vikings Country and a lifelong Vikings fan].  I believe ESPN put his QBR somewhere around 11 for the game last Sunday.  But can we say that Christian Ponder is the problem?  Actually, Christian Ponder is an above average quarterback, completing more passes than an average quarterback would have completed given his stable of receivers.  Where Michael Vick has failed to complete 3.9 passes that an average quarterback would have completed, Christian Ponder has completed 6.5 passes that an average quarterback would not have completed.

Minnesota has a receiver problem.  Specifically, Minnesota has a receivers-not-named-Percy-Harvin problem.  This leaves defenses with a simple solution for stopping the Minnesota offense, load the box to contain Adrian Peterson, double cover Percy Harvin, and sack the quarterback.  Wash, rinse, repeat.

I love sorting through the data that this model puts out because it paints such interesting picture of where one should lay both praise and blame for quarterback and receiver issues.  Michael Vick may deserve a small amount of criticism for his play this year, though probably not as much as is currently being heaped on him.  Christian Ponder, on the other hand, deserves no blame for his struggles and is actually doing amazingly well given the receivers he has at his disposal.

Quarterback Busts

Categories: NCAA FCS, NFL Draft
Comments: No Comments
Published on: November 4, 2012

Will we ever see a quarterback bust the like of JaMarcus Russell ever again?

I recall Todd McShay was asked this question during last draft season. Todd answered that he didn’t think we would due to the large amount of data that is now available on all NCAA quarterbacks. He said there is so much tape on each and every college quarterback that it is difficult to imagine problems wouldn’t be spotted before the draft.

I don’t want to beat up on Todd for something he said more than six months ago, but I want to address the sentiment in that answer. The biggest problem with evaluating quarterbacks is not a lack of tape. There is a tremendous amount of data available on college quarterbacks. Not only that, the NFL spends large sums of money to put on the combine, where we gather even more data on players we may have not taken a close look at up to that point.

The problem with evaluating quarterbacks is the human brain. It is simply impossible to look at someone else’s tape and separate good quarterback play from good receiver play from luck.

Let me illustrate what I mean with an example. Imagine that a wide receiver is supposed to run an in-route on a particular pass play. However, this wide receiver is slightly confused and thinks they are supposed to run an out-route. On a standard play, if the quarterback simply throws to the spot that the receiver is supposed to be, this confusion is likely to result in an incompletion at best or an interception at worst. Now let’s imagine that a linebacker has blown up the play by blitzing through the line and the quarterback has to scramble toward the side of our confused wide receiver. Suddenly, the wide receiver’s mistake is the perfect solution. A pass can be completed and everyone’s happy. In the film room, the coach that called the play knows what was supposed to happen. The coach knows that they were lucky the receiver got confused on the play. An outsider watching that play doesn’t know that. To the outsider, it looks like a brilliant play call.

So, is it possible that we will ever see a quarterback bust like JaMarcus Russel again? Of course it’s possible. Not only is it possible, but it is likely. Why? Because, as Dan Pink says in a very good TED talk, the solution is not more of the wrong thing. We don’t need to watch more tape because we’ve proven we cannot effectively use other people’s tape to make accurate predictions. Instead, we need numbers that truly predict performance.

Why do I bring this up now? Well, because I’ve run my first projections using 2012 NCAA FBS data. This model is still in its “toddler” phase, so I’m not ready to start throwing out names just yet. I have days upon days of work to do before I’m solid on the numbers. That being said, I don’t anticipate the numbers changing outlandishly.

Prepare yourself. Some tremendous quarterback busts are coming.

page 1 of 1
Welcome , today is Wednesday, March 21, 2018