Anchoring on the Combine

Categories: NFL, NFL Draft, Psychology
Comments: No Comments
Published on: February 24, 2013

Since it’s Combine time, I was going to write up a piece about how Combine data is reasonably meaningless.  I was going to say that it’s a very strange and expensive way to collect largely meaningless data.  I was going to say that scores and measurements taken at the Combine don’t reliably predict performance.  We spend time, money, and effort all to collect data that make our decisions worse.  I even had a cute little story about shooting baskets from the bleachers during my first ever basketball practice in 5th grade.

But while researching the piece I found analysts that had already written extensive write-ups detailing exactly which metrics are important at which position.  The conclusion is that most positions have at least one metric that predicts something about career success, but they don’t predict much and they don’t predict well.

After reading the fine work from the folks at Harvard, I dug deeper.  Turns out, lots of people have already hit on the notion that the Combine is overblown spectacle.  I even found scouts willing to say you shouldn’t let Combine scores influence your film grades.  One went so far as to say that the teams that know they should forget about Combine results are the teams that make better decisions.  I even turned on an afternoon sports-talk television show and saw that 65% of responding viewers also believe too much emphasis is placed on Combine numbers.

So then is the Combine just entertainment spectacle designed to make money?  If it was all it was, I wouldn’t have a problem.  I would be perfectly happy if Lucas Oil Field hosts the event so they can sell hot dogs to reporters and the NFL sells the broadcast rights, and we all walk away with fatter wallets.  But that isn’t what happens.  Combine results influence when players are drafted.  The Berri and Simmons paper I referenced last time also includes an analysis of where players are actually drafted.  We see the following Combine numbers influence draft position for quarterbacks; height, Wonderlic score (don’t get me started), and 40 yard dash time.  And this is where I have a problem.  We have an event designed to collect relatively meaningless data.   Data that statisticians, scouts, and the general public all believe is relatively meaningless.  Data that almost never helps us make better decisions.  Yet the data changes how we make decisions.  Isn’t that infuriating?

This effect is far more common than you might imagine.  All of us are influenced by a judgment bias often called the “anchoring effect.”  In the anchoring effect, meaningless, often random numbers change people’s judgments.  When I teach this to college students, I use the following demo.  Play along at home if you like.

  1. Write down the last two numbers of your social security number on a piece of paper
  2. Pretend the two numbers you just wrote down represent a dollar amount that we will reference in the next part.  So, if your last two numbers are 25, your reference amount is $25.
  3. For the following list of items, indicate if you would pay more or less than your reference amount for that item.  a) Laptop, b) Shampoo, c) Rack of ribs, d) New office chair
  4. Go back through the items and indicate the highest dollar amount you would actually pay for each item.

When I take all those judgments from a class of 30 people, the following pattern usually appears.  Those with higher reference numbers are willing to pay more for the items compared to people with low reference numbers.

Scatter plot of Price Willing to Pay for Laptop by Last Two Digits of SSN


Scatter plot of Price Willing to Pay for New Office Chair by Last Two Digits of SSN

The last two digits of your social security number are essentially random.  But the value of your social security number is still taken into account.  Being repeatedly exposed to that number creates a set point in our minds.  When we later try to decide on an actual value to pay for the item, everything is processed in relation to this random number.  We set down our anchor on this meaningless piece of information.  We don’t adjust like we should because our brain is busy focusing around the random number.

As another example, look at how prominently the “Minimum Payment” is displayed on your credit card statements.

Research shows if one isn’t going to pay off the whole balance, the presence of a minimum payment field actually reduces payments compared to when the information isn’t there at all.  We anchor on the minimum payment.  In this case making our financial decisions worse.

Which leads us back to the Combine.  The NFL Combine is a place where numbers are everywhere.  Anyone with an internet connection can look up any number they want on any NFL prospect.  And those numbers will stick with us, even if we don’t want them to.  We know they are largely meaningless, we know they will make your decisions worse, yet we are still influenced by them.

We might not care that Quarterback Prospect X ran a 4.40 40 yard dash.  We might not even know if that is especially good for a quarterback.  But now you’ve heard that number.  That number is in your brain.  And that number is going to become an anchor.  It’s going to bump up that prospect ever so slightly in your decision making.  Just like we are more willing to pay for items when considering if we would pay more or less than two digits in our social security number, we’re more willing to pay for prospects that show good Combine numbers.

We know the number is meaningless, but that doesn’t stop it from working its magic.  And this might be the worst problem the Combine creates.  The simple act of publishing the meaningless numbers and getting people to talk about them is going to lead to worse decisions.  And isn’t that the most infuriating thing of all?

Hi Everyone!

Categories: Uncategorized
Tags: No Tags
Comments: No Comments
Published on: February 22, 2013

Since Wages of Wins featured my latest post, there’s been quite an uptick in readers (seriously, thank you.  I really appreciate it, both the Wages of Wins folks and the people clicking on the link to read  the story).

So everyone knows the routine, I usually update on Sundays.  I’ve found that if I update any faster I don’t get my day job work done.  So, until Sunday, check out the archive, subscribe to the RSS feed, follow me on Twitter (I don’t use it much, but I do post occasionally), feel free to comment, and share anything you might like.

See you Sunday!

Quarterback Myths: Elite Quarterbacks Come From the First Round

Continuing my series on quarterback myths, we come to the notion that “elite” quarterbacks are only drafted in the first round.  It doesn’t take long to run into this myth on the internet.  Everyone from the Bleacher Report to the folks at Harvard will tell you that high quality, franchise quarterbacks are drafted in the first round.  It also doesn’t take long to run into the ridiculous counter-argument of pointing at Tom Brady and going, “nuh-uh.”  I think this myth needs a strong advocate against it.  In this post, I’m going to try and provide a strong argument for why we should believe that quality quarterbacks can be found in any round of the draft.

Let’s first examine the “best-of-the-best.”  What was the draft position of all the quarterbacks that are currently in the Hall of Fame?  Below you see a histogram of the Hall of Fame quarterbacks that played from 1945 to present sorted by draft position.  I have also included Peyton Manning and Tom Brady in this graph as they should be locks for the Hall once they finish playing.  Warren Moon is not included in the graph because he was undrafted.


The first thing to notice is the shape of this graph.  It is a J-shaped distribution with most of the Hall of Fame quarterbacks being drafted early.  At first glance, this might seem to support the argument that quality quarterbacks are more likely to come from the first round.  However, the shape of the distribution should actually lead us to a different conclusion.  In J-shaped distributions, the presence of rare but impactful data points at the right side of the distribution (Tom Brady and Bart Starr) implies the presence and continued occurrence of others (see Nate Silver’s book for this same argument presented with earthquake data).  I am going to argue that the left side of the graph above is not the whole story.  I am certain that there are other quarterbacks that could have been in the Hall of Fame, but were never given the opportunity.

In psychology, we have the notion of a self-fulfilling prophecy.  A self-fulfilling prophecy is any expectation or belief that may alter behavior in a manner that causes those expectations to be fulfilled.  As an example, when teachers are told that certain students are expected to do better academically, those students do better academically.  This is true even when the students are picked completely at random.  This is true even when the students had no idea their teacher was told to expect them to do better.  The teachers alter their behavior around the students they expect to do better.  As a result, those students become more interested in school and begin to show more academic promise.

Looking back at the Hall of Fame quarterbacks, the most recent entrants made their name in the 80’s and 90’s, Marino, Elway, Young, Aikman, and Moon.  I’ve seen it argued that the game is different now.  Since 2000, very few high quality quarterbacks have come from anywhere but the first round, Drew Brees and Matt Schaub being the only obvious exceptions.  So are quarterbacks drafted in the first round actually more talented?

To answer this question, I turn to an article by David Berri and Rob Simmons (2011) published in the Journal of Productivity Analysis (yes a journal exists that is that specific).  They examined all 331 quarterbacks that were drafted between 1970 and 2007 and that played in at least one game in a season.  Once again, at first glance it seems the conventional wisdom is correct.  Quarterbacks drafted at the top of the draft have more passing yards, more touchdowns, participate in more plays, and accumulate more wins.  But Berri and Simmons also ask us to look deeper.  When the statistics are broken down on a per play basis, all of the differences between high and low draft picks wash away.  Quarterbacks at the top have similar completion percentages as quarterbacks at the bottom.  The same is true for passing yards per pass attempt, touchdowns per pass attempt, interceptions per pass attempt, and passer rating.

Quarterbacks drafted at the top of the draft do just as well as quarterbacks drafted at the bottom once you control for playing time.  When we think we see a relationship between draft order and being a franchise quarterback, it is only an illusion.  Instead, what we are seeing is the results of a self-fulfilling prophecy.  Someone expected that high draft pick to do well.  Therefore, that high draft pick, at the very least, gets the opportunity to show what skills he has or doesn’t have.  The massive salaries first round quarterbacks have commanded in the last 15 years only magnifies the effect of the self-fulfilling prophecy.  A rookie quarterback making $15 million a year is not going to sit on the bench for very long, if at all.  The GM is not going to stand by while this massive investment goes unused.  The high paid rookie will get the chance to go out on the field and show everyone if he has what it takes or not.  And if he doesn’t have it, the coach and GM will still stick with him far longer than they should because of sunk costs.

But what about the sixth round pick sitting on the bench as a backup making the league minimum?  He’s less likely to even see the field.  Would we have ever heard about Tom Brady without Drew Bledsoe almost bleeding out on the field?  How many other Tom Bradys have sat on benches waiting for their moment that never happened?  I will wager more than a few.

So now let’s look to the future.  In 2012, we saw a few quarterbacks drafted beyond the first round that started games and made significant contributions.  I expect this trend to continue.  Not because the talent level has increased, but because, with the new rookie salary scale, the financial pressures to self-fulfill the prophecy and stick with a high priced first rounder that isn’t working out will be much less.

All in all, I believe there are quality quarterbacks to be found in later rounds of the draft.  The only question is whether or not they ever get a chance to see the field and actually prove it.

A Critique of “Quarterbacks by the Numbers”

Categories: NFL, Statistics
Comments: No Comments
Published on: February 13, 2013

Blogging is a double-edged sword.  On the one hand, it’s very motivating to write these posts.  I love that I can have an idea, write it up, and post it for the internet to see all in a day.  On the other hand, this also means that anyone can write up anything they want.  How are you supposed to separate the good information from the nonsense?

Which is why I was disappointed to see this article on  I think the official website of the NFL has some authority behind it.  People will trust what is said because it’s on  But the analysis presented is an absolute mess.  Supposedly, Gil Brandt takes an objective look at this year’s quarterback class to arrive at a top 5 list.  I don’t take issue with the players on the list.  I take issue with the process that generated the list, which is a much bigger problem.  What follows is my general critique of this article.  I’m doing this to try and improve the general state of using analysis to evaluate individual players.  This is an important endeavor and we don’t need nonsense like this cluttering up the joint.

First problem:  The article mentions using a “time-tested formula for predicting future success” without ever mentioning how success if being defined or measured.  Does it predict games started, career passing yards, passer rating, or something else?  All of these things could be defined as “success.”  Some of us might put more stock in one over the other.  Not knowing what the analyst means by success handicaps what we can learn from the analysis.

Second problem:  What is the sample that is used to show that this time tested formula predicts future success?  How many quarterbacks have been evaluated by this process?  How far back in history does this measure extend?

Third problem:  How well does this formula predict future success?  The article states that the formula is a “fairly accurate predictor of future success.”  How accurate is it?  We have methods to get precise values of how accurate a formula is at predicting an outcome.  What is the average size error that the process makes?  What percentage of variance is predicted?

Fourth problem:  I realize I don’t hold the moral high ground on this one, but how is this useful measure being calculated?  There are three categories here, but they are all different things.  They’re all measured on different scales.  I don’t even know what the “rusher points” category means.  Are they being weighted somehow?  Which thing is more important?  Without knowing how the measure is calculated, we can’t replicate the findings.  This means we are stuck taking the analysts word for it.  When you have to take the analyst’s word for it, you should take that analyst less seriously – even when it’s me.  A massive amount of evidence will need to accumulate before you should accept my conclusions at face value.  I realize there are potential economic concerns that factor in, but at least give me a measure that has a unit associated with it.

Fifth problem:  “I simply measured several of the high-profile quarterback prospects according to the numbers (from the 2012 season) that I think are the most important”.  There are two big problems in one single sentence.  The first problem is that there is cherry picking in the data going on.  You can’t just select a few prospects and see what comes up.  You need to evaluate them all using the exact same formula.  Who knows?  If you evaluate everyone, maybe you come up with Ryan Aplin at #2.  The second problem is the term I think are most important.  We have objective, explicit ways to determine whether or not a statistic is important.  Those methods do not care what you think.  The analyst does not have to guess at which statistics are important and which aren’t.  A regression analysis will tell you very clearly.

Statistical analysis has great potential to help inform the scouting process in professional football.  But articles like this posted on respected websites are not helping.  This is not how you do analysis.  This is taking your own subjective opinions, saying “it’s numbers, you wouldn’t understand” and hoping nobody peeks behind the curtain while the hapless readers shiver for the wizard.  Ultimately, this is a flawed process.  And all you’re going to get from a flawed process is a bunch of flawed results.

Quarterback Myths: An Elite Quarterback is Required to win the Super Bowl

As we approach the 2013 NFL draft, I’ve decided to look at some pieces of conventional wisdom about quarterbacks that I believe to be myths.  They are myths according to my continued statistical analysis of the quarterback and wide receiver positions.  First on the list:  A team needs an “elite” quarterback to win the Super Bowl.

Looking around the internet, you see this one accepted with almost no argument.  The line of reasoning says that the NFL is currently a pass-first league.  Without a talented player pulling the trigger, your team is going to be left behind.  I’m going to tackle this one from two different angles:  1) how are we defining “elite” quarterbacks and 2) how those making this argument may be confusing two very different concepts.

Defining “elite”

What exactly do we mean when we say elite?  The word itself conjures the best of the best, the absolute top of the profession.  So what makes a quarterback elite?  I did a very unscientific survey of the internet to find out.  I Googled the term “who are the elite quarterbacks nfl” and found every list that I could claiming to rank active NFL quarterbacks into “elite” levels.  And while this was a very unscientific methodology, the results were surprisingly consistent.  Six active quarterbacks appeared consistently on lists of elite quarterbacks.

The list of active elite quarterbacks with their median rank is as follows.  When a quarterback did not appear on a list, they were given a rank of n+1 where n was the number of players on that list (all listed at least 5 players).

  1. Tom Brady – 1
  2. Aaron Rodgers – 2
  3. Peyton Manning – 3
  4. Eli Manning – 4
  5. Drew Brees – 4.5
  6. Ben Roethlisberger – 5

Here we run into our first problem with our definition of “elite.”  Internet sources seem to believe that 18.75% of starting quarterbacks in the league are “elite.”  This seems rather high to me.  I don’t believe the term elite should include such a large percentage of the league.  But others might disagree with me, so let’s look deeper.

What do all the quarterbacks on this list have in common?  They’ve all won Super Bowls, certainly, but here we have another problem with the argument.  Yes, the list of elite quarterbacks includes all Super Bowl winners.  But can we then say that being elite wins their teams the Super Bowl?  I don’t think we can.  All the lists were made after these players won their Super Bowls.  So, is it that elite quarterbacks win Super Bowls or that quarterbacks that win Super Bowls are determined to be elite?

The elite leads to Super Bowls option is certainly likely, but the opposite direction is equally likely.  Eli Manning provides a very nice example.  His median ranking of eliteness is 4th among active quarterbacks.  But what has he really done to earn his place there?  His team won two Super Bowls on some incredibly lucky plays.  But during the regular season they were constantly teetering on the edge of elimination.  I don’t see Eli Manning as an elite quarterback.  I see him as an average NFL quarterback (average in statistics terms meaning that half are better and half are worse).  This still means that he’s the 16th best person at his job in the entire country, which is something I would like to be some day.  I just don’t believe being average within the population of interest qualifies for the title elite.  I will post some numbers sometime soon to back up these assertions.

We can also look to the narrative for Joe Flacco this season.  When the Ravens were losing and in danger of missing the playoffs, Joe Flacco was not an elite quarterback.  He was a fool for turning down all that money at the beginning of the year.  He was not showing up to play.  He was never going to be considered a top passer in this league.  But then his team wins the Super Bowl.  Suddenly, as if we’re living in the world of 1984, of course Joe Flacco is elite.  We have always thought Joe Flacco was an elite quarterback.  How could you say otherwise?  Look at what he did this season!  The 4th and 29!  The long bomb to win on the road in Denver!  He’s going to be paid as an elite quarterback because that’s what he is and so on and so on.  Is Joe Flacco truly elite?  I don’t know.  Feast or famine, long-ball style quarterbacks are currently a blind spot for my evaluation model, so I will suspend judgment.  But we can certainly marvel at the changing attitudes.  People are talking about him as elite because his team won the Super Bowl, not the other way around.

Confusing Concepts

I’ve tried to convince you that people making lists of elite quarterbacks may be confusing the causal direction of the elite quarterback – Super Bowl winner correlation.  What about the other part of this argument?  Because the NFL is a pass-driven league an upper-echelon quarterback is necessary to win.  Strategic analysis does say that passing is a more effective strategy than running (see innumerable articles on Advanced NFL Stats).  However, passing being the more effective strategy means you need an effective passing game.  It doesn’t follow that you must maximize one component of your passing game to be successful.  Having an effective quarterback helps, but there are many different ways to have an effective passing attack.  You could have a great quarterback.  Or you could have a highly talented group of wide receivers.  Or you could have a very effective play caller that knows how to anticipate and deceive the defense.

As an example of the latter, I point to the Indianapolis Colts.  Now, I like Andrew Luck.  I had him as my #1 prospect from the 2012 draft, as many others did.  However, he didn’t have a particularly effective rookie season.  He was close to the top of the league in interceptions (18, top was 19) and close to the bottom of the league in completion percentage (54.1%, bottom was 53.9%) and near the middle for touchdowns (23).  Yet the Colts still had a very effective passing attack.  They were able to move the ball through the air when they needed to and ended the year with a playoff berth.  However, the credit for that berth belongs to the highly talented, Hall of Fame caliber receivers that Indianapolis has and the effective play calling of Bruce Arians, not to Andrew Luck.

The point is that an effective quarterback and an effective passing game are not the same thing.  The quarterback has a part to play in the passing game, but good receivers and good play calling can make up for poor quarterback play.

So, do you need an “elite” quarterback to win the Super Bowl?  No.  You need an effective passing game.  However, once your team wins the Super Bowl, your quarterback is far more likely to be labeled elite than he was before your team won the game.

New Data Redux: Quarterbacks – 2011 Draft Class

Categories: Uncategorized
Tags: No Tags
Comments: No Comments
Published on: February 5, 2013

One more time on the new data.  This time for quarterbacks from the 2011 draft class.  This is as far back as I’ll go historically.  Any further back and the historical estimators start to degrade and I don’t get as good of predictions.  Once again, a few notes.

1)  The further back one goes in time, the more difficult it is to figure out who was actually eligible for the draft.  So the table is a lot shorter

2)  You fool!  Cam Newton is 7th??!!!  Yep…same issue as RGIII, except magnified.  Cam only played one season of FBS football.  I think it’s actually rather amazing that he was able to accumulate that many positive completions away from average in a single season.

New Data – 2012 Draft Class – Quarterbacks

Categories: NCAA FBS, NFL Draft, Statistics
Comments: No Comments
Published on: February 1, 2013

Hi Everyone,

I’ve added some additional data from the 2012 draft class to give a better sense of how well Completions Away from Average helps predict performance.  I’m not willing to call these true predictions, since I didn’t make them public until after the season started.  However, the numbers are what they are, so it’s not like they are going to change.

A couple things to note.

#1)  RGIII is rated 4th.  This is obviously too low for the performance he had during the 2012 NFL season.  All I can say is that any good model is going to have some margin of error to the predictions.  The question is, why?  The answer is because of injury.  RGIII didn’t get to play a whole lot the 2009 season which really set him back in the ratings.  These predictions are based on career ratings, and to have career ratings you need to be on the field.  Completions Away from Average has nothing useful to predict if the player isn’t on the field.

#2)  I made a rather big deal about Kellen Moore being passed over on draft day when I first started this blog, but now he’s nowhere to be found.  That one is entirely on me.  I started talking before I had all the data analyzed.  What happened was that I only looked at the data from the 2011 college season rather than waiting until I had all the information from each player’s entire career.  Kellen Moore had a very good 2011 season, but didn’t have very good years prior to that, leaving the total at roughly average.

#3)  I’m still quite proud of Russell Wilson being #2 on this list.  I hope my 2013 #2 (Ryan Aplin) comes through like Wilson did.  They are both very similar quarterbacks.

page 1 of 1
Welcome , today is Tuesday, June 27, 2017