Analytics in Washington

Comments: No Comments
Published on: December 2, 2014

Tony Kornheiser made some interesting statements on the radio the other day regarding what the organization in Washington D.C. should do about their terrible football team. You can read a more in depth piece about those comments here, but the main thrust of the situation was that Washington should start looking at analytics to improve their player selection process.

On the surface, I agree with this position. I am a firm believer in using useful mathematics to improve decision making processes. I started this blog in an attempt to inform people that we can develop individual player analytics in football and predict something about team performance from them.

However, in my humble opinion, Washington will not be the place that the football analytic revolution begins. Mostly I think that because of the actions of the team’s owner, Daniel Snyder. Before I say what I’m about to say, I want you to know that I have no intimate knowledge of Daniel Snyder. I’ve never even met the man. But I can observe his behavior and his public facing behavior leads me to believe something very important about Daniel Snyder and how he may see the world.

It is my impression that Daniel Snyder loves certainty. His behavior seems to follow a profile of X will do Y, Y will do Z, we want Z so therefore let’s go get X. He made a tremendous amount of money entirely on this principle. He had a small company that wanted to do something slightly different, he would go out and acquire another company that did that specific thing, and incorporate it into the original company’s machinery. He’s made his entire living on being able to understand the needs of his organization and then trusting that the assets he spends a tremendous amount of money to acquire will become worth more than what he originally paid for them.

Don’t get me wrong, Snyder’s certainty has served him well in the contexts he was in when he made his billions. Certainty can be a good thing for business leaders, mostly because it allows them to remain leaders. Humans are really bad at differentiating confidence from competence. Projecting certainty creates an environment where people will follow you. So, in some contexts, having lots of certainty makes a lot of sense. The only problem is, football is not one of those contexts.

It’s like my colleague who had difficulty driving on ice. She was originally from California and went to college in Arizona, so she never had to learn about what driving on icy roads is like. In addition, the person who taught her how to drive was a stock car driver. She was taught that to make the most efficient turns, you steer into the bottom of the turn, accelerate quickly through the bottom of the turn, and then steer out on the high side of the corner. And that works well for getting through turns quickly and efficiently. In addition, living in California and Arizona means you never have to confront the fundamental assumption that driving in such a way rests on – traction. If there’s no traction – like say when you’re driving on ice – one must drive in a completely different, opposite way. Accelerating through the bottom of a turn is a really good way to spin your wheels and wipe out. Instead, you have to always ensure that changes in direction never coincide with changes in speed. You can do one, but not the other. In my mind this is what Snyder is doing. He is taking a method that has always worked in the past, applying it to a different context, not recognizing that the underlying reality is different, and wiping out on the ice that is the process of building an NFL team.

You can see this in the future draft capital he gave away to move up to the #2 pick in the 2012 draft to get RG3. Generally speaking, it is a really bad idea to give away future draft picks to move up.  But such a strategy does make sense in a particular light, the light of certainty. If you absolutely feel like you know that this one particular player is going to work out, then it makes every bit of sense to act as Snyder did in 2012. Unfortunately, in football, having such certainty is disconnected from reality. The dirty little secret about player evaluation in football is that nobody knows who’s going to be good or bad. There are too many things to take into account. The amount of error in prediction is so astounding that no human brain can comprehend it. The best mathematical model I can create was accounting for about 15% of what makes a good NFL quarterback at last check. The reality of the NFL says that the way to create a winning team is to stockpile draft picks, evaluate everyone as if they were all drafted in the same round, and repeatedly draft multiple players at the same position (i.e. at least try out a new quarterback every single year).

The mindset you need to build a football team analytically is a mindset of uncertainty. You must accept the general premise that no one knows anything about anyone, the best models will get you 15% of the way to where you need to be, and you need to put yourself in a position to make luck work for you. I do not believe an individual like Snyder – a self-made billionaire used to projecting certainty from a leadership position – would value these qualities. There’s already the story of the economist that Washington hired to do analytic research for them in 2006 who quit after seven weeks of being marginalized in the organization.

The analytic revolution in football is coming quietly. The teams that end up doing analytics very well are not going to make a bit splash about it. The Seahawks and Packers come to mind as teams that, I believe, are on the forefront of the football analytics movement but are not saying a public word about it. Washington is simply not the place where people will be quiet about a new idea. And, ultimately, talking a new analytics department or bringing in some fresh-faced savior with their fancy mathematical model while demanding mechanistic links between actions and outcome will result in utter failure of the analytics process. If Washington brought someone like me into the organization, I feel like Snyder would demand I hit the gas at the bottom of the turn.   And while I might not be certain about much in football I am certain about this. Either I’d have to jump from the car or we’d both wipe out together.

SeaWorld and the NFL

The orcas at SeaWorld are getting a new habitat. The new habitat will cost SeaWorld hundreds of millions of dollars and basically double the size of the habitat the orcas currently have.

Several events have conspired together to move SeaWorld toward building this habitat, but the catalyst can be traced back to a large male orca named Tilikum who murdered Dawn Brancheau, an experienced trainer who was working with him, in 2010.

A key point in the debate about whether or not “murdered” is an acceptable word for the events that transpired rests on whether or not any trainer at any level of experience is safe in the pool with an orca whale. In court, SeaWorld contended that trainers well versed in methods of animal learning and operant conditioning were perfectly capable of controlling a 3 ton whale. The Occupational Safety and Health Administration, on the other hand, contended that no amount of contact with orcas could be considered safe. After some legal wrangling, it was decided that any and all contact between orcas and trainers had to be done with a solid material, such as a concrete barrier, between the trainer and the whale. Direct contact was no longer allowed.

I’ve been returning to my feelings about the SeaWorld incident a lot during this week of terrible football fandom that just won’t end. I wonder about organized football, both in college and the NFL, from the perspective of a workplace. I think about the increased rates of brain damage, along with the recent instances of domestic violence, child abuse, and rape and I wonder what this game that I love to watch is doing to the individuals that get a chance to play it at a high level.

Last week I wrote about one of the great benefits to using statistical methods for employee selection. Basically, using math to find people means it is much easier to find other people should the ones you originally find not work out socially. That’s a great boon to employers in a typical workplace. They become less beholden to talent. Talented people no longer become insulated from the consequences of socially unacceptable acts. And, for any typical place of employment, that works quite well. However, I did not consider in that piece that the NFL is not a typical place of employment. Professional football has some troubling facts associated with it. As a psychology professor, I know all about what brain damage to the prefrontal cortex can do to an individual. Increased emotional reactivity and impulsivity are just the tip of the iceberg. I wonder if this game is destroying the lives of the people that play it. And I worry that anyone that works to identify and project success from one level of football to another, including myself, may be complicit in that destruction.

I am well aware that this blog isn’t particularly popular. As of this writing, I have 45 followers on Twitter and I get about 6 readers a day, 3 of which are spam crawlers. I am, at the moment, insulated from a looming ethical dilemma. No one with actual decision making power is calling me to learn my opinion on whether or not a particular player should be granted the “privilege” of continuing to play the game. The minute that happens, though, I will have an important decision to make. I will need to choose whether or not I should use my intellect to grant someone else the opportunity to potentially destroy theirs in the name of a sporting contest. I will need to decide if I believe the game of football can be played in a way that doesn’t destroy the lives of its players or if I believe my ability to identity talented football players is the same as placing the best trainers in a pool with 3 tons of socially maladjusted rage.

I do know one thing. Talking about football has never seemed more hollow than in these last 10 days. The heart-rune of my fandom is ripped. I’m not sure it will ever heal.

2014 Draft in Review

This will probably be my last post before I go into summer hibernation to work on my receiver model. I was going to review the first round pf the draft, but that’s been done to death at this point. There isn’t much to be gained from rehashing all the details. Instead, I only want to talk about two teams, the Minnesota Vikings and Cleveland Browns.

Minnesota Vikings

Regular readers know that the Vikings are my team. They’re the team I grew up watching and the logo  on the blanket currently draping my couch. So when the Vikings traded up back into the first round, I reacted emotionally. The reaction was about the trade-up, not the actual pick itself. I was worried the Vikings had learned the wrong lesson from their experience in the draft last year.

Last year, the Vikings gave up a number of picks to move back into the first round and make a third selection in the first round. For doing this, they were applauded, given high post-draft marks, and given general kudos all around. The problem is that the actual move was a terrible one. And the reason the 2013 trade was terrible was the number of picks they gave up to make it. Every single credible analyst has shown that the way to get the most value out of the draft is to make more picks (see this new, inventive analysis showing how number of picks predicts total draft value). So the psychological problem is a problem of reinforcement. The Vikings made a move. In the short term, they were told that this move was a great idea. However, the negative consequences of the move they made will either never be realized because the Patriots rarely play the Vikings, or they won’t be felt for three to four years down the road, which is far too long to learn something from reinforcement. And not only that, but the Vikings got incredibly lucky that their target in the trade up is showing promise as a wide receiver and kick returner. All this is positive reinforcement for making a trade up more likely in the future.

I was worried that the Vikings had learned the wrong lesson from their experience last year. I was worried they were going to give up a two to three picks to move back up into the first round as they did last year and then not recoup those picks in the later part of the draft. I’m very happy to say I was wrong. At the end of the draft the Vikings had made, by my count, the 4th most picks of any team in the draft. That’s a path to success through the draft if ever there was one.

As for the actual pick itself, I don’t mind the Vikings picking Bridgewater. My model isn’t precise enough to get upset about a difference between a predicted passer rating of 80 vs. 75. Good luck to the Vikings in the 2014 season.

Cleveland Browns

The Cleveland Browns are run by Blitzwing from Transformers: Animated. If you don’t know Transformers, Blitzwing is a triple-changer. He has forms of a jet, a tank, and a robot. To drive home the “triple” metaphor, the character has three distinct personalities. One of them is cunning, cold, and calculating, one of them is emotional and explosive, and one is completely wack-a-doodle. Or, as my 4-year-old nephew named them, Smart, Dumb, and Scary.

 photo blitzwing.jpgAny move the Browns make seems to fit into one of these three categories and you can never tell which personality is going to be in control for any specific decision. They achieve great outcomes, but project this image of ineptitude. They do really smart things to set up the situation in their favor, but then bungle the execution.  You could do Homer-Buying-the-Cursed-Krusty-Doll for an entire day with this team.

They commissioned a $100,000 study that (supposedly) said Bridgewater was the best quarterback in the draft.

That’s good!

They released the results of that study before the draft.

That’s bad.

They entered the draft with 10 selections.

That’s good!

They ended the draft selecting 6 players.

That’s bad.

They traded down with the Bills and secured picks in future drafts.

That’s good!

They gave up a pick to move one spot up.

That’s bad.

They selected Manziel with the 22nd pick.

That’s good!

They outbid three other teams to trade up to get there.

That’s bad.

They signed Joe Haden, one of the best DB’s in the league to a big contract.

That’s good!

They also signed Miles Austin, a 30-year-old, oft injured wide receiver who hasn’t been productive since 2010.

That’s bad.

But he came with a free frogurt.

That’s good!

The frogurt is also cursed.

Can I go now?

How can a franchise make such good decisions at one time and such bad decisions 15 minutes later? Being run by a robot in disguise with dissociative identity disorder is the only possibility I could come up with.

How the Combine is Like Diagnosing Mental Disorders

Comments: No Comments
Published on: February 23, 2014

This is the Diagnostic and Statistical Manual of Mental Disorders, 5th Edition – commonly known as the DSM-5.  It is the newest edition of the DSM and was just updated from the 4th edition in 2013.

This manual serves two purposes.  First, it provides a sort of dictionary for mental disorders.  By that I mean that it takes collections of symptoms and gives them a name.  This was one of the first purposes of the DSM.  It was handy to make sure that people that were researching different constellations of symptoms were calling those the same thing.  Without that common language, it’s very difficult to make progress in research on mental disorders.  The second purpose of this document is to provide insurance codes for clinicians so they can get reimbursed by insurance companies for their services.  If you are treating someone for a mental disorder, ostensibly they have something.  If you want an insurance company to give you money for a treatment, you have to tell the company what the patient has.

You might not think it, but the DSM-5 and the NFL Combine have a lot in common.

You see, the DSM-5 as a resource actually kind of sucks.  You don’t have to take my word for that.  The Director of the National Institute of Mental Health, the primary government funding body for mental health research, recently made a statement saying that “…patients with mental disorders deserve better [than DSM-5].”  You can read the full post, but to summarize, the director is saying that what is stated in the DSM-5 does not hold much water when we subject the assumptions and statements of fact contained in the book to serious, empirical, systematic testing.

In the same vein, the NFL Combine kind of sucks.  We drag 300 very large men to Indianapolis every year to collect a large amount of largely useless data.  In some cases, like the 40-yard dash, that data is useless because it doesn’t predict anything important that decision makers care about.  In other cases, like hand size for quarterbacks, we see a number, can come up with endless theories about what that number means, but really have no idea what to do with that number.


The DSM-5 and the NFL Combine both suck for the same reason – epistemology.  Epistemology is the notion of how we know what we think we know.  In other words, epistemology is the study of the criteria and factors we use to determine truth.  In the case of both the DSM-5 and the NFL Combine, we have decided to determine truth on the basis of authority.

In the case of the DSM-5, the book says there is a particular disorder called, for instance, borderline personality disorder and that disorder has a certain set of symptoms.  And the reason the book says that is because a high ranking and highly influential psychiatrist said so.  If you look in section three of the DSM-5 (the section on new and emerging trends) you will see a dramatically different picture of personality disorders based on evidence and collected data.  You can still get to a disorder called borderline personality disorder but the data paint a much different picture of the disorder and what to do about it than the “official” section of the book that clinicians must use to get reimbursed for their services.

In the case of the NFL Combine, we have decided that a small subset of tests will be the ones used at the Combine.  It doesn’t really matter that most everyone recognizes that the tests are largely worthless.  In modern times, even NFL teams don’t really use the 40-yard dash to evaluate wide receivers.  The correlation between draft position and 40-yard dash time for wide receivers since 2011 is 0.25 for relative draft position and 0.28 for absolute draft position.  Statistically significant correlations, but accounting for such a small percentage of the decision (about 5%) that they’re barely worth talking about.

So why do we continue this way?  Well, because authority tells us that this is the way things are.  Most NFL teams know that the Combine drills are useless, but the NFL itself wants to maintain a sense of authority.  And so everyone continues to talk about 3-cone drill times and hand sizes, and Wonderlic scores and all sorts of other useless data that should make no difference to anyone trying to find the best football player.  Change is unlikely to happen because that would mean the people that put on the event would have to admit that their publicly stated authority isn’t as correct and proper as they have indicated.

However, the data definitely show that the Combine is a large waste of everyone’s time.  Sometimes the wrong data is collected, sometimes the right data is used in the wrong way, and sometimes teams have no idea how to use the data they get.  In all cases, this amounts to useless data and incorrect evaluations.  And until the authority figures can admit that what we knew 30-40 years ago is different from what we know now, we will continue to get poor outcomes with both mental health treatment and evaluation of NFL players.

Anchoring on the Combine

Categories: NFL, NFL Draft, Psychology
Comments: No Comments
Published on: February 24, 2013

Since it’s Combine time, I was going to write up a piece about how Combine data is reasonably meaningless.  I was going to say that it’s a very strange and expensive way to collect largely meaningless data.  I was going to say that scores and measurements taken at the Combine don’t reliably predict performance.  We spend time, money, and effort all to collect data that make our decisions worse.  I even had a cute little story about shooting baskets from the bleachers during my first ever basketball practice in 5th grade.

But while researching the piece I found analysts that had already written extensive write-ups detailing exactly which metrics are important at which position.  The conclusion is that most positions have at least one metric that predicts something about career success, but they don’t predict much and they don’t predict well.

After reading the fine work from the folks at Harvard, I dug deeper.  Turns out, lots of people have already hit on the notion that the Combine is overblown spectacle.  I even found scouts willing to say you shouldn’t let Combine scores influence your film grades.  One went so far as to say that the teams that know they should forget about Combine results are the teams that make better decisions.  I even turned on an afternoon sports-talk television show and saw that 65% of responding viewers also believe too much emphasis is placed on Combine numbers.

So then is the Combine just entertainment spectacle designed to make money?  If it was all it was, I wouldn’t have a problem.  I would be perfectly happy if Lucas Oil Field hosts the event so they can sell hot dogs to reporters and the NFL sells the broadcast rights, and we all walk away with fatter wallets.  But that isn’t what happens.  Combine results influence when players are drafted.  The Berri and Simmons paper I referenced last time also includes an analysis of where players are actually drafted.  We see the following Combine numbers influence draft position for quarterbacks; height, Wonderlic score (don’t get me started), and 40 yard dash time.  And this is where I have a problem.  We have an event designed to collect relatively meaningless data.   Data that statisticians, scouts, and the general public all believe is relatively meaningless.  Data that almost never helps us make better decisions.  Yet the data changes how we make decisions.  Isn’t that infuriating?

This effect is far more common than you might imagine.  All of us are influenced by a judgment bias often called the “anchoring effect.”  In the anchoring effect, meaningless, often random numbers change people’s judgments.  When I teach this to college students, I use the following demo.  Play along at home if you like.

  1. Write down the last two numbers of your social security number on a piece of paper
  2. Pretend the two numbers you just wrote down represent a dollar amount that we will reference in the next part.  So, if your last two numbers are 25, your reference amount is $25.
  3. For the following list of items, indicate if you would pay more or less than your reference amount for that item.  a) Laptop, b) Shampoo, c) Rack of ribs, d) New office chair
  4. Go back through the items and indicate the highest dollar amount you would actually pay for each item.

When I take all those judgments from a class of 30 people, the following pattern usually appears.  Those with higher reference numbers are willing to pay more for the items compared to people with low reference numbers.

Scatter plot of Price Willing to Pay for Laptop by Last Two Digits of SSN


Scatter plot of Price Willing to Pay for New Office Chair by Last Two Digits of SSN

The last two digits of your social security number are essentially random.  But the value of your social security number is still taken into account.  Being repeatedly exposed to that number creates a set point in our minds.  When we later try to decide on an actual value to pay for the item, everything is processed in relation to this random number.  We set down our anchor on this meaningless piece of information.  We don’t adjust like we should because our brain is busy focusing around the random number.

As another example, look at how prominently the “Minimum Payment” is displayed on your credit card statements.

Research shows if one isn’t going to pay off the whole balance, the presence of a minimum payment field actually reduces payments compared to when the information isn’t there at all.  We anchor on the minimum payment.  In this case making our financial decisions worse.

Which leads us back to the Combine.  The NFL Combine is a place where numbers are everywhere.  Anyone with an internet connection can look up any number they want on any NFL prospect.  And those numbers will stick with us, even if we don’t want them to.  We know they are largely meaningless, we know they will make your decisions worse, yet we are still influenced by them.

We might not care that Quarterback Prospect X ran a 4.40 40 yard dash.  We might not even know if that is especially good for a quarterback.  But now you’ve heard that number.  That number is in your brain.  And that number is going to become an anchor.  It’s going to bump up that prospect ever so slightly in your decision making.  Just like we are more willing to pay for items when considering if we would pay more or less than two digits in our social security number, we’re more willing to pay for prospects that show good Combine numbers.

We know the number is meaningless, but that doesn’t stop it from working its magic.  And this might be the worst problem the Combine creates.  The simple act of publishing the meaningless numbers and getting people to talk about them is going to lead to worse decisions.  And isn’t that the most infuriating thing of all?

Quarterback Myths: Elite Quarterbacks Come From the First Round

Continuing my series on quarterback myths, we come to the notion that “elite” quarterbacks are only drafted in the first round.  It doesn’t take long to run into this myth on the internet.  Everyone from the Bleacher Report to the folks at Harvard will tell you that high quality, franchise quarterbacks are drafted in the first round.  It also doesn’t take long to run into the ridiculous counter-argument of pointing at Tom Brady and going, “nuh-uh.”  I think this myth needs a strong advocate against it.  In this post, I’m going to try and provide a strong argument for why we should believe that quality quarterbacks can be found in any round of the draft.

Let’s first examine the “best-of-the-best.”  What was the draft position of all the quarterbacks that are currently in the Hall of Fame?  Below you see a histogram of the Hall of Fame quarterbacks that played from 1945 to present sorted by draft position.  I have also included Peyton Manning and Tom Brady in this graph as they should be locks for the Hall once they finish playing.  Warren Moon is not included in the graph because he was undrafted.


The first thing to notice is the shape of this graph.  It is a J-shaped distribution with most of the Hall of Fame quarterbacks being drafted early.  At first glance, this might seem to support the argument that quality quarterbacks are more likely to come from the first round.  However, the shape of the distribution should actually lead us to a different conclusion.  In J-shaped distributions, the presence of rare but impactful data points at the right side of the distribution (Tom Brady and Bart Starr) implies the presence and continued occurrence of others (see Nate Silver’s book for this same argument presented with earthquake data).  I am going to argue that the left side of the graph above is not the whole story.  I am certain that there are other quarterbacks that could have been in the Hall of Fame, but were never given the opportunity.

In psychology, we have the notion of a self-fulfilling prophecy.  A self-fulfilling prophecy is any expectation or belief that may alter behavior in a manner that causes those expectations to be fulfilled.  As an example, when teachers are told that certain students are expected to do better academically, those students do better academically.  This is true even when the students are picked completely at random.  This is true even when the students had no idea their teacher was told to expect them to do better.  The teachers alter their behavior around the students they expect to do better.  As a result, those students become more interested in school and begin to show more academic promise.

Looking back at the Hall of Fame quarterbacks, the most recent entrants made their name in the 80’s and 90’s, Marino, Elway, Young, Aikman, and Moon.  I’ve seen it argued that the game is different now.  Since 2000, very few high quality quarterbacks have come from anywhere but the first round, Drew Brees and Matt Schaub being the only obvious exceptions.  So are quarterbacks drafted in the first round actually more talented?

To answer this question, I turn to an article by David Berri and Rob Simmons (2011) published in the Journal of Productivity Analysis (yes a journal exists that is that specific).  They examined all 331 quarterbacks that were drafted between 1970 and 2007 and that played in at least one game in a season.  Once again, at first glance it seems the conventional wisdom is correct.  Quarterbacks drafted at the top of the draft have more passing yards, more touchdowns, participate in more plays, and accumulate more wins.  But Berri and Simmons also ask us to look deeper.  When the statistics are broken down on a per play basis, all of the differences between high and low draft picks wash away.  Quarterbacks at the top have similar completion percentages as quarterbacks at the bottom.  The same is true for passing yards per pass attempt, touchdowns per pass attempt, interceptions per pass attempt, and passer rating.

Quarterbacks drafted at the top of the draft do just as well as quarterbacks drafted at the bottom once you control for playing time.  When we think we see a relationship between draft order and being a franchise quarterback, it is only an illusion.  Instead, what we are seeing is the results of a self-fulfilling prophecy.  Someone expected that high draft pick to do well.  Therefore, that high draft pick, at the very least, gets the opportunity to show what skills he has or doesn’t have.  The massive salaries first round quarterbacks have commanded in the last 15 years only magnifies the effect of the self-fulfilling prophecy.  A rookie quarterback making $15 million a year is not going to sit on the bench for very long, if at all.  The GM is not going to stand by while this massive investment goes unused.  The high paid rookie will get the chance to go out on the field and show everyone if he has what it takes or not.  And if he doesn’t have it, the coach and GM will still stick with him far longer than they should because of sunk costs.

But what about the sixth round pick sitting on the bench as a backup making the league minimum?  He’s less likely to even see the field.  Would we have ever heard about Tom Brady without Drew Bledsoe almost bleeding out on the field?  How many other Tom Bradys have sat on benches waiting for their moment that never happened?  I will wager more than a few.

So now let’s look to the future.  In 2012, we saw a few quarterbacks drafted beyond the first round that started games and made significant contributions.  I expect this trend to continue.  Not because the talent level has increased, but because, with the new rookie salary scale, the financial pressures to self-fulfill the prophecy and stick with a high priced first rounder that isn’t working out will be much less.

All in all, I believe there are quality quarterbacks to be found in later rounds of the draft.  The only question is whether or not they ever get a chance to see the field and actually prove it.

page 1 of 1
Welcome , today is Wednesday, March 21, 2018