How the Combine is Like Diagnosing Mental Disorders

Comments: No Comments
Published on: February 23, 2014

This is the Diagnostic and Statistical Manual of Mental Disorders, 5th Edition – commonly known as the DSM-5.  It is the newest edition of the DSM and was just updated from the 4th edition in 2013.

This manual serves two purposes.  First, it provides a sort of dictionary for mental disorders.  By that I mean that it takes collections of symptoms and gives them a name.  This was one of the first purposes of the DSM.  It was handy to make sure that people that were researching different constellations of symptoms were calling those the same thing.  Without that common language, it’s very difficult to make progress in research on mental disorders.  The second purpose of this document is to provide insurance codes for clinicians so they can get reimbursed by insurance companies for their services.  If you are treating someone for a mental disorder, ostensibly they have something.  If you want an insurance company to give you money for a treatment, you have to tell the company what the patient has.

You might not think it, but the DSM-5 and the NFL Combine have a lot in common.

You see, the DSM-5 as a resource actually kind of sucks.  You don’t have to take my word for that.  The Director of the National Institute of Mental Health, the primary government funding body for mental health research, recently made a statement saying that “…patients with mental disorders deserve better [than DSM-5].”  You can read the full post, but to summarize, the director is saying that what is stated in the DSM-5 does not hold much water when we subject the assumptions and statements of fact contained in the book to serious, empirical, systematic testing.

In the same vein, the NFL Combine kind of sucks.  We drag 300 very large men to Indianapolis every year to collect a large amount of largely useless data.  In some cases, like the 40-yard dash, that data is useless because it doesn’t predict anything important that decision makers care about.  In other cases, like hand size for quarterbacks, we see a number, can come up with endless theories about what that number means, but really have no idea what to do with that number.


The DSM-5 and the NFL Combine both suck for the same reason – epistemology.  Epistemology is the notion of how we know what we think we know.  In other words, epistemology is the study of the criteria and factors we use to determine truth.  In the case of both the DSM-5 and the NFL Combine, we have decided to determine truth on the basis of authority.

In the case of the DSM-5, the book says there is a particular disorder called, for instance, borderline personality disorder and that disorder has a certain set of symptoms.  And the reason the book says that is because a high ranking and highly influential psychiatrist said so.  If you look in section three of the DSM-5 (the section on new and emerging trends) you will see a dramatically different picture of personality disorders based on evidence and collected data.  You can still get to a disorder called borderline personality disorder but the data paint a much different picture of the disorder and what to do about it than the “official” section of the book that clinicians must use to get reimbursed for their services.

In the case of the NFL Combine, we have decided that a small subset of tests will be the ones used at the Combine.  It doesn’t really matter that most everyone recognizes that the tests are largely worthless.  In modern times, even NFL teams don’t really use the 40-yard dash to evaluate wide receivers.  The correlation between draft position and 40-yard dash time for wide receivers since 2011 is 0.25 for relative draft position and 0.28 for absolute draft position.  Statistically significant correlations, but accounting for such a small percentage of the decision (about 5%) that they’re barely worth talking about.

So why do we continue this way?  Well, because authority tells us that this is the way things are.  Most NFL teams know that the Combine drills are useless, but the NFL itself wants to maintain a sense of authority.  And so everyone continues to talk about 3-cone drill times and hand sizes, and Wonderlic scores and all sorts of other useless data that should make no difference to anyone trying to find the best football player.  Change is unlikely to happen because that would mean the people that put on the event would have to admit that their publicly stated authority isn’t as correct and proper as they have indicated.

However, the data definitely show that the Combine is a large waste of everyone’s time.  Sometimes the wrong data is collected, sometimes the right data is used in the wrong way, and sometimes teams have no idea how to use the data they get.  In all cases, this amounts to useless data and incorrect evaluations.  And until the authority figures can admit that what we knew 30-40 years ago is different from what we know now, we will continue to get poor outcomes with both mental health treatment and evaluation of NFL players.

2014 NFL Draft Predictions: Quarterbacks

Categories: NCAA FBS, NFL Draft, Statistics
Comments: No Comments
Published on: February 12, 2014

Welcome back everyone.  I took a couple weeks off to recharge and enjoy the Super Bowl.  Now it’s time to get to the heart of the matter.

This post marks my predictions for the 2014 NFL Draft Quarterback class.  On the Draft Numbers page you will see predictions for both NFL Passer Rating and ANY/A for every draft eligible prospect.  This class is a very interesting one.  It’s very similar to the 2011 class in its number of potential starters, one.  Unlike the 2011 class, though, the one player that has the potential to start in the NFL isn’t getting very much buzz, is unlikely to be drafted highly, and probably won’t be a first year starter.  That player is Keith Price from Washington.  Many of the other potential prospects will get playing time, some have potential to be career backups in the league, but this class will be very short on quality starters.  All the numbers are available here.

Prediction Model Details

These predictions are generated using Bayesian analysis procedures.  If you would like details on the priors, you can ask in the comments.  The data set used to generate the equation includes all quarterbacks that played FBS football for at least one season from 2007-2012 and threw at least one pass in the NFL during the 2008-2013 seasons.

First off, the analysis finds that Career Completions Away from Average effectively predicts both 4-year passer rating and 4-year ANY/A.

When we make predictions like this, it’s important to evaluate the model to see how precise it is.  When I tell you that Aaron Murray is predicted to have a Passer Rating of 77.5 after four years in the NFL, how much uncertainty is there in that prediction?  Below you will see a plot showing how much we can reasonably expect the predictions to be off.

The plot gives you an indication of how much the predictions based on CAA can be expected to be off.  The circles represent one quarterback in the data set that was used to generate the prediction model.  The vertical blue lines represent the region that is 95% likely that the prediction will fall into.  You might look at that plot and rightly say that there is a lot of uncertainty in these predictions.  And you would be right.  There is a lot that isn’t accounted for by this single number.  However, let’s make a comparison.  One of the best ways to gauge the general league’s opinion of a prospect’s chances of being successful is using relative draft rank broken down by position.  In other words, was this quarterback the 1st, 2nd, 3rd, etc. quarterback selected.  The plot below uses the same data set that was used to generate the CAA plot, but uses positional draft rank to generate the regions.

The direction of the effect is reversed, so it might be difficult to see, but the length of those blue lines is almost exactly the same as with CAA.  Which I like to see.  It tells me I’m on the right track with this thing.  For completeness sake, here is the same plot predicting ANY/A.

page 1 of 1
Welcome , today is Wednesday, March 21, 2018