Sports, perception and sample-size bias


New York Rangers goalie Henrik Lundqvist is the Tim Tebow of hockey. Actually, he's even better, having won nine of his last ten. Photo by Robert Kowal.

Apologies in advance, but if you don’t follow sports this post may not make much sense.

This afternoon, the Denver Broncos picked up their sixth win in seven games this season with Tim Tebow starting at quarterback. If you haven’t heard, Tebow has what might be called non-traditional passing mechanics, but as many commentators have noted, he “just wins.” There’s a lot that could be said, and already has been said, about the strange way in which quarterbacks are credited with team success in football, but that’s not really the point of this post. Rather, I want to point out how odd it is that seven games — even seven games that include six wins — can be considered so meaningful in football.

This stretch has turned Denver’s season around, to be sure. They lost four of their first five, but now find themselves tied for the lead in their division. But this is only possible because of the NFL’s relatively tiny schedule. Consider that, for a hockey goaltender — probably the only every-game player in North American major team sports with as much impact as a quarterback — six wins in seven games is barely noticeable; it’s less than a tenth of the season. For a baseball player (where there isn’t such a great analogue, since starting pitchers only go every fifth game), six wins in seven games is a good week. You could win your league’s player of the week award in May and be sent down to the minors in June. For Tebow, six wins in seven games is two months and half the season, and it’s especially significant when one of your divisional rivals (San Diego) is imploding at the same time.

But this is all perception; if we’re trying to think about what this seven-game sample means in terms of predicting the larger population of games that is a player’s career, seven tells us nothing. It doesn’t matter that an NFL season is only 16 games long; seven games don’t provide enough observations to reduce the error to an acceptable level. If we were to look at a proportionally similar number of baseball games — 70 — we’d keep the proportion the same, but reduce the sampling error by examining nominally more cases.

So where does this perception error come from? Is it just the kind of rank innumeracy we see in many contexts? Maybe, but I suspect there’s also an important media effect as well. Sports media — both reporters and game broadcasters — and the sports culture they’re embedded in frequently express hostility toward data-driven strategy. Narratives and tradition rule in sports, and when data contradict them it’s because data can’t possibly figure out the relevant “intangibles.” Noting that a seven-game span isn’t really an illuminating sample gets in the way of a lot of narrative structure.

Share this post:

    Tags: , , , , ,

    • Guest

      Klosterman wrote about this for Grantland, hitting at the media narrative’s effect a little:http://www.grantland.com/story/_/id/7319858/the-people-hate-tim-tebow

      The final paragraph sums it up, I think:

      “The machinations of his success don’t matter as long as they’re inexplicable. 

      The crux here, the issue driving this whole “Tebow Thing,” is the matter of faith. It’s the ongoing choice between embracing a warm feeling that makes no sense or a cold pragmatism that’s probably true. And with Tebow, that illogical warm feeling keeps working out. It pays off. The upside to secular thinking is that — in theory — your skepticism will prove correct. Your rightness might be emotionally unsatisfying, but it confirms a stable understanding of the universe. Sports fans who love statistics fall into this camp. People who reject cognitive dissonance build this camp and find the firewood. But Tebow wrecks all that, because he makes blind faith a viable option. His faith in God, his followers’ faith in him — it all defies modernity. This is why people care so much. He is making people wonder if they should try to believe things they don’t actually believe.”