per-capita failure (a pre-theory)

Hi all,

I recently posted a short piece describing my thinking about how statistics and science relate to each other. While writing the post, I struggled with whether today's topic fit into the larger 'science/statistics' idea. I eventually decided the best way was to separate the idea into its own post, call it a 'pre-theory', and explore it further today.

Before I begin, I should clarify for the reader what I mean with these 'pre-theory' blogs. 'Pre-theory' is the best phrase I came up with to generally describe my many half-formed ideas. In other words, a 'pre-theory' is like a guess about the rustle you just heard in the bushes. It is obvious something is there and it might even be possible that the guess is completely accurate. But because the truth remains just beyond my field of vision, it is impossible to be entirely sure of just exactly what it is making all noise.

Let's hope it isn't a skunk.

Tim

*********

I once addressed the crucial difference in how statistics and science approached the challenge of aggregation. For statistics, aggregation is like air. Without it, the field would cease to exist. But because of this dependence, statistics is very poor at recognizing the critical nature of certain individual observations. As a field, statistics struggles to reconcile these outliers with the theories or practices established as part of its status quo.

In science, aggregation also plays a vital role. But the right observation made under strictly defined conditions is sufficient to undo thousands of years of knowledge. In science, the right single observation is given veto power to reject centuries of convention. Truth is the final arbiter ahead of others concerns such as consensus or convention and there are no convenient hiding places to conceal the inconvenient fact (such as The 95% Confidence Interval).

I have no illusions that my post about science and statistics broke any new ground (longtime readers will know most of what I post here fails to break new ground). But like the case of the unhealthy eater who regularly clears out the fridge of rotting vegetables or the couch potato who pays each month for the yet-unused gym membership, knowing what to do and actually doing it remain unconnected concepts for many. The end result is aggregation applied to tiny sample sizes or a 95% probability becoming synonymous with absolute certainty. What are the obstacles preventing us all, especially those educated in one field or the other, from throwing our hands up and demanding the methods of science or the tactics of statistics be applied to the right problems?

One possible obstacle I worry about is the regularity with which aggregated statistics are used to uniformly describe its varied component parts. We describe 'the average American' so frequently that we never bother to question if using the collected information about hundreds of millions to describe individuals represents clear thinking. Every time the numbers are added up so that individuals become families, communities become states, and states become US, we lose the critical details required to break the data back down to the individual level. And yet, we still divide the numbers back down, perhaps in one fell swoop with the national population serving as the denominator, to produce neat 'per-capita' representations about the American you will most likely run into next, on average, most of the time.

I remember the first time I thought about this. It was during a high school statistics class. I read aloud the following basic detail about the 'average' American family - it owns two and a half cars! The statistic simultaneously applied to everyone and described no one. But instead of talking about why people seek out such figures in the first place or why no one with any power bothered to challenge a system mass-producing such inane metrics, the finding was laughed away as a quirk of statistics.

Silly stats!

They can lie about anything...

How much cuter can statistics get?

It was a tiny, irrelevant lie, and we all accepted it. Unfortunately, accepting a tiny little lie doesn't make it true. But each time it happens, the slightly bigger fib applauds. Students forced to describe the mythical garage of two and a half cars will be forced to inhabit two worlds. In the first world, they go to mom or dad and ask to borrow the keys for the WHOLE CAR they want to drive. In the second world, they sit in class and describe a world of half-windshields to the fourth decimal place in the name of greater 'precision'.

Like any skill, practice today improves performance tomorrow. Is statistics an exception? No chance, I think, so to me teaching students how to blubber convincingly about misleading metrics in the present prepares students to blubber convincingly about misleading metrics in the future. The truly creative types might even come up with some of their own such metrics!

Since schools tend to follow a certain progression, generally most students learn about increasingly important matters with each new grade. Over time, a properly trained 'statistician' emerges, armed with the tools to describe individuals using 'per capita' characteristics gleaned from averaging the larger groups these people belong to. What else explains the stories of banks just a decade ago, despite having access to all the statistical know-how needed to make good decisions, approving mortgages for prospective home owners without asking for documentation of sufficient income, a good credit score, or substantial savings? I mean, even just one bit of evidence? Well, you see, as a group everyone was good, so individually everyone must be good...

I suspect these sorts of things happen because of how casually... the accordion rhythm... of individual... to aggregate... to per-capita... is pounded... into our heads... from the earliest age. Most people are not statisticians and just go along with the methods they see used by the experts every day.

I understood, sort of, how this worked from an early age based on a simple mistake people made about me. Many referenced my Japanese mother as an explanation for my math skills while others pointed to my active father as the source of my own athletic ability. Those comparisons were perfectly fine. My dad still runs today and I'm sure if I'd asked my mom would have revealed that she did long division in her head fun.

But my mom was all-Japan in high school tennis and my dad's math skills put him into the top percentile among his high school peers. You'd think basic facts about my background available to anyone with the time to ask two questions might be more commonly known. But the per-capita 'statistics' about Asian basketball players were, like the per-capita 'statistics' about Asian math students, more easily obtained. Jeremy Lin didn't become 'LINSANITY' by getting into Harvard.

But even though these little miscues are difficult to understand, they are pretty easy to explain. Most people prefer to make a bad assumption rather than ask an open question. It is certainly true of me and its a habit I'm trying to change. But it has been hard work, no doubt. Making assumptions is easy if only because it excuses asking open questions. Or maybe the hard thing about it is asking the open questions.

A person who becomes accustomed to making assumptions will become good at making assumptions. After some time, this person will have a tough time even coming up with an open question. Curiosity, like many skills, atrophies if left unused over time. And in a world which seems to encourage per-capita thinking, the first casualty is curiosity about individuals.

I worry this is the cause of the many problems created by statistics. Many successful applications of aggregation are tempting its proponents to step outside the field's area of expertise. It's not automatically a bad thing because per-capita metrics are useful for explaining or describing the characteristics of a group in a way accessible to individuals. Per-capita measurements can put huge numbers into context and can be a powerful way to explain otherwise difficult concepts. But as a tool for understanding others, the science of putting everything on a curve often falls flat.

science v statistics (a pre-theory)

"A good scientist, in other words, does not merely ignore conventional wisdom, but makes a special effort to break it." - from Paul Graham's Hackers and Painters

I've thought for a long time about how often science is misused as a label. It started in college when I noticed how certain subjects - including political science, behavioral science, and notably, economics (because it was often referred to as a 'social science') - referred to themselves as sciences despite deviating from the scientific method whenever doing so proved convenient, conventional, or uncontroversial.

Using my somewhat narrow and certainly 'true on average' definition for scientific process (observe, try to break, and test under strict experimental conditions), it seems to me that the opposite of science is statistics. This is not because statistics necessarily fails to do those things. I remember many economics lessons where our class discussed an observation, the attempts to model it, and the results which proved or disproved the hypothesis.

But in statistics, a 95% likelihood in is good enough to represent fact. Some say that zero to non-zero is the biggest quantifiable jump - if so, the move from 'not 100%' to '100%' is surely the only relevant runner-up.

The key to knowing whether a given topic is science is found in the quote at the top of the post. The sole reason a scientist tries to break conventional wisdom is because, once broken, there is no repair. This is not quite how it works with statistics.

In statistics, a rock which suddenly floated off the ground would be described as 'variation' or 'error'. It would be noted with bemusement but the event would likely pass without further incident. In science, a rock doing the same would not be dismissed as a single fluke observation among many other well behaved rocks - it would be known as The End of Gravity and usher in a new era of scientific progress.

Now, the statistical approach is far from the worst mentality. In many cases, rigorous statistical analysis is the only possible approach. But statistics is useful only if applied to the correct problem and only if the measured inputs are perfectly understood. Otherwise, the key step of aggregating the results is marred with possible bias or distortion.

If a field deludes itself into considering its statistical approach 'scientific', what's the worst that can happen? I think there are two potentially significant ramifications.

First, it makes reversing the accepted wisdom a much more difficult task. I used to wonder what took so long for scientists to figure out that smoking was a major health risk; when I realized how not every 'two packs a day for two decades' smoker got lung cancer, I stopped wondering. The problem of proving the harm of a cigarette was not scientific question but a statistical one because every lifelong smoker with a healthy set of lungs disputed the hypothesis that smoking causes cancer.

The scientific problem thus redefined as a statistical one, the task became to show how an addiction to cigarette smoking reduced quality of life and increased the likelihood of a smoking-induced death. Unfortunately, to do this properly required carefully comparing groups with similar-enough characteristics to identify cigarette smoking as the lone factor influencing different health outcomes. To do such a thing properly using the methods of statistics took, unfortunately, a lifetime.

The second ramification is how it may tempt those with something to lose into leaning heavily on special-case explanations. I could again return to cigarettes here and discuss all the ways tobacco companies fought against research projects into their products. But let's instead imagine if the 16th century world viewed the 'flat Earth' idea in a statistical way. Magellan's descendants would probably still be sailing in circles around the globe, seeking a significant p-value to prove their hypothesis! Science proved a much more efficient guide in this case because it knew the Earth wasn't flat if someone sailed one way and came back from the other. Magellan did it and that was that.

I'd like to acknowledge the reverse of the argument here, even if it risks discrediting ('discrediting') the title of this post, I do not really see this as a one or the other issue. I see science as the combination of Questions Without Answers and Questions Definitively Answered. Statistics occupies a strange in-between: Questions Prematurely But Probably Answered. It sounds stupid when I phrase it that way, of course, but it's totally understandable. A different way to describe the above:  conclusions from good statistics often predicts future science. Again, clunky, but the main idea here is the role statistical methods play in fulfilling a basic human need to understand how the world works. Like any product serving a need, statistical results have a healthy market share and will maintain that market share until something better comes along.

I don't prefer one approach to the other despite my tone thus far in this post. I studied statistics all through college and I believe in its approach when applied correctly. And sometimes, I acknowledge the methods of science take a very long time. It is understandable in such cases if people look around for a faster way. But acquiring a taste for cookie dough is no way to resolve the problem of waiting for cookies to finish baking.

I think the key is to know when one approach is more appropriate than the other. Someone able to do this well will have a world of opportunity open up for them. The challenge in determining if a new observation refutes a known truth (science) or if it simply is something to keep in mind for later (statistics) is no doubt a difficult one. But as we continue to accelerate the pace of accumulating irrelevant data about every aspect of our lives, I cannot think of a skill more important to have for navigating the unseen challenges of the future.

token diversity (a pre-theory)

One of the first things I learned in college was that if somebody was talking about diversity then it probably meant diversity didn't exist.

I formulated this theory after comparing how the school discussed two types of diversity – gender and racial. My school was very white, around 90% or so, and we therefore talked all the time about racial diversity. The conversation always made it seem like the demographics were some kind of unfortunate mix-up, perhaps due to rounding error, and that with a little more conversation and a little less action, maybe, some new future of diversity was about to arrive on campus. It might even be the next day! This was the way it seemed to go every single time diversity was discussed during my four years on campus.

On the other hand, the gender demographics were pretty balanced, at least from the simplified perspective of comparing enrolled men and women. Since the numbers were pretty close to even, the topic of gender diversity was never discussed.

It’s possible that my theory needs a little adjustment. It might not be that discussions about diversity happen only if it doesn’t exist – perhaps they simply stop happening as soon as it does exist. Regardless of the exact way my theory should be worded, I think the point is that it is important to recognize the basic idea that if diversity is being discussed then it is almost a surefire signal that it doesn’t exist. And until it does exist, anyone in the ‘out’ group is certain to feel like a token representative of that group within a larger community that is struggling to live up to its stated ideals of inclusion, equality, and opportunity for all.

leftovers: there’s always a better live version (a pre-theory)

There are some obvious problems with the math I presented in this post. The biggest problem is the built in assumption that every live performance has a chance (at the start of the song) of eventually being seen as the best version of a given tune. As I briefly alluded to in the post, this isn’t usually true – a studio version tends to have more investment in terms of production and this makes it more likely a studio version is better than any one live version picked at random.

However, the point of the argument isn’t to pick out one live version and compare it against the studio version. The point is to take the entire set of live versions and consider whether there is at least one song in the set that is better than the studio version. When you factor in considerations such as a band’s natural performance variation, the possibility of an artist becoming a better performer over time, or improvements in technology that make music sound better, it is apparent that the set of live versions will cover a wide range of performance levels. Surely, there is at least one performance in that set which is superior to the studio version, right? Given the immense popularity of concerts, I’m inclined to suggest that most people at least recognize the possibility that the live version might be better than the studio version. Otherwise, why would anyone go to a concert?

Of course, someone who firmly believes in one side or the other in this critical debate isn’t going to care all that much for the generalized arguments. This gets at the larger concept hidden in this post – what does an answer to this question say about a person’s worldview?

I think the person who rejects the premise of the ‘live version’ argument is the sort who demands details before making any firm determinations. This type of personality will want to listen to a few songs before making any decision but, if pressed, will side with the ‘studio version’ argument because the record label has already given it a stamp of approval. In general, this person looks for nuances or exceptions and prefers to make decisions on a case-by-case basis. If the process is unclear, this person will side with an established authority. Society has benefited greatly from the many daily contributions from this type of person. Undoubtedly, there are certain fields or functions that require such people in key positions of responsibility in order to fully benefit us all.

However, this type of person is not cut out for the problems demanding a statistical approach. I think a lot of people (perhaps a great majority) are not cut out for a statistical approach yet it seems like many try their hand at applying some of the tools of statistics in their workplaces, communities, or homes. Nothing beats a try, of course, but when it comes to thinking in terms of patterns, making decisions within generalized frameworks, and considering the impact of small adjustments on long-term outcomes, someone whose understanding requires first accounting for all the contingencies will always struggle to accept how the generalized attributes underlying an idea can sometimes render the little details of any one observation irrelevant.

there’s always a better live version (a pre-theory)

I remember a regular debate I used to have with a friend during high school. This debate was about the song ‘Hotel California’. His position was that a particular live version of the song (in other words, a recording from a concert) was better than the studio version (the recording made for the album that was generally played on the radio). In these pointless debates, I usually took up the opposite position.

If this debate came up again today, I’m not sure I would take up the same side of the argument. This isn’t to admit defeat, of course, for I’m pretty sure I was right about the two versions of ‘Hotel California’ in question. But I think if a similar question came my way today, I would begrudgingly admit that the odds favored whoever advocated for the live version.

This isn’t due to some special understanding I’ve acquired about music in the last decade. Rather, it’s just math. A studio version is recorded once. Now, this recording might be meticulously prepared, impeccably performed, and carefully edited. But it’s still just one performance. The song will then be played over and over, perhaps hundreds of times, and in all these repeated performances I think the odds are pretty good that at least one of the performances will exceed the heights of the studio version (even if this happens purely by fluke).

Here’s the breakdown for you math nerds out there:

n = number of performances
p ( probability best version of a song is the one studio version ) = 1 / n
p ( probability best version of a song is the live version ) = ( n – 1 ) / n

I guess the question comes down to this – are you the kind who bets on 1 out of n... or n-1?

true, false, racist, or… not funny? (a pre-theory)

I was thinking the other day about how a sense of humor is perhaps the most universally admired quality. Some people want so much to be thought of as funny that they’ll say or do almost anything to get a laugh.

It led me to an insight about a problem I seem to encounter far too often these days – casual racism. My insight into casual racism is simple. I think most people who get caught up in these minor, completely unnecessary incidents of casual racism are not guilty of being bigoted – they are guilty of being unfunny.

I think this is explained partly by how we seem to be in general agreement that if something is really funny, it is OK to make the joke, and those offended need to just ‘lighten up’. I’m not here to agree or disagree with that idea, reader – I think I’ve ‘lightened up’ on plenty of occasions in my life (and sometimes I'm glad I did, but not every time).

I will instead add this caveat – though the above theory might be OK, I think very few people are actually funny enough to transcend the offensive nature of their jokes. So when I don’t laugh at something that is borderline offensive, I'd like to clarify that I probably didn't laugh because the comment wasn't funny.

And yet, this train seems to barrel on from station to station. It leads me to a theory, or maybe just a pre-theory for now as I work out the kinks – most people would rather be incorrectly considered racists than correctly considered not funny. If my theory is right – which I think it is (which means it probably is not) – then I think I can understand a little better how casual racism comes about: it is far easier for a good and decent person to clear up a misunderstanding about allegedly being a racist than it is for an unfunny person to change accurate perceptions about a poor sense of humor.