Not A Triple Tautology

In explaining the phrase épater le bourgeois, Wikipedia uses the wonderful phrase "French Decadent poets." This suggests worlds I had no idea of; poets who aren't decadent, Frenchmen who aren't poets, and even French poets who aren't decadent. Who'd a thought it.

If You:

1. Are in the left-most lane;

2. Are traveling slower than the speed limit; and

3. Have no one to your right,

you are doing it wrong.

Statistics III

The third installment of our little series on statistics is perhaps the most counter-intuitive lesson of all. The lesson is two-fold:

1. A sample only represents the population of interest as a whole if every member of the population of interest had the same chance of being sampled, which is what "random sample" means, regardless of how large the sample is.

2. Given a truly random sample, the accuracy of population estimates based on the sample depends only on the size of the sample, not on the size of the population.

("Sample" is used here to mean a subset of the population available for study, and thus not the population itself.)

Statistics II

Why don't statistics work backwards? There are actually many reasons, but three are key. First, statistics requires data and data requires theory. Second, theory is about causation and statistics are about correlation. Correlation does not imply causation. Third, it just can't work backwards.

1. You can't do statistics unless you have data, so data proceeds statistics. But you can't collect data without some theory that tells you which data is relevant. There is lots of unstructured information available about the world. It can't be analyzed unless its been sorted and classified, and since that necessarily must come before statistical analysis, it can only be sorted through theory. (Even with archival data, it only exists before it's relevant.) No one is ever going to regress corporate performance against CEO hair color because, even though the information is available, it makes no theoretical sense.

2. Theory is a proposed causal relationship between two constructs. Statistics depends upon the correlation between one or more independent variables (the possible cause(s)) and the dependent variable (the supposed effect). You can't get to theory from statistics because correlation does not imply causation, though causation requires correlation. There are lots of pairs of things that correlate, even very strongly, without being causally related. Sometimes that's because both are caused by some third construct. Consider, for example, this graph of the relationship between the number of lemons imported from Mexico and US highway fatalities:



(I stole this graph from Derek Lowe, who in turn stole it from a letter to the editor in the Journal of Chemical Information and Modeling (Johnson, 2008).)

"R² = 0.97" means that Mexican lemons explain approximately 97% of the variability in US highway fatalities over time, an almost unheard of result in the social sciences and much better than lots of studies that have caused otherwise rational people to upset their way of life. Why shouldn't we, then, repeal the traffic laws and just start importing lemons willy-nilly? Because theory tells us that lemons don't cause a reduction in highway deaths, no matter what statistics tells us.

What's really going on in this graph is that we've gotten richer over time, mostly as a result of increased productivity resulting from advanced technology. Richer means that we're importing more luxuries, like Mexican lemons. Advanced technology means that cars can be safer and richer means that safer cars are affordable. (For all I know, it might also be that technology advances have made importing lemons from Mexico easier and cheaper, increasing demand.) In any event, imported lemons and highway fatalities only correlate with each other because they both have causal relationships with a missing factor or two. Only theory can tell us whether a factor is missing and what it might be. Even then, it usually doesn't.

3. The third reason we can't work backward is because that's just not how it works. This has to do with our acceptance of false positives, and with null hypothesis testing, so it's a little more complicated.

Statistics works by trying to figure out, if our two data sets really were random rather than systematically different based on the independent variable of interest, how likely it would be that we would get that distribution. Let's say we suspected, for example, that altitude effects whether a coin lands heads or tails. To test our theory, we flip a coin at ground level and at the summit of Mt. Everest. On the ground, we get 48 heads and 52 tails. At altitude, we get 62 heads and 38 tails. How likely is it that our distribution could happen at random? As it happens, I can test that: the chance of getting that distribution randomly is 0.047, or just under 5%. If I had gotten 61 heads on Everest, the chance would have risen to 6.5%.

As most of you know, the convention in science is that if the probability of a particular distribution being random is less than 5%, we can claim that the two sets of numbers are significantly different. The first thing to note about this is that this is only a convention. It is entirely arbitrary; there is no theory that makes 0.05 better than 0.04 or 0.06. If it were higher, we'd have more false positives, if it were lower, we'd have more false negatives. Just like deciding whether the speed limit should be 55 or 65, in the end all you can do is pick one.

The second thing to note is that this is not the same as saying that the chance that my result is random (and thus wrong) is 0.047. Implicitly, statistical tests compare my actual hypothesis (altitude causes a change in coin flipping results) to an opposite "null" hypothesis (altitude doesn't cause a change in coin flipping results). Generically, the hypothesis being tested is always "these two sets of numbers are significantly different" and the null hypothesis is always "these two sets of numbers are not significantly different." So the 0.047 really means, "the chance that my results are a false positive is 4.7%, if the null hypothesis is true. Of course, if the null hypothesis is false, then my theory is true and my results are necessarily not a false positive. In the real world, we can't know whether my theory is true separate from statistics, but strong theory, well founded on prior results, should be true. In other words, if my theory is convincing, the total chance that my results are a false positive is much less than 5%. (In some fields, the chance that my results are a false positive will be further reduced by replication, but in management we don't do replication.)

Now, what if we try to work backwards? The problem is that, for every hundred random regressions I do, I'll get (by definition) an average of five "significant" but false results even if there's no actual relationship in my data. Once, simply doing the regressions was onerous and something of a brake on this kind of fishing, but now, if I have the data on my computer, I can easily do 100 regressions in an hour. If I do a thousand regressions, I'll have 50 that are significant, and 10 that look really significant (p < 0.01). (One of the annoying things about popular science reporting is the focus on really small values of p. For reasons not worth going into here, if the probability of the null hypothesis being true is less than 5%, it really doesn't matter how small it is.) It is much, much, much more likely (actually, all but certain) that I'll find significance when fishing around than when testing theory. If we work backwards, we have no way of knowing what the chance of a false positive is.

[Because I want to make sure this point is clear, I'm going to beat it over the head a bit. If I come up with a theory and then test it, I know that the chance that my results are random (a false positive) is no more than 5%. Because my theory is strong and based on prior research -- and to be published it has to be vetted by experienced scholars in the field -- I know that the real chance of a false positive is actually quite a bit less than 5%. But if I work backwards, I have no idea what the chance of a false positive is. The particular relationship I found is less than 5% likely to be random, but I know -- by definition, and thus with certainty -- that if I do 100 regressions on completely random data, some of those regressions will be sufficiently unlikely (p < 0.05). In other words, when I'm fishing, the chance of a false positive is 100%, even though it is still true that the chance of any particular significant result being false is less than 5%. If I take that result and then fit a theory to it, I have no idea what the real chance of that result being a false positive is. Inherent in the math behind statistics is the assumption that I am testing a relationship I theorized a priori. That assumption is necessarily violated if I find the relationship and then develop the theory.]

Why can't we just go find significance and then go see if we can develop strong theory? Because we can always find a theory to fit if we know the end point.

Statistics I

Since it's become clear that I'm not going to write one long post on statistics, I've decided to write an infinite series of short posts on statistics.

Don't say I didn't warn you.

In this post, I'll start with the purpose of statistics: the purpose of statistics is to tell us whether two sets of numbers, that differ based on some characteristic that we suspect might be important, are really different. Although it can get very sophisticated (for example, we sometimes don't know the second set of numbers), that's really all statistics does or can do.

In particular, statistics can't work backwards. We can't see that two sets of numbers are different, and then work our way back to figure out what the difference is. Theory must drive statistics, but statistics can't drive theory.

Next time, what alpha means, and what what alpha means means.

The Annotated Brit

The inimitable John Inman:

The Nice Schizophrenic Next Door

We're making plans for a family trip to Montreal next summer, to correspond with a conference I've got to go to. (Peter -- Are you going to be in the vicinity in early August?) One of the conference hotels is the Fairmont Le Reine Élizabeth. There is something so deliciously Canadian about their insistence upon rendering the name of the Queen of England in French; it is just a constant delight.

Irony Is A Dish Best Served Cold

Remember when it was treason to sell the UK-based manager of six US ports to a state-owned UAE company? You probably don't remember who ended up buying out the six US ports instead: AIG, which is obviously a much more trustworthy manager of our precious national assets.

Are There People

... for whom it is good news when the world conforms to their expectations?

Remember The Uninsured?

Every once in a while, I have to forcibly remind myself that the long march to "Health Care Reform" started with the uninsured. It is wrong, we were told, to deny 15 20 30 40 million Americans healthcare. And who can argue?

So, the American people, coming off 8 exciting Bush years, when we had done pretty well but weren't really feeling that we had done much good (mistakenly, in my opinion, but that's neither here nor there), elected Barack Obama to do good -- including by getting health care to the uninsured.

But how to do that. One way, the traditional American way, is to provide health care insurance to the poor, the young and the old and simply let the uninsured show up when they need treatment. The other way is to get health insurance to everyone, which apparently requires spending trillions of dollars, mucking up everyone else's health care and losing sight of the uninsured along the way. The problem is that everyone who has insurance is pretty much happy with their health care, they just wanted to help their neighbors out.

Now, my neighbors are fine people and I'm more than willing to help them out, now and then. But there are limits and there's no point in my helping them out by giving up myself what they don't have. In other words, we might have reached the Robin Hood point, where the rich, having been robbed, are now the poor.

It Looks Too Much Like A Bribe To Actually Be A Bribe

Is It Just Me?

Or is Curious George at the Park Touch-and-Feel an unfortunately creepy name for a children's book?

Thought Experiment

AMENDMENT 28:

Except in time of war, the federal government shall have no power to tax but shall pay for all of its expenditures through borrowing, save only for fees charged to participants that cover the cost of their participation in enumerated voluntary activities.

The Zeitgeist Rolls On

In a New York Times article on fraud in a new book on the Hiroshima bomb, a historian describes the book this way:

“This book is a Toyota,” said Robert S. Norris, the author of “Racing for the Bomb” and an atomic historian. “The publisher should recall it, issue an apology and fix the parts that endanger the historical record.”

Just putting down a marker for the first time I saw the phrase "It's a Toyota" used to mean that the subject is a clunker.

Why Is Internet Advertising So Bad?

We all have, in the back of our minds, the idea that advertising is going to keep giving us our internet for free. But why is internet advertising so bad? If I see another ad about Acai berries, or about how a [fill in your location here] mom makes $77.00 an hour stuffing envelopes, I will cry.

In The United States

One of our faculty members has a habit of adding "in the United States" to any generalized statement made in her presence. If one of us says that pay tends to be the largest factor in employee motivation, she'll say, "in the United States." If we say that investing in research and development tends to be associated with increased profitability, she'll say "in the United States." If we claim that reciprocity is a universal human behavior, she'll say "in the United States."

Her point is two-fold. First, she doesn't believe in universal human behavior. Second, she's reminding even those of us who are positivists -- who believe that there can be generalizable rules of human behavior and thought -- that our conclusions can't outrun our data, and in management almost all data is from the United States. (In the social sciences generally, almost all data is Euro-American. In psychology, most data is from college students. Next time the media tries to tell you about some universal truth derived from psychology experiments -- usually some left wing truth -- remember that all they're really telling you about is how college students behave.)

I'm reminded of this warning by the recent proliferation of supposed "racial code-words" identified by the racialist left and defenders of President Obama. Various observers have suggested that calling Obama "socialist," or "un-American," or "Professor" is subtle racism. As James Taranto notes, "un-American" is not particularly subtle, but it seems odd to object to it as racist. Professor, on the other hand, is a very subtle insult, and not just as racial code.

Why are these insults and honorifics both being described at racist? Partly, I'm sure, simply to deflect criticism of President Obama. But partly for a peculiarity of thought in the United States. For a certain portion of our population, blacks are not only "other," but the only other.

It's fair enough to note that, if asked to imagine the prototypical American, most of us (and not just in the United States) would imagine a white male. "Aha," shout the racialists, "we knew it. Blacks aren't really American." I have a more benign explanation, of course, that allows for the Americanism of women and minorities (who are called minorities for a reason). Our prototypical American is probably also Christian, though probably goes to Church only on Christmas, Easter, to be married and buried. Nonetheless, Jews can be Americans, too. But if you can only imagine one American, you go with the majority.

For the racialists on the left and right, though, our little thought experiment has proved, conclusively, that blacks are the other in the United States. For certain racialists on the left, however, blacks are the only other. They'd admit, if pressed, that women, atheists, Asians, gays, etc., also seem to be other, but only to the extent that their experience is like that of American blacks. When they say that "gay is the new black," they don't mean to imply that black isn't still black.

So, in the United States, the racialists can't admit that calling someone "socialist" or "un-American" or "professor" is really about whatever those words literally signify. Rather, it's a way of pointing out that the object of those words is other than prototypically American, which is to say, black.

Today Makes Me Think ...

That sanity consists of being able to step outside the logic of your own argument.

RIP Dick Francis

Proof is a nearly perfect book.

What To Do, What To Do?

Although it's not really my concern, I'm fascinated by the question of what the Democrats should do about health insurance reform between now and the mid-term elections. Should they do nothing, and alienate their base, or should they force it through and energize their opponents even more?

Their base, which might even earnestly believe that HCR will accomplish something good, badly wants reform to pass. They want the Dems in Congress to pull out all the stops, using the reconciliation process (designed for budget changes that reduce the deficit) to nationalize 1/6th of the national economy over Republican opposition. If Congress won't do that, the base at least threatens not to turn out in the fall. After all, what's the point of voting for Democrats if you can't get one measly earth-shattering, unprecedented reform through?

On the other hand, the Tea Partiers and conservative Republicans hate reform, think that it is the tool of the devil designed to make all our lives worse and think that, by electing Scott Brown, we've won. Cheat us of our victory and we'll turn out in droves. Plus, using reconciliation in this way would be, in effect, the end of the filibuster.

Add to this that the Administration probably does truly believe that HCR is not only good, but the culmination of 100 years of striving for national health care and this might be the last real chance of passing it for a generation. What should they do?

Sometimes I Wonder

why I bother (here and here).

This study is the poster child for bad reporting of social science. The reporting is that the authors found that the judge's race makes an "enormous" or "dramatic" difference in the outcome of discrimination cases. Leaving aside the fact that the study is completely unreliable and doesn't allow us to draw any conclusions at all (which I don't really blame a lay reporter for missing), the fact is that the authors' own results show that the judge's race makes little (R-squared under the best possible circumstance is only 0.03, meaning that judge's race explains only three percent of the variance in outcomes) or no (in the authors' best analysis, judge's race was not significant) difference.

That's actually an interesting result -- if we could rely on it -- since everyone assumes (look at the comments) that judge's race will make a difference. A finding that suggests that we're too cynical is an interesting finding.

P.S. The comment thread at the ABA Journal, which has now degenerated to "it's the JOOOOs," amply demonstrates the problem with bad reporting of bad studies.