03 March 2007

Now We Go Ahunting

As a secret blog, Instapundit is our natural enemy and we wouldn't ordinarily give it even more traffic, but today Instapundit points to a fascinating story in the Economist about why so much medical research is rot. The lead in particularly well-done:
PEOPLE born under the astrological sign of Leo are 15% more likely to be admitted to hospital with gastric bleeding than those born under the other 11 signs. Sagittarians are 38% more likely than others to land up there because of a broken arm. Those are the conclusions that many medical researchers would be forced to make from a set of data presented to the American Association for the Advancement of Science by Peter Austin of the Institute for Clinical Evaluative Sciences in Toronto. At least, they would be forced to draw them if they applied the lax statistical methods of their own work to the records of hospital admissions in Ontario, Canada, used by Dr Austin.
It turns out that, when seeking more than one effect from a database of medical records, scientists forget to adjust their calculations and find causal connections where there aren't any.

The Economist points out that this means we should be suspicious of studies based on databases. But it doesn't point out that we need to be even more skeptical of so-called meta-studies that gather together many studies and tries to tease new relationships out of the agglomerated data. Meta-studies are almost entirely a means of finding a politically correct result. It wouldn't be at all surprising if the designers were also just doing their math wrong.


Harry Eagar said...

Even conceptually, without statistics, the idea that by piling up a lot of mediocre results, you can get a better result is dubious.

Jim Lindgren at Volokh Conspiracy is often good on absurd statistics.

And I am looking forward to receiving a copy of "Useless Arithmetic," which got a favorable mention in the NY Times (that I very much doubt will affect the miserable level of reporting on statistically-derived concepts).

It is by Orrin Pilkey, whose work on coastal erosion I admire, and reportedly it has a lot to say about General Circulaton Models, little of it good.

erp said...

David, may one ask why the secret blog?

David said...

There's no one good answer to that. It has to do with my mood on the day I set up the blog and my odd sense of humor. It has to do with the fact that I posted for a while before I let anyone know about the blog. I also wanted someplace I could park thoughts on things that I was working on/thinking about. It also has to do with the fundamental life change I might go through with in the next few months.

Just a whole lotta reasons.