Apathy Sketchpad

Recently, I have been reading Ben Goldacre’s Bad Science. The book, predictably, covers much of the same ground as his Guardian column of the same name, but deeper, and in logical order: chapter one covers very simple claims made by cranks, and shows the curious reader how to test them at home. This is very much a play-along-at-home kind of a book (for doubters, or readers who just like that sort of thing). The following chapters each (mostly) examine the claims, methods and tactics of another form of deception or pseudoscience, each a bit more subtle than the last, and gives the reader the mental tools to examine them at every stage. It builds into a good understanding of trial design — by the end of the book you should be spotting some things before they’re flagged in the text. (That’s a good feeling. I like books that make me feel smart.)

I can’t personally vouch for this teaching: I had much of it drummed into me when I started my PhD (because it’s important), but it’s clearly expressed and illustrated with a range of examples somewhere between lavish and obsessive. The examples also show how universal the methods are: the chapter on Homeopathy also explains about good experimental methods, overuse of antibiotics, superstition, detox programmes, acupuncture, and meta-analysis and its role in assessing the effectiveness of steroids. It’s a structure I like a lot: interestingly varied without seeming like a collection of unconnented anecdotes, and with a strong theme to each chapter and a sense of progression through the book.

The book features chapters themed around the ideas used by Gillian McKeith and Patrick Holford, which discusses their own various publications. McKeith’s, he says,

have an air of ‘referenciness’, with nice little superscript numbers … but when you follow the numbers, and check the references, it’s shocking how often they aren’t what she claimed them to be in the main body of the text

and of Holford’s,

If Professor Patrick Holford is a man of science, and an academic, then we should treat him as one, with a scrupulously straight bat.

I heartily agree. I think that is, in part, why Goldacre’s book, as well as telling you things, shows you experiments and references you can use to check it all yourself (although the references are ferreted away in an appendix where they belong, rather than gaudily paraded in the body text, looking authoritative but basically just getting in the way). I think this is also because the book strongly agrues against the depiction of science as “didactic truth statements from… arbitrary, unelected authority figures”, and that would look pretty silly if presented with no evidence. (Although Goldacre makes a point of never claiming any authority: he doesn’t put “Dr” on the front cover — he doesn’t even have a capital ‘G’.) Transparency is the key to good science, a topic touched upon every time a quack mentioned in the book refuses to publish their research methods.

Holford’s book is also of interest, because Goldacre picks apart his reference list in great detail. Not out of malice or to make fun of him, but because as we know,

If Professor Patrick Holford is a man of science, and an academic, then we should treat him as one, with a scrupulously straight bat.

Okay. Let’s do that.

Goldacre’s book has fourteen pages of notes and references (at least the first edition does). I selected one from each page at random, and checked that it said what he says it said: 2 I couldn’t read, 1 was a note only, the other 11 checked out ane way or another, so that’s basically a 100% hit rate. I didn’t critique the referenced articles, mostly because I don’t have the time or inclination, but none of the research says anything very controversial anyway. Anyone who has read chapter 12 will realise that I am biased here, but that’s why I’m being transparent so you can check up on me too if you like: I’ve put the list of references I looked up, and a brief verdict on each, after the fold.

The other main theme of the book is the problems with ‘dumbing-down’ (a depressingly autological phrase) of the world of science and health: miracle cures, medicalised syndromes for everything, reports of conclusions rather than evidence, and so on. I think this was my favourite passage on that theme:

Nobody dumbs down the finance pages. I can barely understand most of the sports section. In the literature pull-out, there are five-page-long essays which I find completely impenetrable, where the more Russian novelists you can rope in the cleverer everybody thinks you are. I do not complain about this: I envy it.

I’d love to hear what someone with no scientific training thought of this book. But I expect they would learn a lot about the nature of evidence and the mental traps that make it so, acquire a lot of useless trivia about the proponents of pseudoscientific bullshit, learn to spot future nonsense, and have a good laugh along the way. I rate that as worth the price — not least because if you pay attention then the book will pay for itself the first time you don’t buy a pack of useless pills.

Since we’re talking transparency, the author declares that he received his copy of Bad Science free from the publishers and that, not being what you’d call a professional critic, the novelty of this kind of thing hasn’t worn off even a bit.

The references I checked:

1. Schultz et al, p46: I can’t read the paper with my university login, but the abstract agrees with the book and even I recognise one of the author’s names — an impressive feat for a statistician.
2. Shang et al, (PDF) p56: This is a hugely well-known paper. It says what we all know it says. I especially like the funnel plots: like Goldacre’s book, they don’t make drug companies look all that great either.
3. Giona F, p70: This paper is still in press and I can’t find it. To be fair, the claim it supports is of sufficient irrelevance to be in brackets.
4. S Wolf, p78: This seems to support Goldacre’s precis (at least to the extent that I’m reading these, which is much less than I would were I doing real science here).
5. Buske-Kirschbaum et al, p80: This is a great paper, and one which doesn’t seem to have been misrepresented.
6. H Frankfurt, p89: This is a book which I have not read, but is quoted in the main body, so you can judge it for yourself.
7. Moynihan et al (open-access), p153: This is provided more as further reading: the title is enough to justify the ‘claim’, such as it is.
8. p164: This is a note and not a reference.
9. S Mayor, p183: This doesn’t require much reading to check. Yes.
10. Curfman, et al, p202: Again, pretty unequivocal. Yes.
11. Nathan et al, p203: This is also an unequivocal yes.
12. Manning et al, p262: This article describes what the book says it describes, albeit briefly.
13. B Deer, p280: I couldn’t find the relevant information on this page. I would have referenced this page on the same site instead.
14. Grilli et al, p307: This (guardedly — this is Cochrane) supports the claim.