Journalists Are Shit, Study Finds

Most of the country’s newspapers (indeed, most of the world’s, judging by a Google News search) have recently published a story about acupuncture, and most of them got the facts wrong. I should declare now that my University of Manchester Athens login doesn’t provide me with access rights to read the full article, but I can read the abstract online, and so can you.

But first, let’s look at what the papers say about it. The Independent has this to say:

Acupuncture is best way to treat back pain, study finds By Jeremy Laurance, Health Editor Published: 25 September 2007 The ancient Chinese practice of acupuncture works better than anything modern medicine has devised for the treatment of back pain, scientists have concluded. In trials among 1,100 patients with chronic lower back pain which had lasted for an average of eight years, almost half (47 per cent) of those who received acupuncture showed significant improvement ’“ compared with barely a quarter (27 per cent) of those given conventional treatment. ... The results showed that 44 per cent of volunteers suffering from back pain showed a significant improvement with sham acupuncture.

The Telegraph said this:

Acupuncture 'best therapy for back pain' By Nic Fleming, Science Correspondent Last Updated: 2:39am BST 26/09/2007 Acupuncture can provide significantly more relief from lower back pain than conventional therapies, scientists say. The Chinese needle treatment was 74 per cent more likely to lead to a sustained reduction in pain or improved ability to function normally than physiotherapy, medication and advice on exercise, according to German researchers. ... However, the study also found "sham acupuncture" ’” in which needles are applied away from points usually used in traditional Chinese medicine ’” to be almost as effective.

The Daily Mail went with this:

Acupuncture 'provides twice the pain relief of standard medicine' By SIMON CABLE Last updated at 09:38am on 25th September 2007 Acupuncture is twice as effective at reducing lower back pain than conventional medicines, according to researchers. But pretend acupuncture, where the needles are inserted less deeply, has also been found to have a similar effect, suggesting that the pain relief could be psychological. ... It had previously been believed that the process whereby needles are inserted into the skin, was only effective if the needles were inserted at precisely the right points on the body. But the study suggests that there are in fact no physical effects at all, and that the healing of pain stems from the patients psychological conviction that they are getting better.

Here’s the BBC’s coverage:

Needles 'are best for back pain' No author credited. Last Updated: Tuesday, 25 September 2007, 07:44 GMT 08:44 UK Acupuncture - real or sham - is more effective at treating back pain than conventional therapies, research suggests. A German team found almost half the patients treated with acupuncture felt pain relief. But the Archives of Internal Medicine study also suggests sham acupuncture works nearly as well as the real thing.

And Channel 4’s (who have been congratulating themselves for their news coverage on More4 lately):

Acupuncture 'better for back pain' Last Modified: 25 Sep 2007 Source: PA News Acupuncture and even sham acupuncture are more effective treatments of lower back pain than conventional therapy, a new study has suggested. ... After six months, almost half of the verum acupuncture group (47.6%) and only slightly fewer (44.2%) of the sham acupuncture group had met this criteria. This compared to only 27.4% of the conventional therapy group.

Each of the above extracts states explicitly that the study shows conventional treatments are less effective than acupuncture, and all of them except The Daily Mail and The Times’ opinion piece state or imply that “real” (or verum) acupuncture is better than sham acupuncture. Neither of these conclusions are supported by the study. (For bonus points, the BBC incorrectly attributed the study to the Archives of Internal Medicine whose involvement was just to publish it, and the Independent mis-spells the name of the university whose employees conducted the study.)

The study’s abstract states that “response rate was 47.6% in the verum acupuncture group, 44.2% in the sham acupuncture group, and 27.4% in the conventional therapy group”. It’s easy to see why the journalists thought that the study showed verum acupuncture was better than sham acupuncture, but the abstract then states:

Differences among groups were as follows:

A p-value is the odds of getting a result this compelling if you assume there’s no difference between the groups. A p-value of less than 0.05 is considered significant, and a p-value of less than 0.001 is preferred. In this case, we see that the results of any kind of acupuncture compared to the control group are very unlikely if the acupuncture has no effect, but there was a 39% chance of getting a result this compelling by chance alone between the verum and sham acupuncture groups. That’s hardly unlikely.

All scientific results have margins of errors in them. We can’t be sure what the absolute value of results are, but we can say there’s a 50% chance it’s within x range, and a 60% chance it’s within y range, and so on. A “95% confidence interval” is the range that we’re 95% sure the answer lies in. Generally, if the range includes zero we shouldn’t draw conclusions. Again, by this test the sham-vs-verum is the only pair of groups that show no difference at all.

This study does nothing to suggest that verum acupuncture is better than sham. Quoting the 47%, 42%, 27% statistics without also mentioning that the difference between the first two is statistically insignificant is misleading.

As for the other conclusion, that acupuncture is better than conventional treatment, that’s not supported either, because the study simply didn’t address that. All the subjects were, according to the abstract, ones “with a history of chronic low back pain for a mean of 8 years”, so they’re obviously pre-selecting people whose back pains don’t respond well to conventional treatment. The 27% who responded to such treatment in the study are therefore likely to be mostly examples of a form of the Placebo Effect and of the Hawthorne Effect, where people who know they’re in a trial respond to treatment better than they otherwise would, even when the treatment is the same. If you selected people whose back pain had just started then you would address the question of how effective acupuncture is compared to conventional treatment. Of course, you’d have a job getting it past an ethics committee.

Slightly more worryingly, the National Review Of Medicine’s article is little better than the Daily Mail’s, opening with a false conclusion, and Nursing In Practice’s was much the same. The Times’ Dr Copperfield observed that it was strange that the conventional treatment was as much worse than acupuncture but didn’t realise why – probably because he’d read the press release but not the original research paper.

Personally, I’m happy to attribute a lot of this nonsense to said press release. It comes from the journal’s parent organisation, JAMA, and says:

Got a backache? Get acupuncture Sep 24, 2007 WASHINGTON (AFP) ’” Acupuncture could prove more effective then conventional treatment in curing back pain, according to a new study released Monday. ... After six months, 47.6 percent of those receiving Chinese acupuncture had noticed an improvement in their condition, along with 44.2 percent in the sham group. Only 27.4 percent of the group receiving conventional therapy however, reported any improvement, noted the study in the journal which is part of the JAMA (Journal of the American Medical Association) group. "The superiority of both forms of acupuncture suggests a common underlying mechanism that may act on pain generation, transmission of pain signals or processing of pain signals by the central nervous system and that is stronger than the action mechanism of conventional therapy," the authors said.

Nowhere does it mention that the difference between sham and verum acupuncture is not significant. (The final paragraph was reproduced by all the newspapers, but it’s really just speculation by the authors of the study. It isn’t supported by the study, which by design does nothing to suggest that the effect is anything other than placebo.)

The newspapers all faithfully reproduced this press release without apparent reference to the original paper. Possibly this is because the press release didn’t contain a link to the original paper, or even the abstract. But frankly I demand more of journalists than this. It took me less than a minute to get from the Independent’s article, via Google, to the original paper. Surely it’s not asking too much for journalists to expend the same amount of effort for their national newspapers as I do for this blog, and do one minute’s research in order to produce a story that isn’t wrong?

Why do we even have journalists if they can’t be trusted to report accurately? What possible use are they, given that?

And exactly how far into the End Times must we be when The Daily Mail have the most accurate report of a scientific study? I think it’s time to repent.