Apr
I’m a supporter of mathematical modeling as another way to get a handle on what might happen in an influenza pandemic. But a recent paper by the group at London’s Imperial College, published in Nature, shows what can happen when modelers allow their work to bear more weight than it can sustain. When a prestigious scientific journal, Nature, publishes such a paper, it also gets attention it wouldn’t get if published in a more appropriate place — meaning a place where its scientific contribution could be judged in the usual way, not under the glare of global publicity. I’m not blaming the wire services. The reporting on this paper is pretty high quality. I am blaming the scientists and the journal. But first, a sketch of the findings:
Closing schools during a flu pandemic might slow spread of the disease but probably won’t have as big an impact on overall cases as some pandemic planners hope, a new study suggests.
British and French researchers reported that one in seven cases of pandemic flu might be averted if schools are closed and parents ensure that dismissed children don’t simply congregate elsewhere, such as in formal or informal daycares or at the mall.
[snip]
Earlier modelling studies have predicted closing schools could dramatically lower the number of cases in a pandemic. This new work suggests those studies may have been overly optimistic.
“I think our predicted impact is quite limited, but not so limited that school closures should not be considered as an option,” said senior author Neil Ferguson of the department of infectious diseases epidemiology of London’s Imperial College.
“I mean, we’re in the regime of ‘It may be worthwhile, but the costs need to be borne in mind.”‘ (Helen Branswell, Canadian Press)
One of the problems in modeling is trying to estimate the effects of particular parameters, like transmission in school settings. These researchers used data on school breaks in France, where effects on influenza-like illnesses could be estimated from concurrent medical data. That’s interesting and this kind of science is worth doing. Imperial College’s Neil Ferguson and his group are well-published and adept mathematical modelers. They know what they are doing. But Ferguson is not always careful about what he says about it (and this is not the first time). Branswell quotes him about practical consequences he believes are implied by his model results, consequences he should have little faith in as these kinds of models are complex, depend on many assumptions and are often run under conditions intentionally meant to isolate certain features, like school closing, independent of all other factors. The Canadian Press reporter, Helen Branswell, is the best in the business. I feel quite confident she quoted Ferguson accurately and in context.
These models are genuinely useful when used as one would use a laboratory experiment, a highly stylized and somewhat artificial setting that can still tell us important things if properly interpreted. Models allow us to get a feeling for the relative effects of different parts of a very complicated, interacting system, but a system where the impact of one intervention like closing schools may be strongly affected by another intervention, like reducing attendance at work of adults. It may also us to see the range of possible behaviors, behaviors that are often counter-intuitive. But the effects need to be seen as qualitative, not quantitative. The equations are highly non-linear, meaning in practical terms you cannot take the system apart, analyze the parts separately, then try to put them back together again. They are coupled so each affects the other. The estimates of effect given here are not only ridiculously over precise, but almost certainly biased in magnitude and direction in ways we don’t know and can’t predict.
This kind of paper and the attendant publicity doesn’t help anyone. Michael Osterholm, Director of the Center for Infectious Disease Research and Policy (CIDRAP) is right to say it is unhelpful for planners, or worse, confusing. There is a sense in which this kind of modeling could even be said to be irresponsible, although I won’t go quite that far. It isn’t the modeling effort itself. It is the promotion of the work that I object to and Ferguson, like many scientists, is promoting his work. Nature also deserves blame here. Nature is probably the most prestigious scientific journal in the world and I hold it in high regard. But they shouldn’t have given any of their precious print real-estate to this paper. They did it because they know it makes news. Nature, like the scientists, was promoting itself. This is a paper that would have been better published in a specialty journal, not in a high profile journal. It does not make a significant and urgent contribution to the scientific literature. It’s just another modeling effort, another data point in that literature.
A year ago I wrote seventeen posts here describing a mathematical modeling paper in great detail. I thought then, and I think today, that the paper I chose was extremely informative, very well done and a genuine contribution to how we think about what might or might not be going on in the complicated dynamics of a pandemic. I wanted to show the value of this kind of work. That paper was also measured and clear in what it did and did not mean and its authors, Marc Lipsitch and his group at Harvard, did not promote its significance inappropriately.
I can’t say the same for this latest paper by Ferguson and his colleagues or the editorial decision by Nature to publish it.
Just our opinion, of course. But then it’s our blog, too.