Escape from Model Land by Erica Thompson review – the power and pitfalls of prediction | Science and nature books

“The only function of economic forecasting,” wrote the great American economist John Kenneth Galbraith, “is to make astrology look respectable.” It is characteristic of Erica Thompson’s sprightly and highly original new book on the uses and abuses of mathematical modelling that she dares to turn Galbraith’s verdict on its head. The medieval practice of casting horoscopes, she shows in one typically engaging section that embodies her most important themes, has a surprising amount to teach us about the modern practice of using models to guide policy.

The topic is an exceptionally important and timely one. The Covid-19 pandemic, the climate crisis, and turbulence in financial markets are just three examples of how fundamental mathematical modelling has become to decision-making in many areas of modern life.

Thompson’s argument is not, of course, that scientific forecasting has made no progress over the past half-millennium. Today’s researchers benefit from a world awash with data on natural phenomena and human behaviour, making the raw material for model-building vastly richer than it once was. Mathematical and statistical techniques are far more sophisticated – and we have modern computing power to help us crunch the numbers. These differences make the artificial worlds which modern economists, meteorologists and epidemiologists build dramatically more hi-res than anything the benighted court astrologer could come up with.

But just like their medieval counterparts, today’s “Model Lands” – the hypothetical worlds we construct in order to explore the future – have no practical value until their analyses and predictions are applied in real life. It is in this all-important step – the escape of Thompson’s title – that the parallels between astrology and mathematical modelling become particularly relevant. The central common challenge is working out how much of what we learn in pristine but artificial models remains valid in messy but concrete real life.

One way of figuring this out is quantitative: you compare the predictions of the model against new, incoming data. A critical obstacle here is that predictions based on modern mathematical models, no less than those based on medieval horoscopes, usually depend on an extensive hinterland of assumptions. That makes testing the validity of their forecasts intrinsically difficult: were the assumptions wrong, or was it just that not enough assumptions were included?

Another problem is that the fresh, real-world data needed to test the results is often not even available. It will flood in quickly and easily for day-ahead weather forecasts, for example – but might arrive centuries too late to discriminate between today’s long-term climate models.

That’s why, Thompson explains, a second, qualitative way of determining the success of predictions is much more common: reliance on expert judgment. The pitfalls of this route were also well known to the medieval courts. Only those versed in the most cutting-edge mathematical knowledge were skilled enough to interpret medieval horoscopes. As such, it was in practice impossible for the client to come to their own conclusions. The result was that an exclusive guild, whose true competence remained unknown, ended up marking their own homework. The same could be said today.

Another hazard stalking ancient and modern modellers alike is that they fall in love with the sheer beauty and complexity of their own constructions. Having eaten the lotuses of Model Land, they can’t bring themselves to escape. Scenarios and predictions are simply accepted as if the model actually is real life.

“Such naive Model Land realism,” Thompson warns, “can have catastrophic effects because it invariably results in an underestimation of uncertainties and exposure to greater-than-expected risk.” Anyone who remembers Goldman Sachs’s chief financial officer blaming the global credit crunch of 2007 on the occurrence of “twenty-five standard deviation events, several days in a row” knows what Thompson is getting at. If it couldn’t happen in the model, it just wasn’t meant to happen in real life.

It’s not all bad news. Thompson is a data scientist and mathematical modeller herself, and her book is far from an exercise in model-bashing. It is instead a nuanced and constructive critique of what remains an invaluable analytical method – just not necessarily for the reasons you might expect.

For example, even though the astrologers’ models of natural forces and human behaviour were wrong, the practice of casting horoscopes could still be a useful aid to policymaking. They brought systematic thinkers into the orbit of otherwise impulsive rulers; it allowed the discussion of important, otherwise taboo subjects in the safe context of interpreting the stars; and it could give decision-makers the public narrative they needed in order to act.

The same applies today. As Thompson shows, mathematical model-building can still be a constructive tool, even if the models themselves are flawed. As Dwight D Eisenhower said: “Plans are useless, but planning is indispensable”.

“The only function of economic forecasting,” wrote the great American economist John Kenneth Galbraith, “is to make astrology look respectable.” It is characteristic of Erica Thompson’s sprightly and highly original new book on the uses and abuses of mathematical modelling that she dares to turn Galbraith’s verdict on its head. The medieval practice of casting horoscopes, she shows in one typically engaging section that embodies her most important themes, has a surprising amount to teach us about the modern practice of using models to guide policy.

The topic is an exceptionally important and timely one. The Covid-19 pandemic, the climate crisis, and turbulence in financial markets are just three examples of how fundamental mathematical modelling has become to decision-making in many areas of modern life.

Thompson’s argument is not, of course, that scientific forecasting has made no progress over the past half-millennium. Today’s researchers benefit from a world awash with data on natural phenomena and human behaviour, making the raw material for model-building vastly richer than it once was. Mathematical and statistical techniques are far more sophisticated – and we have modern computing power to help us crunch the numbers. These differences make the artificial worlds which modern economists, meteorologists and epidemiologists build dramatically more hi-res than anything the benighted court astrologer could come up with.

But just like their medieval counterparts, today’s “Model Lands” – the hypothetical worlds we construct in order to explore the future – have no practical value until their analyses and predictions are applied in real life. It is in this all-important step – the escape of Thompson’s title – that the parallels between astrology and mathematical modelling become particularly relevant. The central common challenge is working out how much of what we learn in pristine but artificial models remains valid in messy but concrete real life.

One way of figuring this out is quantitative: you compare the predictions of the model against new, incoming data. A critical obstacle here is that predictions based on modern mathematical models, no less than those based on medieval horoscopes, usually depend on an extensive hinterland of assumptions. That makes testing the validity of their forecasts intrinsically difficult: were the assumptions wrong, or was it just that not enough assumptions were included?

Another problem is that the fresh, real-world data needed to test the results is often not even available. It will flood in quickly and easily for day-ahead weather forecasts, for example – but might arrive centuries too late to discriminate between today’s long-term climate models.

That’s why, Thompson explains, a second, qualitative way of determining the success of predictions is much more common: reliance on expert judgment. The pitfalls of this route were also well known to the medieval courts. Only those versed in the most cutting-edge mathematical knowledge were skilled enough to interpret medieval horoscopes. As such, it was in practice impossible for the client to come to their own conclusions. The result was that an exclusive guild, whose true competence remained unknown, ended up marking their own homework. The same could be said today.

Another hazard stalking ancient and modern modellers alike is that they fall in love with the sheer beauty and complexity of their own constructions. Having eaten the lotuses of Model Land, they can’t bring themselves to escape. Scenarios and predictions are simply accepted as if the model actually is real life.

“Such naive Model Land realism,” Thompson warns, “can have catastrophic effects because it invariably results in an underestimation of uncertainties and exposure to greater-than-expected risk.” Anyone who remembers Goldman Sachs’s chief financial officer blaming the global credit crunch of 2007 on the occurrence of “twenty-five standard deviation events, several days in a row” knows what Thompson is getting at. If it couldn’t happen in the model, it just wasn’t meant to happen in real life.

It’s not all bad news. Thompson is a data scientist and mathematical modeller herself, and her book is far from an exercise in model-bashing. It is instead a nuanced and constructive critique of what remains an invaluable analytical method – just not necessarily for the reasons you might expect.

For example, even though the astrologers’ models of natural forces and human behaviour were wrong, the practice of casting horoscopes could still be a useful aid to policymaking. They brought systematic thinkers into the orbit of otherwise impulsive rulers; it allowed the discussion of important, otherwise taboo subjects in the safe context of interpreting the stars; and it could give decision-makers the public narrative they needed in order to act.

The same applies today. As Thompson shows, mathematical model-building can still be a constructive tool, even if the models themselves are flawed. As Dwight D Eisenhower said: “Plans are useless, but planning is indispensable”.