Support Aeon

‘I learn more about how to think about the world from Aeon than probably any other publication I regularly read.’

Elizabeth B, USA, Friend of Aeon

Aeon is a registered charity committed to the spread of knowledge and a cosmopolitan worldview.
But we can’t do it without you.

Support Aeon

Aeon is a registered charity committed to the spread of knowledge and a cosmopolitan worldview. Our mission is to create a sanctuary online for serious thinking.

No ads, no paywall, no clickbait – just thought-provoking ideas from the world’s leading thinkers, free to all. But we can’t do it without you.

Support Aeon

How statistics are twisted to obscure public understanding

Jonathan R Goodman

Jonathan R Goodman

is a postgraduate researcher at the department of biology at Queens College, CUNY, where he studies cultural evolution. He is also the editor of Cancer Therapy Advisor.

Brought to you by curio.io, an Aeon partner

800 words

Edited by Pam Weintraub

Republish
<em>Alexander Baxevanis/Flickr</em>
Alexander Baxevanis/Flickr

Jonathan R Goodman

Jonathan R Goodman

is a postgraduate researcher at the department of biology at Queens College, CUNY, where he studies cultural evolution. He is also the editor of Cancer Therapy Advisor.

Brought to you by curio.io, an Aeon partner

800 words

Edited by Pam Weintraub

Republish
<em>Alexander Baxevanis/Flickr</em>
Alexander Baxevanis/Flickr

Jonathan R Goodman

is a postgraduate researcher at the department of biology at Queens College, CUNY, where he studies cultural evolution. He is also the editor of Cancer Therapy Advisor.

Brought to you by curio.io, an Aeon partner

800 words

Edited by Pam Weintraub

Republish

Mark Twain attributed to Benjamin Disraeli the famous remark: ‘There are three kinds of lies: lies, damned lies, and statistics.’ In every industry, from education to healthcare to travel, the generation of quantitative data is considered important to maintain quality through competition. Yet statistics rarely show what they seem.

If you look at recent airline statistics, you’ll think that a far higher number of planes are arriving on schedule or early than ever before. But this appearance of improvement is deceptive. Airlines have become experts at appearance management: by listing flight times as 20-30 per cent longer than what the actual flight takes, flights that operate on a normal to slightly delayed schedule are still counted as arriving ‘early’ or ‘on time’. A study funded by the Federal Aviation Administration refers to the airline tactic as schedule buffering.

It is open to question, however, whether flights operating on a buffered schedule arrive ‘on time’ in the sense that ordinary people use the term. If a flight is scheduled for 2.5 hours and takes, on average, only 1.5 hours to reach its destination, then is any flight that arrives at its scheduled time really on time? Or have the airlines merely redefined the term ‘on time’ to generate more favourable statistics?

This example of airlines twisting meaning – and, consequently, public perception – might be irritating, but it is by no means the only industry where semantic manipulation of statistics is found. University rankings are especially prominent: numerous publications rank universities on a variety of metrics, relying on factors such as a university’s acceptance rate, average student test scores and job placement, to name a few.

But in recent years the competition among universities has become so intense that several have admitted to dishonestly manipulating the stats. In August 2012, Emory University admitted, after an internal investigation, that the administration had misreported incoming students’ test scores for a decade. And Emory was not the only culprit: in 2013, Forbes magazine removed three other major colleges from its rankings for similar infractions. The quantitative data weighed by rankings publications determine how we, ordinary people, understand phrases such as ‘the best universities’. But how can a system that rewards semantical manipulation be said to explain where students receive the best education?

Similar problems plague the healthcare system in the United States. An important concept for ranking hospitals is ‘survivability’, which the US News & World Report defines as ‘30 days after hospitalisation, adjusted for severity’. Avery Comarow, health rankings editor at US News, said in an email that ‘30-day mortality is the most common benchmarking period used by researchers, insurers and hospitals themselves for evaluating in-hospital mortality, because it recognises that a hospital is responsible for patients not just during their hospital stay but for a reasonable period after they are discharged’.

But what if a group of patients lives only for 32 days after hospitalisation? Ordinary people do not think of ‘survival’ as 30 days after any event; why should we trust a ranking system that uses such a fundamental and important notion in an unrecognisable way? Furthermore, does this definition favour hospitals that choose not to admit patients believed unlikely to live until the 30-day mark? What implicit pressures are placed on hospitals when society relies on the statistical analysis of a ranking publication as a guide to quality?  

Writing in the journal Statistical Science, the sociologist Joel Best argues that we ought to avoid calling statistics ‘lies’, and instead educate ourselves so that we can question how and why statistical data are generated. Statistics are often used to support points that aren’t true, but we tend to attack only the data that conflict with some preexisting notion of our own. The numbers themselves – unless purposefully falsified – cannot lie, but they can be used to misrepresent the public statements and ranking systems we take seriously. Statistical data do not allow for lies so much as semantic manipulation: numbers drive the misuse of words. When you are told a fact, you must question how the terms within the fact are defined, and how the data have been generated. When you read a statistic, of any kind, be sure to ask how – and more importantly, why – the statistic was generated, whom it benefits, and whether it can be trusted.

Republish

Jonathan R Goodman

is a postgraduate researcher at the department of biology at Queens College, CUNY, where he studies cultural evolution. He is also the editor of Cancer Therapy Advisor.

aeon.co
Get Aeon straight
to your inbox
Join our newsletter Sign up
Follow us on
Facebook
Like

‘Aeon is consistently the place to find excellent, provocative and thoughtful writing.

One of my favourite places to find new writers and new ideas.’

Professor Sophie Kerttu Scott, University College London

‘Aeon is my favourite online magazine.

I have found no other place that so regularly provides interesting, clear and informative articles across academic disciplines on topics that matter.’

Dr Robert J. Hartman, University of Gothenburg

‘Aeon is always providing great, thought-provoking content, and it would be a crime for me to not donate to a service that I use daily.’

Jarry A, Canada, Friend of Aeon

‘Aeon is one of the most consistently great publications on the internet.

My life would be poorer without it.’

Assistant Professor Lee Vinsel, Virginia Tech

‘Aeon is one of the liveliest, most wide-ranging and imaginative sources of good content on the web.

A fabulous resource for the intelligent and curious.’

Professor Barry C. Smith, University of London

Aeon is a registered charity committed to the spread of knowledge. Our mission is to create a sanctuary online for serious thinking.
But we can’t do it without you.

Aeon is a registered charity committed to the spread of knowledge and a cosmopolitan worldview.
But we can’t do it without you.

Support Aeon

Essay/
Mental Health
When the self slips

Individuals living with depersonalisation disorder bring vivid insight to the question of whether the self is an illusion

Anna Ciaunica & Jane Charlton

Essay/
Love & Friendship
Buddhists in love

Lovers crave intensity, Buddhists say craving causes suffering. Is it possible to be deeply in love yet truly detached?

Lisa Feldman Barrett & John Dunne