What journalists can learn from scientists and the scientific method
by Matt ThompsonPublished Sep. 1, 2011 7:26 amUpdated Sep. 1, 2011 7:33 am
======================================================
I don’t need to tell you that the Scientific Revolution kickstarted the modern age. Kevin Kelly lists the discovery of the scientific method alongside the invention of libraries and the printing press as “meta-changes” — the evolution of evolution, changes in the ways things change.
The scientific method — the foundation of modern science — was a way to systematize knowledge production. Once discovered, it became a discovery-producing machine. Humankind had developed a way to both ask better questions about the nature of the world and to continually find better answers to those questions. Soon, the tools of this discovery-producing machine began to improve; we devised techniques such as double-blind studies and placebos. And over time, we arrived at where we are today, having landed on the moon, split the atom and cured smallpox.
But the scientific method isn’t a single thing. It’s a collection of conventions and best practices, rigorously applied. While the need for journalism and the demands on it differ in many ways from those of science, it is a discipline that — like science — seeks truth. There is no “journalistic method” to rival the scientific one. But in that body of conventions and practices, there’s much that journalists should emulate.
This is an ancient argument. Just last week, Robert Niles made a very similar case in the Online Journalism Review. And my understanding of these thoughts is built on a century of thinking from folks like Walter Lippman and the University of North Carolina’s Philip Meyer, who’ve been banging this drum for many, many years. But with the rise of the Web, there have never been more opportunities to put this thinking into practice, to create models of journalism truly informed by the science. Here are a few places we could start.
======================================================
I don’t need to tell you that the Scientific Revolution kickstarted the modern age. Kevin Kelly lists the discovery of the scientific method alongside the invention of libraries and the printing press as “meta-changes” — the evolution of evolution, changes in the ways things change.
The scientific method — the foundation of modern science — was a way to systematize knowledge production. Once discovered, it became a discovery-producing machine. Humankind had developed a way to both ask better questions about the nature of the world and to continually find better answers to those questions. Soon, the tools of this discovery-producing machine began to improve; we devised techniques such as double-blind studies and placebos. And over time, we arrived at where we are today, having landed on the moon, split the atom and cured smallpox.
But the scientific method isn’t a single thing. It’s a collection of conventions and best practices, rigorously applied. While the need for journalism and the demands on it differ in many ways from those of science, it is a discipline that — like science — seeks truth. There is no “journalistic method” to rival the scientific one. But in that body of conventions and practices, there’s much that journalists should emulate.
This is an ancient argument. Just last week, Robert Niles made a very similar case in the Online Journalism Review. And my understanding of these thoughts is built on a century of thinking from folks like Walter Lippman and the University of North Carolina’s Philip Meyer, who’ve been banging this drum for many, many years. But with the rise of the Web, there have never been more opportunities to put this thinking into practice, to create models of journalism truly informed by the science. Here are a few places we could start.
Making more accurate predictions
Part of the genius of the scientific method is that it tests how true a hypothesis is by tracking how well it predicts future outcomes. Your theory is only as good as its ability to predict the results of a trial.
We journalists like to think of ourselves as describing the present as faithfully as we can. So journalism contains no such predictive test. Yet so much of journalism involves collecting and assessing claims about the future. Take a typical story, plucked at random from a July 2009 front page of politics news aggregator Memeorandum, conveying the fears of a group of state governors that the plans to overhaul the nation’s health care systems would foist unpaid obligations on the states.
These claims are eminently trackable. In fact, PolitiFact is already doing some of this. The site’s ingenious Obameter and GOP Pledge-O-Meter systematically follow up on one common type of future-claim: a pledge that a politician will do something once elected. As the 2012 election approaches, these tools are becoming fantastic barometers for voters to assess how well incumbent candidates delivered on their promises.
Imagine if other types of future-claims — e.g. “this bill will spur hiring among small-business owners,” “if we don’t strengthen the levees, a hurricane could be catastrophic for our city” — were tracked and assessed in as vigorous and organized a fashion. Journalists would have more of an incentive to prompt their sources for testable claims. The public would have a framework for assessing sources’ credibility.
Aside from the claim-tracker approach to predictions, though, there’s a lot to be said for journalism that accurately anticipates news events. If we’ve done a thorough and accurate job of describing the present, our work should leave our users better prepared for the future.
One of my favorite works of journalism is James Fallows’ November 2002 cover story in The Atlantic Monthly, “The Fifty-First State?” In it, Fallows reports with amazing foresight the chain of events likely to occur if the U.S. were to invade Iraq. Almost nine years later, the story reads like a now-familiar catalogue of major issues the U.S. encountered after invading Iraq in March 2003, five months after the article was published. “The Fifty-First State?” won a National Magazine Award in the Public Interest category in 2003. It makes me wish there were a Pulitzer Prize for journalism from five or more years ago that presaged something important about the world we live in today.
Part of the genius of the scientific method is that it tests how true a hypothesis is by tracking how well it predicts future outcomes. Your theory is only as good as its ability to predict the results of a trial.
We journalists like to think of ourselves as describing the present as faithfully as we can. So journalism contains no such predictive test. Yet so much of journalism involves collecting and assessing claims about the future. Take a typical story, plucked at random from a July 2009 front page of politics news aggregator Memeorandum, conveying the fears of a group of state governors that the plans to overhaul the nation’s health care systems would foist unpaid obligations on the states.
These claims are eminently trackable. In fact, PolitiFact is already doing some of this. The site’s ingenious Obameter and GOP Pledge-O-Meter systematically follow up on one common type of future-claim: a pledge that a politician will do something once elected. As the 2012 election approaches, these tools are becoming fantastic barometers for voters to assess how well incumbent candidates delivered on their promises.
Imagine if other types of future-claims — e.g. “this bill will spur hiring among small-business owners,” “if we don’t strengthen the levees, a hurricane could be catastrophic for our city” — were tracked and assessed in as vigorous and organized a fashion. Journalists would have more of an incentive to prompt their sources for testable claims. The public would have a framework for assessing sources’ credibility.
Aside from the claim-tracker approach to predictions, though, there’s a lot to be said for journalism that accurately anticipates news events. If we’ve done a thorough and accurate job of describing the present, our work should leave our users better prepared for the future.
One of my favorite works of journalism is James Fallows’ November 2002 cover story in The Atlantic Monthly, “The Fifty-First State?” In it, Fallows reports with amazing foresight the chain of events likely to occur if the U.S. were to invade Iraq. Almost nine years later, the story reads like a now-familiar catalogue of major issues the U.S. encountered after invading Iraq in March 2003, five months after the article was published. “The Fifty-First State?” won a National Magazine Award in the Public Interest category in 2003. It makes me wish there were a Pulitzer Prize for journalism from five or more years ago that presaged something important about the world we live in today.
Borrowing the concept of “reproducibility”
We use the term “transparency” a lot in conversations about journalistic ethics. It’s one of the key values of our profession: journalists should give the public the best tools they can to evaluate their work, including as much information as possible about how the work was conducted, or any relevant disclosures that might affect its credibility.
Science has a neighboring concept worth borrowing: “reproducibility.” A scientist’s findings should be both consistent enough and described well enough that another independent scientist can repeat the research and achieve a very similar outcome.
For many years, UNC’s Meyer has encouraged us to do away with the common, oft-derided understanding of journalistic objectivity that prompts us to ask questions like, “Is the story objective?” “Traditionally,” Meyer once said, “[objectivity] has sort of meant you sprinkle your ink around to lots of different groups and hope that it comes out even or fair. I think we ought to emphasize objectivity of method. That’s really what scientific method is.”
I think “reproducibility” is another way to get at what Meyer’s describing. Are the conclusions embedded in a work of journalism reproducible? Has the reporting been thorough, fair and accurate enough that another reporter digging into the same issue would reach a similar outcome? If someone else had tried to replicate Fallows’ inquiry into the aftermath of an invasion of Iraq, would they describe a similar future?
I especially like the second element of the test of reproducibility. Does the story describe its methods and inputs well enough that someone else could repeat them? Since the rise of the Web, this has been a growing part of the ethic of investigative journalism. I now expect whenever I see a big data project from a news organization that I can inspect the data myself and test what the reporters have told me. ProPublica has probably taken this ethic the furthest that I’ve seen; the site has published “reporting recipes” that lay out the steps to reproduce its work at a local level.
I would love for this to become a part of all journalism, not just investigative and data journalism. If all news stories were reported with an eye toward reproducibility, we’d be even warier about using anonymous sources. We’d be better about describing not just what our sources said, but where and how they said it. This is a pet peeve of mine. There’s a material journalistic difference between “so-and-so said” and “so-and-so said to me over drinks yesterday evening” and “so-and-so said to a group of reporters in a conference call for the press.” Only the latter versions give some indication of how one might go about recreating the work.
The beauty of process
In my work on NPR’s Project Argo, I have the pleasure of working with several terrific science reporters, at least one of whom is a bona fide scientist herself. I feel as though I now have a better sense of some of the tensions between scientists and journalists.
Science places much more of an emphasis on processes. Journalism places much more of an emphasis on outcomes. To journalists, the process of scientific discovery is tortuously incremental. We want to push all of those double-blind, randomized, peer-reviewed, clinical bits out of the way and get straight to the delicious creme filling — “NEW RESEARCH COULD LEAD TO INVISIBILITY CLOAK!!” (I particularly love the phrase “could lead to.” You can tuck so much under that rug. Obligatory link to “the science news cycle.”Obligatory link to satirical news article about scientific papers.)
Journalists think in discrete stories. As in, “I’m finished with this story. Onto the next.” We often aim to produce these polished gems of Aristotelian narrative, bearing arresting ledes, explosive kickers and genuine catharsis somewhere in the midst. In science, the continued journey toward greater knowledge is an unending quest. Scientists spend entire careers advancing the state of knowledge in their field, not whizzing from discovery to discovery, but gradually pursuing an ever-greater understanding.
The ability to turn the process of reporting into a compelling, unending story of its own is becoming an increasingly vital journalistic skill. I encourage the journalists I work with to think in streams, not just discrete stories. Your feed is more important than any single post, and a subscriber is more valuable than a visitor. If you can hook people into your ongoing quest for greater understanding of a beat or topic, you wield a much more powerful asset than the crowd of random people who happen to find one of your stories.
Paul Ford recently wrote a beautiful, much-forwarded essay for New York Magazine about the rising importance of the stream. Here’s a taste:
Social media has no understanding of anything aside from the connections between individuals and the ceaseless flow of time: No beginnings, and no endings. These disparate threads of human existence alternately fascinate and horrify that part of the media world that grew up on topic sentences and strong conclusions. This world of old media is like a giant steampunk machine that organizes time into stories. I call it the Epiphanator, and it has always known the value of a meaningful conclusion. The Epiphanator sits in midtown Manhattan and clunks along, at Condé Nast and at the Times and in Rockefeller Center. Once a day it makes a terrible grinding noise and spits out newspapers and TV shows. Once a week it spits out weeklies and more TV shows. Once a month it produces glossy magazines. All too often it makes movies, and novels.
At the end of every magazine article, before the “■,” is the quote from the general in Afghanistan that ties everything together. The evening news segment concludes by showing the Secretary of State getting back onto her helicopter. There’s the kiss, the kicker, the snappy comeback, the defused bomb. The Epiphanator transmits them all. It promises that things are orderly. It insists that life makes sense, that there is an underlying logic.
These two concepts — the Epiphanator and the stream — can and should live alongside each other. My buddy, co-blogger and fellow Poynter alum Robin Sloan, calls the interplay between the two “stock and flow.”
What we call “media literacy” depends on the public taking an interest in the particulars of the journalistic process, not just the stories it produces. A fact that a reporter has taken great pains to prosecute and verify should hold more weight than a soundbite delivered with plenty of pizazz and zero evidence to support it. But to make that distinction, we need to present the process. And we need to make it as arresting as the outcome.
I’ll appeal to that immortal James Fallows article one last time. To quote myself:
Among the article’s most notable characteristics is Fallows’ willingness to show his work — the story begins with a remarkable catalogue of Fallows’ process and assumptions. Almost everybody quoted is on the record (side note: just think of the thousands of inches of anonymously sourced stories that totally got it wrong right around this time), and we see Fallows’ perspective shift as the piece progresses. By putting all that in there, Fallows makes the story accessible, engaging, and deeply informative, not overly reflective or self-indulgent.
Journalism isn’t science, and science isn’t perfect. In fact, there are many ways that the field of science is falling behind journalism in adapting to changes in our society. But I’ve only started to scratch the surface of how journalism can build on the practices science has evolved. I’m really interested to hear how journalism might benefit from concepts analogous to peer review and theory construction.
So, keeping with Lippman’s exhortation that we approach journalism in “the scientific spirit,” let’s make this an ongoing conversation, not an end point. I hope to continue this discussion in a SXSW session I’ve pitched with my friend Gideon Lichfield, a journalist with the Economist who has two degrees in the philosophy of science. And I invite you to share your thoughts with me in the comments section of this story.
We use the term “transparency” a lot in conversations about journalistic ethics. It’s one of the key values of our profession: journalists should give the public the best tools they can to evaluate their work, including as much information as possible about how the work was conducted, or any relevant disclosures that might affect its credibility.
Science has a neighboring concept worth borrowing: “reproducibility.” A scientist’s findings should be both consistent enough and described well enough that another independent scientist can repeat the research and achieve a very similar outcome.
For many years, UNC’s Meyer has encouraged us to do away with the common, oft-derided understanding of journalistic objectivity that prompts us to ask questions like, “Is the story objective?” “Traditionally,” Meyer once said, “[objectivity] has sort of meant you sprinkle your ink around to lots of different groups and hope that it comes out even or fair. I think we ought to emphasize objectivity of method. That’s really what scientific method is.”
I think “reproducibility” is another way to get at what Meyer’s describing. Are the conclusions embedded in a work of journalism reproducible? Has the reporting been thorough, fair and accurate enough that another reporter digging into the same issue would reach a similar outcome? If someone else had tried to replicate Fallows’ inquiry into the aftermath of an invasion of Iraq, would they describe a similar future?
I especially like the second element of the test of reproducibility. Does the story describe its methods and inputs well enough that someone else could repeat them? Since the rise of the Web, this has been a growing part of the ethic of investigative journalism. I now expect whenever I see a big data project from a news organization that I can inspect the data myself and test what the reporters have told me. ProPublica has probably taken this ethic the furthest that I’ve seen; the site has published “reporting recipes” that lay out the steps to reproduce its work at a local level.
I would love for this to become a part of all journalism, not just investigative and data journalism. If all news stories were reported with an eye toward reproducibility, we’d be even warier about using anonymous sources. We’d be better about describing not just what our sources said, but where and how they said it. This is a pet peeve of mine. There’s a material journalistic difference between “so-and-so said” and “so-and-so said to me over drinks yesterday evening” and “so-and-so said to a group of reporters in a conference call for the press.” Only the latter versions give some indication of how one might go about recreating the work.
The beauty of process
In my work on NPR’s Project Argo, I have the pleasure of working with several terrific science reporters, at least one of whom is a bona fide scientist herself. I feel as though I now have a better sense of some of the tensions between scientists and journalists.
Science places much more of an emphasis on processes. Journalism places much more of an emphasis on outcomes. To journalists, the process of scientific discovery is tortuously incremental. We want to push all of those double-blind, randomized, peer-reviewed, clinical bits out of the way and get straight to the delicious creme filling — “NEW RESEARCH COULD LEAD TO INVISIBILITY CLOAK!!” (I particularly love the phrase “could lead to.” You can tuck so much under that rug. Obligatory link to “the science news cycle.”Obligatory link to satirical news article about scientific papers.)
Journalists think in discrete stories. As in, “I’m finished with this story. Onto the next.” We often aim to produce these polished gems of Aristotelian narrative, bearing arresting ledes, explosive kickers and genuine catharsis somewhere in the midst. In science, the continued journey toward greater knowledge is an unending quest. Scientists spend entire careers advancing the state of knowledge in their field, not whizzing from discovery to discovery, but gradually pursuing an ever-greater understanding.
The ability to turn the process of reporting into a compelling, unending story of its own is becoming an increasingly vital journalistic skill. I encourage the journalists I work with to think in streams, not just discrete stories. Your feed is more important than any single post, and a subscriber is more valuable than a visitor. If you can hook people into your ongoing quest for greater understanding of a beat or topic, you wield a much more powerful asset than the crowd of random people who happen to find one of your stories.
Paul Ford recently wrote a beautiful, much-forwarded essay for New York Magazine about the rising importance of the stream. Here’s a taste:
Social media has no understanding of anything aside from the connections between individuals and the ceaseless flow of time: No beginnings, and no endings. These disparate threads of human existence alternately fascinate and horrify that part of the media world that grew up on topic sentences and strong conclusions. This world of old media is like a giant steampunk machine that organizes time into stories. I call it the Epiphanator, and it has always known the value of a meaningful conclusion. The Epiphanator sits in midtown Manhattan and clunks along, at Condé Nast and at the Times and in Rockefeller Center. Once a day it makes a terrible grinding noise and spits out newspapers and TV shows. Once a week it spits out weeklies and more TV shows. Once a month it produces glossy magazines. All too often it makes movies, and novels.
At the end of every magazine article, before the “■,” is the quote from the general in Afghanistan that ties everything together. The evening news segment concludes by showing the Secretary of State getting back onto her helicopter. There’s the kiss, the kicker, the snappy comeback, the defused bomb. The Epiphanator transmits them all. It promises that things are orderly. It insists that life makes sense, that there is an underlying logic.
These two concepts — the Epiphanator and the stream — can and should live alongside each other. My buddy, co-blogger and fellow Poynter alum Robin Sloan, calls the interplay between the two “stock and flow.”
What we call “media literacy” depends on the public taking an interest in the particulars of the journalistic process, not just the stories it produces. A fact that a reporter has taken great pains to prosecute and verify should hold more weight than a soundbite delivered with plenty of pizazz and zero evidence to support it. But to make that distinction, we need to present the process. And we need to make it as arresting as the outcome.
I’ll appeal to that immortal James Fallows article one last time. To quote myself:
Among the article’s most notable characteristics is Fallows’ willingness to show his work — the story begins with a remarkable catalogue of Fallows’ process and assumptions. Almost everybody quoted is on the record (side note: just think of the thousands of inches of anonymously sourced stories that totally got it wrong right around this time), and we see Fallows’ perspective shift as the piece progresses. By putting all that in there, Fallows makes the story accessible, engaging, and deeply informative, not overly reflective or self-indulgent.
Journalism isn’t science, and science isn’t perfect. In fact, there are many ways that the field of science is falling behind journalism in adapting to changes in our society. But I’ve only started to scratch the surface of how journalism can build on the practices science has evolved. I’m really interested to hear how journalism might benefit from concepts analogous to peer review and theory construction.
So, keeping with Lippman’s exhortation that we approach journalism in “the scientific spirit,” let’s make this an ongoing conversation, not an end point. I hope to continue this discussion in a SXSW session I’ve pitched with my friend Gideon Lichfield, a journalist with the Economist who has two degrees in the philosophy of science. And I invite you to share your thoughts with me in the comments section of this story.
=================================================
poynter.org/latest-news/top-stories/144581/what-journalists-can-learn-from
No comments:
Post a Comment