KOPEL: Too often a crutch
Studies important enough to mention in a story should be cited
By Dave Kopel, Rocky Mountain News (Contact)
Saturday, March 8, 2008
- Email this
- Print this
- Comments
- Change text size
- Subscribe to print edition
- iPod friendly
Studies have shown you shouldn't blindly accept everything journalists tell you about what studies have shown. Consider, for example, "Growth spurt for kids' health plan," the Feb. 10 Denver Post article by Katy Human, writing about the push to expand taxpayer subsidies for health care for middle-class and poor children.
Human's article announced: "Children with health insurance, studies have shown, are less likely than uninsured kids to end up in emergency rooms, more likely to get key vaccinations, and less likely to be absent from school."
My friend Ari Armstrong is a columnist for the Grand Junction Free Press, and also the publisher of the Colorado Freedom Report weblog. Armstrong used to be a senior fellow at the Independence Institute and, although he's no longer formally affiliated with the institute, he still writes for us from time to time. Armstrong e-mailed Human and politely asked if she could send him the names of two or three of the studies she had in mind.
She refused. E-mail exchanges continued, as Armstrong copied Jason Salzman, my counterpart on these pages, and me. Eventually, Human did send Armstrong a list of five studies, along with a note: "I won't be doing this for you again . . . you can do it yourself, and I don't have time to repeat these types of searches for everyone who asks."
One wonders why Human had to spend time searching for the citations, since presumably she knew about each of the studies before she wrote the article. The official standards of the Post so require.
None of five studies Human cited after the fact support her article's statement about what "studies have shown" regarding the effects of insurance on emergency room use, vaccinations and school absences. Indeed four of the five studies she cited do not even address those topics (Cousineau, Medical Care, 2008; Skinner, BMC Health Services Research, 2007; Ward, CA: A Cancer Journal for Clinicians, 2008; Morbidity & Mortality Weekly Report, Sept. 7, 2007).
One study cited by Human was relevant, and it directly contradicted her article's claim. The study looked at the effect of providing SCHIP coverage (subsidized insurance for children whose families have too much income to qualify for Medicaid). Emergency room usage "did not change," the study found. (Szilagyi, Pediatrics, 2004).
I e-mailed Human some questions, and got a response from the Post's public affairs editor, Chuck Murphy. Via Murphy, Human supplied two more citations to substantiate her article's claim about emergency rooms. One of the studies was irrelevant, a 2002 Centers for Disease Control and Prevention report which reported no data about frequency of emergency room use for the insured and uninsured.
Human supplied another study which did support her claim. William Johnson and Mary Rimsza investigated Yuma County, Ariz., and found that uninsured children there use emergency rooms more often. The Johnson and Rimsza article, published in Pediatrics in 2004, forthrightly acknowledged that four other studies have found that taxpayer-funded insurance for children actually increases emergency room usage, and a fifth study finds that there is no effect. Johnson and Rimsza suggested that results were different in Arizona because the state's medical welfare program links recipients to pediatricians, and having a pediatrician drastically reduces ER visits for both the insured and uninsured.
So Human's pronouncement in her Post article - "Children with health insurance, studies have shown, are less likely than uninsured kids to end up in emergency rooms" - turns out to be not entirely accurate. A large body of research contradicts her claim, and that research is in the very studies which Human pointed to when she was challenged to support her claims.
In the last two years, the phrase "studies have shown" has appeared in staff-written pieces 31 times in the Rocky Mountain News, and 36 times in the Post. About half the time the phrase is used in a direct quote, or in another way which tells the reader the source of the information. For example, "According to professor Roy Hinkley, studies have shown that minnows . . . "
But the other half of the time, the dailies used "studies have shown" with no source. The unattributed locution was especially common in Post editorials, and in health and nutrition coverage in both papers.
The phrase ill-serves readers who want to learn more about a subject, but who are left in the dark about where to look. The phrase can be used to falsely declare scholarly consensus about a subject. And the phrase can be a crutch for a writer who feels "sure" about a supposed fact, but who doesn't want to take the time to verify it.
If "studies" are important enough to mention in an article, they should usually should be named in the article. Print space is scarce, but a quick citation can be offered in three to five words. For online versions, full citations can be supplied at the end of the article.
Dave Kopel is research director at the Independence Institute, an attorney and author of 10 books. He can be reached at kopeld@RockyMountainNews.com.
Comments
Posted by Gene on March 8, 2008 at 9:10 a.m. (Suggest removal)
Thanks again Dave for doing the spadework. Half the stuff one reads brings back Saturday Night Live's Roseanna-Danna, . . . "oh never mind."
Posted by gs on March 8, 2008 at 2:31 p.m. (Suggest removal)
I think the misreporting requires some disclpinary action on the part of the Denver Post.
Posted by paulhsiehmd on March 8, 2008 at 2:56 p.m. (Suggest removal)
Thank you, David, for a very illuminating piece.
If I didn't know better, I might almost wonder if a reporter had a particular ideological agenda and tried to slant a news story to support a political view favoring more government control of medicine, rather than trying to write the news in an objective fashion based on the actual facts.
But that would imply that there was some sort of liberal media bias, and we all know *that* couldn't be the case...
Paul Hsieh, MD
Sedalia, CO
Freedom and Individual Rights in Medicine: www.WeStandFIRM.org
Posted by anderson on March 8, 2008 at 3:02 p.m. (Suggest removal)
Good column. However, one way to get around the "studies have shown" dilemma is to simply make an inaccurate claim of fact without citing any evidence. Legislators get away with this stuff all the time.
Posted by anderson on March 8, 2008 at 4:23 p.m. (Suggest removal)
It's funny that someone would call for objectivity and claim a "liberal media bias" in the same breath. To demonstrate this alleged bias it would be helpful to have some objective standards, such as a definition for "liberal" (for many the term seems to mean nothing other than "something I oppose") and deliniations of "media". And it would be helpful to have some objective measures (how do we identify bias?). It isn't going to happen. In fact, this sort of claim is ripe for a "studies have shown" sort of analysis--where sources are cited at all.
Posted by AngelontheSidelines on March 8, 2008 at 9:16 p.m. (Suggest removal)
Gene, a nit to pick...
Roseann Rosanadanna used the line,:It's always something"
It was Emily Latella and her deaf take on news that triggered,"Oh never mind"
But I know what you mean:)
Posted by kathyM on March 9, 2008 at 3:01 p.m. (Suggest removal)
Researchers who can't support their claims with specific data don't win research grants. Lawyers who can't support their claims with specific data don't win many cases. Executives who can't support their claims with specific data harm their companies' performance. High school and college kids who can't support their claims with specific data earn a big F on their research papers. Reporters who can't support their claims with specific data should be disciplined or fired--and so should their editors for allowing such sloppiness.
Posted by freethinker07 on March 10, 2008 at 1:32 a.m. (Suggest removal)
Dave, you are being too soft on those reporters. Once a study reports a conclusion, the proper approach is to examine the study to see how well it was done. Far too often, newspapers will report a study with controversial and far reaching conclusions done on 6 subjects. A study with 6 subjects is a mockery of the scientific method.
Posted by Gene on March 10, 2008 at 6 a.m. (Suggest removal)
Sorry about the Saturday Night Live analogy. I am not an expert on the subject. Heck, I should probably be writing columns for R. M. N.
Posted by spencerr on March 10, 2008 at 8:52 a.m. (Suggest removal)
Here's some good research for Anderson and those who are like-minded. A neutral study, ironically from academia, that shows findings of both liberal and (a little bit of) conservative bias in the media as well as a list of the most neutral and biased sources.
Anderson and a lot of lefties keeps talking about how it's a myth, but the article does a good job of defining media bias and then convicting the media, in general, of it.
I guess if you happen to be one of the people who votes in the same direction that the media reports, you would probably not be able to see it. I, myself, thought Fox News was very biased to the Right...but it is actually one of the more neutral media sources (it is Right leaning, but it is more neutral than most sources, which lean Left).
http://newsroom.ucla.edu/portal/ucla/...
Posted by anderson on March 10, 2008 at 11:40 a.m. (Suggest removal)
spencerr, is you're calling me a lefty supposed to be an argument for something?
I never said media bias was a myth. But it's clear those who talk about liberal media bias usually have nothing to support their claim save that they heard it from "the media" (liberally-biased of course).
I see several problems with the report of the study you cite (which I believe Kopel and Salzman discussed at length in their columns--these are my criticisms not theirs):
As far as I can tell, the ADA subjectively defines liberal and conservative votes (no real definition of liberal or conservative except, I guess, they know it when they see it), then, from this, chooses a hypothetical middle based on the votes of Congress.
What if Congress swings one way or the other, say, on a vote to go to war with Iraq? Don't we then have a shifting middle?
We still have no objective definition of liberal and conservative--which means all these sorts of assessments have a subjective basis.
The report seems to self-select media--what it considers mainstream. Even though they mention NPR, I see no mention of talk radio, which is listened to by 30 million people or so. Maybe they are calling it "entertainment" (as some do) and not counting it as "media".
I see no mention of internet sources. I see a discussion of select media, and whether they lean one way or another, but I see no support for the one guy's statement that "media" (in toto) leans left.
As for the claim that reporters personal voting preferences translate into a bias of a particular sort in the news, I challenge you to think of ways in which this might not be accurate.
Posted by spencerr on March 10, 2008 at 12:32 p.m. (Suggest removal)
Lefty is not meant to be inflamatory. You should be proud of being on the left, as I am of being on the right. It is simply how I define you...it is not an argument.
This whole topic is subjective based on where the person writing or reading stands. That being said, how is it possible to find a completely objective set of standards for measuring it? They did as good as is possible...they got students from both sides of the isle to do the study. Also, I don't see any major mainstream telivision or newspaper sources missing. I also don't see any minor ones, with the exception of media that is admittedly Right, missing.
Anyway, deciding what is left, right, and middle is relatively straight forward...markets vs. government, abortion vs. life, rights to gay marriage vs. not. And based on the way the politicians from either side of the isle might vote, this would be fairly simple to confirm. Here it is, simplified for sake of argument.
As the study author, maybe I want to claim that voting for legislation limiting the right to have an abortion to those who are four months or less pregnant is a conservative issue. Well, then I could see how Republicans and Democrats voted. Assuming that they voted according to the assumption, it has been confirmed.
And as for radio...it is definitely Right wing, but it is not considered mainstream media. Most consider it fringe conservative media, and I believe that it exists because many Americans do not trust the Liberal mainstream media. Thirty million listeners is nothing to flinch at, but neither does this type of media claim to be neutral, nor does it claim to be mainstream. Rush and Rosen know that ninety percent of their listeners are Republicans of some sort.
Anyway, I like to argue...especially with you. You're like jay, except that you actually make valid points and back your stuff up sometimes. And like me, you make concessions when someone has made a valid point running contrast to your own...unlike Leftside and twosense.
Your two main arguing points regarding Righty authors on this site, however, drive me nuts...you claim that the mainstream media is not liberal, and you are always yelling about straw men.
I didn't see you chiming in in support of Rosen when he deconstructed Campos's entire argument full of straw men arguments.
Posted by anderson on March 10, 2008 at 2:13 p.m. (Suggest removal)
Preceding my name with "lefty" was clearly meant to signal to anyone reading this that my statements are not to be trusted because I'm "left" instead of "right" (and there are presumably only two kinds of people in the world). You may have intended it lightly but it's a form of a smear, used on talk radio frequently.
Thank for admitting the whole premise of liberal v. conservative is subjective. As for your question: what objective standards could we use, I'd say there probably aren't any--which begs the question: why are we trying so hard to define the media as left or right? (us v. them) How about accuracy and fairness and an informed public? (us and us) Do those matter?
Defining left and right votes may seem simple on its face, but I don't think they are. How do you reconcile Robert Taft's staunch isolationism with George Bush's push for war? Both men clearly identified themselves as conservatives. Most questions before Congress are not 100-0 or 464-0 votes. And there are a great number of issues voted on. ADA whittled this down to 25?
That's laughable that you would say talk radio and its 30 million listeners is not mainstream. What is mainstream then? Whatever is not talk radio? I suppose you mean "mainstream" to be something like: generally acceptable or not explicitly partisan, but then liberalbiasinmedia advocates are claiming that mainstream is partisan, no? You can't have it both ways. I like how you mention Rosen and Rush in the same breath. So is Rosen mainstream when he writes for the RMN but not mainstream when he does his talk radio show?
Moreover, I see the hateful ideas presented on talk radio launched on blogs such as this all the time--further "mainstreaming" certain ideas like petty resentments and fear of those who are diffrent than us.
I didn't even mention (or does the study) explicitly right-wing sources such as newsmax and worldnetdaily that I've often seen cited on general(neutral) blogs such as this one.
I usually don't read Campos' columns (he lost me after he claimed obesity is not a health problem in America), so whatever Rosen said about him, I don't know. Why do you bring it up if I didn't comment on it?
Posted by anderson on March 10, 2008 at 3:46 p.m. (Suggest removal)
See the Eliot Spitzer story here on the RMN and how so many commentators glory in his fall. The most important part of the story for them is not that the governor of a major state was involved in some wrongdoings, or his resignation, or that his wife stood by his side, or that public policy regarding prostitution is implicated. No. THE most important fact for most commentators was the fallen governor is a Democrat. THE most important issue for most commentators is whether "the media" is fair to Democrats or Republicans now or in the past. Nothing more than the attitude: let's choose sides and get in a pissing contest. Does that say anything about media in America? About what is mainstream?
Posted by John_II on March 30, 2008 at 5:24 p.m. (Suggest removal)
This was my first time reading a Dave Kopel column. Excellent work, Dave!
The word 'study' certainly has acquired a much more powerful connotation than was intended. To hear the way reporters, and many liberal bloggers, cling to the word 'study', you'd think Descartes himself had performed the work.
Here's a study for you. A recent study has found that drinking liquid from glass bottles causes intoxication.
Ex nihilo nihil fit.
Post your comment (Requires free registration.)
Comments are the sole responsibility of the person posting them. You agree not to post comments that are off topic, defamatory, obscene, abusive, threatening or an invasion of privacy. Violators may be banned. Click here for our full user agreement.