Build Your Own Baloney Detector

A tool-kit for avoiding being fooled

Friday, December 14, 2012

Are These the Right Statistics?

Not all statistics are created equal. And not all statistics are right for analyzing a given problem. I’m reminding of that today, in the wake of the latest mass shooting. While I have not intention of weighing in on the gun control debate (seriously, I don’t know the right statistics to say much, so I’d be dishonest to try), I do want to warn everyone about what statistics we pay attention to in the coming days and weeks. In particular, I’ve already seen a lot of statistics about violent crime in this country and how it’s down. Which is true, of course. But it’s not necessarily the right set of statistics for mass-shooting prevention. Those statistics include a majority of things that are not mass shootings: robberies, fights, domestic violence, and so forth. And while I can’t say that mass shootings have different root causes and enabling factors than these crimes, I suspect that they do. (If I’m wrong, I’d like to see an analysis that shows that we can lump them all together, anyway.) That being the case, looking at the bulk statistics where the mass-shootings are a small minority probably won’t give us a lot insight into these shootings.

So what should you do? Ask yourself whether you think that the data presented are what you really need to reach the conclusions suggested and, if you can, complain if not.

posted by John Weiss at 19:33  

Thursday, August 9, 2012

The Illusion of Balance 2: Are Both Sides Equally Valid?

The Illusion of Balance problem has another facet: The notion that there are two sides to every issue and that both sides are equally valid. This is usually manifest in the news by interviewing people of “both” sides of a given issue and giving them equal time and weight. Doing so implies that both sides are just as valid as the other. This is not necessarily the case.

This illusion of balance stems from the fact that most of our political and social debates do have two sides, if only by fiat. (It often seems like there might be more than two sides, doesn’t it? But we have a two-party system and two sides generally seems to make for a tidier debate in any case.) Fair enough. If we’re discussing, say, immigration reform, getting a Democrat and a Republican to discuss their parties’ views on the issue is just good reporting.

The problem is that sometimes there really aren’t two sides (or, rather, the two sides aren’t equally valid). If, for example, an astronomer was being interviewed and the shape of the Earth came up, you’d be appalled if the media interviewed a flat-earther for a contrary view. It’d be daft. The shape of the Earth isn’t really in doubt and even giving equal time (and thus,implicitly, equal weight) to the counter-argument is a best bad practice and worst dishonest.

So that was an extreme example, but this comes up all the time in less extreme cases. Consider the perpetual “debate” the country seems to have over teaching evolution. There’s little scientific debate about whether species evolve in time; the evidence for that is much too compelling for anything short of extraordinary counter-evidence to reopen that issue. And yet the news media hardly ever seems to interview a biologist without getting someone from the Discovery Institute or some other group that’s not interested in the science. The same is true with Anthropogenic Global Warming, the connection between tobacco and lung cancer, etc.

What do all of these have in common? They’re questions of science. While a lot of scientific issues are unsettled and honesty requires talking to both sides. (Take the issue of how much water might have been on Mars at some point in the past. It’s not a settled issue and a good story on this would interview several people with different views.) But listen to the scientists involved: if the actual researchers are overwhelmingly saying the same thing, then odds are any “debate” you see is manufactured. This is doubly true if some other group has something to gain by there being a debate at all. (In the case of Evolution, the Creationists have made it clear that they want Christianity taught in public schools. In the case of Global Warming… well, you know who has an interest in not admitting that the science there is pretty much decided.)

posted by John Weiss at 16:46  

Sunday, July 22, 2012

The Illusion of Balance (Part 1): Did they Check these Facts?

One of the ideals of reporting has long been “balance”, reporting the facts and giving the audience or reader a chance to decide the truth for themselves. This is a laudable ideal, of course, except that sometimes the pursuit of the appearance of balance (and fairness) gets in the way of actual careful reporting of the facts. I’ve seen this happen in two ways, so I’m covering it in two posts. Today, I’ll look at the problem with treating everyone’s facts equally. In my next post, we’ll look at the problem with giving equal time to all sides.

News media (and I’m not saying “reporters” because this also involves their editors and everyone else up and down the line) can choose to report facts in a few ways. A book I was reading recently (The Inquisition of Climate Science by James Lawrence Powell) suggested that this is best viewed as a continuum. I’m going to argue that there are at least two dimensions. The first is neutrality. The story can take a strong stance or it can be entirely neutral. On the former extreme, you have editorials, which aren’t usually considered news pieces per se. (Although in some cases, they seem to get slipped into the news without comment or label.) We tend to think that news stories should be on the neutral extreme, and with good reason. We don’t want them too neutral, however. We want the reporters to make logical inferences from the facts they collect and to some degree, taking a stand on some issues is perfectly warranted. (“Terrorist attacks are bad,” is probably not a viewpoint that almost anyone would object to, for example.) This is the continuum that most people think about when they think about news balance. It’s not the one I want to mostly talk about, though.

The dimension I’m interested in is the trust dimension. Reporters interview experts, witnesses, and parties concerned with stories on a daily basis. They also read countless press releases and studies. They do these things in order to report the facts or views that they collect in their stories. But are their sources accurate? There’s a fact-checking continuum, too. On the one extreme, you can fact-check nothing at all and just report statements (perhaps with attribution to save yourself from lawsuits). On the other, you can fact-check every single thing. The latter is clearly not economically feasible, nor is it reasonable. When the family of an accident victim reports their reaction, there’s no need to verify the relationship (a little trust is in order, in general) or to check on their credibility. In fact, doing so would probably cause unnecessary added insult and injury to already miserable people. So we can’t fact-check everything.

Sadly, it often seems that the news media has gone the other way lately and fact-checks nothing at all, simply reporting whatever either political party claims is true. The problems with this extreme are obvious when we think about them, so why does it happen? In part, because of time and money. Media outlets are on limited budgets and are being squeezed all the time. But there’s also an illusion of balance at play: to fact-check a politician might seem like taking sides against them. If the media appears to overwhelmingly check on one side or the other, that is seen as bias. So I wonder if the media also avoids checking on politicians’ facts for fear of seeming hostile to one side or the other. (This is also true of areas other than politics. When reporting a medical breakthrough, a good reporter will contact other experts in the field and get their views of the discovery, for example.)

Of course, fact-checking isn’t hostility, it’s the media’s job. I take it as a warning sign when I read a fact attributed to someone, especially a politician or pundit, without having been checked.

posted by John Weiss at 10:45  

Thursday, June 28, 2012

Headlines that Disagree with their Stories

I recently read an article about workers using medical leave as a way of avoiding performance improvement guides. At least, that’s what the headline said. And admittedly, the first few paragraphs said that, too. At least, they were about one specific case of this. But as I read down the article, I was surprised to see that when it presented the hard data on medical leave and why people have been taking it, the reporter admitted that no one knows why the leave is specifically taken. In other words: there is no real data, other than anecdotes, to support the claim.

This happens not infrequently (weasel word! I know) in journalism. The headlines aren’t necessarily written by the reporter. The editor could well slap a title on there and may not understand the story well enough to summarize it accurately or may even not care about accuracy as much as getting attention. In fact, even an accurate headline is often rather hyperbolic about what the story actually contains, probably for this reason

So what? Read the entire story and don’t trust the headlines.

posted by John Weiss at 16:21  

Powered by WordPress