Build Your Own Baloney Detector

A tool-kit for avoiding being fooled

Monday, July 30, 2012

An Interesting Example of a Context-Free Statistic

Here’s an article about an out-of-court settlement between a firm that was collecting payments on behalf of a hospital system and the State of Minnesota. The details aren’t of interest, here, except this: the firm is paying $2.5 million to settle the suit. Got that?

So, I have a vague idea of how much $2.5 million is. You most likely to do, too. But it’s still a low-context (or context-free) statistic. Why? Because we don’t know where the money will go, first of all. Will it go to pay back people who were wrongfully harassed? To pay people who were wrongfully charged money that they then paid? Or is this a punitive payment to punish the firm for their actions? In any of these cases, we’ll need more information to assess how much $2.5 million really is.

Consider: if we’re in the first case, then I’d want to know how many people will get the money and how this average payment compares to other such payment. In the second case, how does the total compare to how much people were wrongly charged? If $2.5 million is much less than that figure, this payment is symbolic at best. If it’s a lot more, then clearly — even with interest — this money is for more than just paying back those wronged.

And in the final case, I’d really want to know what this firm’s profits were like (at least in Minnesota). If $2.5 million is compared to $1 billion in profits, well, that seems like a very different punishment than $2.5 million compared to $10 million in profits, doesn’t it?

I can’t think of a case where I don’t need to know what this money is for and how it compares to some other related figure. We need context!

posted by John Weiss at 18:17  

Sunday, July 22, 2012

The Illusion of Balance (Part 1): Did they Check these Facts?

One of the ideals of reporting has long been “balance”, reporting the facts and giving the audience or reader a chance to decide the truth for themselves. This is a laudable ideal, of course, except that sometimes the pursuit of the appearance of balance (and fairness) gets in the way of actual careful reporting of the facts. I’ve seen this happen in two ways, so I’m covering it in two posts. Today, I’ll look at the problem with treating everyone’s facts equally. In my next post, we’ll look at the problem with giving equal time to all sides.

News media (and I’m not saying “reporters” because this also involves their editors and everyone else up and down the line) can choose to report facts in a few ways. A book I was reading recently (The Inquisition of Climate Science by James Lawrence Powell) suggested that this is best viewed as a continuum. I’m going to argue that there are at least two dimensions. The first is neutrality. The story can take a strong stance or it can be entirely neutral. On the former extreme, you have editorials, which aren’t usually considered news pieces per se. (Although in some cases, they seem to get slipped into the news without comment or label.) We tend to think that news stories should be on the neutral extreme, and with good reason. We don’t want them too neutral, however. We want the reporters to make logical inferences from the facts they collect and to some degree, taking a stand on some issues is perfectly warranted. (“Terrorist attacks are bad,” is probably not a viewpoint that almost anyone would object to, for example.) This is the continuum that most people think about when they think about news balance. It’s not the one I want to mostly talk about, though.

The dimension I’m interested in is the trust dimension. Reporters interview experts, witnesses, and parties concerned with stories on a daily basis. They also read countless press releases and studies. They do these things in order to report the facts or views that they collect in their stories. But are their sources accurate? There’s a fact-checking continuum, too. On the one extreme, you can fact-check nothing at all and just report statements (perhaps with attribution to save yourself from lawsuits). On the other, you can fact-check every single thing. The latter is clearly not economically feasible, nor is it reasonable. When the family of an accident victim reports their reaction, there’s no need to verify the relationship (a little trust is in order, in general) or to check on their credibility. In fact, doing so would probably cause unnecessary added insult and injury to already miserable people. So we can’t fact-check everything.

Sadly, it often seems that the news media has gone the other way lately and fact-checks nothing at all, simply reporting whatever either political party claims is true. The problems with this extreme are obvious when we think about them, so why does it happen? In part, because of time and money. Media outlets are on limited budgets and are being squeezed all the time. But there’s also an illusion of balance at play: to fact-check a politician might seem like taking sides against them. If the media appears to overwhelmingly check on one side or the other, that is seen as bias. So I wonder if the media also avoids checking on politicians’ facts for fear of seeming hostile to one side or the other. (This is also true of areas other than politics. When reporting a medical breakthrough, a good reporter will contact other experts in the field and get their views of the discovery, for example.)

Of course, fact-checking isn’t hostility, it’s the media’s job. I take it as a warning sign when I read a fact attributed to someone, especially a politician or pundit, without having been checked.

posted by John Weiss at 10:45  

Powered by WordPress