Build Your Own Baloney Detector

A tool-kit for avoiding being fooled

Friday, June 7, 2013

Confirms My Biases

I’ve noticed a lot of bogs/satirical stories lately getting picked up by the media or by social media lately. It might just be my own perception, of course, but things seem to be on the uptick. Especially the satirical stories. Perhaps it’s because there are more and more places online to find satirical news. Many people know not to believe The Onion, but other sites are harder to know about.

Here’s a rule I try to follow: if a story seems to confirm my political/ethical/musical/culinary beliefs, be more skeptical. Especially if it seems like it’s a little too on the nose. It seems unlikely that all of my political opposites are horrible people who do really crass or cruel things, so any story suggesting that this is the case probably needs more caution. Then I trace back to the source and try to figure out if it’s a legit media source. Often, the site is either clearly a satire site or clearly a highly biased site. Either way, not good.

(Why should you check before sharing these kinds of stories? Apart from wanting to be right for its own sake, you don’t want to be the person who spreads crazy untruths. If you really want to deliver a blow to your ideological foes, you want your best shot to really count. So make sure it’s accurate.)

posted by John Weiss at 18:00  

Monday, July 30, 2012

An Interesting Example of a Context-Free Statistic

Here’s an article about an out-of-court settlement between a firm that was collecting payments on behalf of a hospital system and the State of Minnesota. The details aren’t of interest, here, except this: the firm is paying $2.5 million to settle the suit. Got that?

So, I have a vague idea of how much $2.5 million is. You most likely to do, too. But it’s still a low-context (or context-free) statistic. Why? Because we don’t know where the money will go, first of all. Will it go to pay back people who were wrongfully harassed? To pay people who were wrongfully charged money that they then paid? Or is this a punitive payment to punish the firm for their actions? In any of these cases, we’ll need more information to assess how much $2.5 million really is.

Consider: if we’re in the first case, then I’d want to know how many people will get the money and how this average payment compares to other such payment. In the second case, how does the total compare to how much people were wrongly charged? If $2.5 million is much less than that figure, this payment is symbolic at best. If it’s a lot more, then clearly — even with interest — this money is for more than just paying back those wronged.

And in the final case, I’d really want to know what this firm’s profits were like (at least in Minnesota). If $2.5 million is compared to $1 billion in profits, well, that seems like a very different punishment than $2.5 million compared to $10 million in profits, doesn’t it?

I can’t think of a case where I don’t need to know what this money is for and how it compares to some other related figure. We need context!

posted by John Weiss at 18:17  

Sunday, October 31, 2010

Context-Free Statistics

I’ve been noticing this more of late, although I realize it’s been with us for quite a long time. You’ve seen it, too. A new report or an opinion piece will quote some statistic, whether it be a percent of people who watch the Daily Show who lean left or how many cubic kilometers of ice sheet we’ve lost this year. But what they won’t tell you is some context in which to interpret those statistics.

Take the ice sheets: if I told you that Antarctica loses 100 cubic kilometers of ice annually, what would you make of that number? Large? Small? Cause of worry? Or just an interesting datum? Honestly, by itself it’s impossible to tell. What you need to know is how many cubic kilometers of ice are in Antarctica, for example. Or how much that melt will raise sea levels. Or whatever else context will let you interpret that number appropriately for the story at hand. But by itself, unless you’re an expert in this field or have a particularly good sense of how large ice sheets are (or, at least, how large a cubic kilometer really is), the author might as well have not given you this number at all.

Another example, taken from news of yesterday’s “Rally to Restore Sanity and/or Fear”: some of the media coverage was giving the breakdown of Daily Show fans’ political leanings. It was something like 40% liberal, 38% independent, and 19% conservative. So why is this a problem? Well, it’s more subtle than the last example since we all know what 40% is, but ask yourself: what is the point of these stats? If the only point it to know what Stewart’s audience thinks, politically, they’re fine as they are. But if the author is trying (perhaps surripticiously) to suggest that Stewart and/or his audience is more liberal than normal, we need more information to interpret these statistics. Specifically, how representative is this of the demographic the audience is drawn from. Other studies have shown that the Daily Show audience is younger than normal, so you can’t compare their politics or other habits with the entire adult population and be really fair about it. You need to tell us how this compares with the background population that they more specifically belong to. (Similarly, you never see anyone compare the outcomes of a political survey like this with world-wide leanings because it’s not really helpful to know if an American sub-population is more or less conservative than China or South Africa.)

I suspect that often times, reports fall into this trap unintentionally because they’re not necessarily well-trained in the meaning to numerical data and how to interpret it. But I also suspect (yes, this is me ascribing motivations; take from it what you will) that this is done some of the time to gloss over inconvenient contexts and use numbers of pure shock-and-awe.

posted by John Weiss at 14:05  

Tuesday, July 7, 2009

What a Baloney Detector Is Not

It’s worth thinking about what a Baloney Detector (BD) is and what it isn’t before really getting started.

One thing it is not is fool-proof. While your BD can tell you to worry or even to turn down an offer, it’s not the same as knowing something isn’t right. To really determine if an idea is correct (at least to current evidence), you need to fully research the idea and data pertaining to it. This is a time-consuming processes, however. Often, life doesn’t give us the chance to do do this before making a decision. Even when we do get that kind of time, we have to pick and chose what we research. Hence the BD.

A BD tells you to ask more questions. To be wary of claim. In a pinch, it may warn you say “No.” However, just because your BD doesn’t detect anything fishy, it doesn’t follow that nothing is wrong. A clever shyster can evade any set of rules you use to catch them, so don’t get overconfident.

In many ways, a BD is like a wrist-watch. Sure, there are clocks that keep more accurate time and there are clocks with many more features (like weather indicators). But a watch has a singular redeeming feature: it can go everywhere with you and is there when you need it. Your BD is the same way: it’s not as good as peer-reviewed research or being an expert in the field, but it is portable and easy to use with a variety of topics.

posted by John Weiss at 21:25  

Tuesday, July 7, 2009

Welcome to Build Your Own Baloney Detector!

First of all, welcome to Build Your Own Baloney Detector. I hope you find this blog interesting, useful, or at least entertaining. I’ll try to make it all three, although I cannot make promises on the latter.

Of course, you’re probably wondering what this blog is all about. I’ll tell you. In A Demon Haunted World, Carl Sagan talked about a “baloney detector kit” as a set of tools to figure out when something isn’t on the level. This could be because someone is lying to us or that they’re just confused themselves. It doesn’t matter, many of the tools carry over, regardless of whether the source means to mislead us.

That’s what this blog is about: tools you can use to at least begin to determine when you should be extra critical. (Sure, we should question everything, all the time. But practically speaking, that doesn’t work well. And it tends to offend friends and family when we ask for confirmation that they had tuna for lunch. Man, that’s a dinner conversation I’m glad I don’t have to repeat!)

There are a few types of posts I expect to be making:

  • Flags — Flags are warnings that something may be amiss. They’re not proof, they’re just little things that should cause concern.
  • Questions to Ask — Questions we can ask, either of others or of ourselves, to help us determine if a given idea is reasonable.
  • Examples — Examples of skepticism at work. I suspect many of these will be drawn from my own life, but I welcome hearing about your experiences.
  • Building Blocks — Some basic facts, figures, and science that may help you evaluate ideas you encounter.

Apart from that, we’ll have to see how this beast evolves. I may not have enough material to keep this up for a long time, but as long as I do, I hope you keep coming back.

posted by John Weiss at 16:33  

Powered by WordPress