Build Your Own Baloney Detector

A tool-kit for avoiding being fooled

Wednesday, December 29, 2010

Speaking out of Your Area of Expertise

I’m pleased to say that while this trap is seductive, it’s usually easy to guard against if you watch out for it.

What is this trap? It’s simply that just because someone is an expert in one field, it doesn’t follow that they’re an expert in another (even related) field. For example, I’m an astrophysicist by training. I study Saturn’s rings. While I can’t claim my pronouncements on the topic are authoritative, I’d hope you might listen carefully to my thoughts. But as you get farther away from that topic, my expertise wanes. If I explained the geology of Venus, you might want to take it with a grain of salt. If I gave you medical advice, you’d be well-advised to take it with a heaping spoonful of salt. Now, there are other areas that I have at least minor expertise in, but you can’t know what they are from just my professional degree. For example, I think I do know more than average about migraines, but that has nothing to do with my degree so I’d entirely support you asking me about that before taking what tell you as valid.

Does this really happen? Yes! I just finished reading Merchants of Doubt (which I recommend to everyone, everywhere). One common theme in the book was scientists talking about issues from the medical dangers of cigarette smoke to the environmental dangers of ozone depletion and global warming. What’s interesting about these scientists is that nearly all of them were (and are) speaking well outside their expertise area. Usually, it was physicists (solid state of nuclear) talking about medical effects of smoke or the effects of CO2 on climate. While it’s possible that they did, in fact, do the work needed to become experts in those areas, it’s by no means assured. (I know of no reason to think that they did and, in fact, their pronouncements on the issues suggest that they didn’t. Or that they were willfully misleading, which is even more depressing.) And yet they were (and still are) taken very seriously by congress, the White House, and by the press. People see “scientist” and assume that our fields are interchangeable. They’re not.

Similarly, an engineer is not qualified to diagnose your sore arm. Even a dentist, a medical professional, is probably well outside of her qualifications. (In fact, a lot of MDs are probably unqualified to give a solid diagnosis. A skin specialist, for example, is probably out of his depth with joint problems and you’d be better off talking to either your GP/family medicine/internalist or a specialist in joints.)

So what’s the solution? Check the qualifications of the speaker and think carefully about whether or not they match up with the claims being made. The internet has made this a lot easier, although generally if someone is touting their qualifications, they tell you what you need to know when they do. And if you’re not sure what an impressive-sounding qualification means, check. Not everyone practicing a medical field is well-verses in biochemistry, for example, so not everyone with a degree in a medical field should necessarily be trust if they claim that they have a new theory about some aspect of biochemistry. (They might be right, of course, but don’t rely on it! Check!)

posted by John Weiss at 17:51  

Tuesday, November 23, 2010

Context Free Stats in the Wild

Here’s an example of context-free statistics that I came across a few days after my last post.

Without commenting on the underlying statistics or political conclusions (really, even if I wanted to, there’s not enough information for me to fairly do so, I think), consider what we’re told here:

For example, Marissa Mayer, known as “the face of Google,” gave $30,400 to the Democratic Congressional Campaign Committee in 2009.

OK… so what? Do we know who else she donated to? In the case of political donations, it’s certainly not uncommon for the wealthy to donate to more than one party, so knowing only that she donated to this campaign doesn’t tell us much even about this one individual. And that last word is also important: it’s just one individual. What about others?

In fact, of the top 10 contributions made by Google in 2009, only one — by CEO Eric Schmidt — was to the Republican National Committee.

On the surface, this seems damning. Maybe it is. But I can’t tell! To whom did the other nine contributions go? All Democrats? The DNC? Or was this only contribution to the RNC and the others were distributed to various candidates (possibly form both parties)? It’s entirely possible that the implied meaning of this statistic really represents the underlying data set, but I can’t tell from this stat. (It definitely has an implied meaning, though.)

It gets weird from here.

Steve Ballmer donated $5,000 to the Democratic Congressional Campaign Committee as well, but other Microsoft donations show an even split between Democrats and Republicans.

So… why single out that one contribution? (It could have been the largest or just the one from the most prominent Microsoft exec, but there’s no explanation.)

And then,

However, according to data collected by Consumer Watchdog, a consumer and advocacy group, Google has made more financial contributions to Republicans lately than to Democrats. The company has contributed 55 percent to Republicans and 45 percent to Democrats.

Wait, what? OK, just for the record, this article is contradicting is major claim by suggesting the opposite of what the headline proclaims. But I can’t even tell that, to be honest: has Google made more contributions in terms of number or dollar amounts? And are talking Google or Google employees/execs. (There’s a difference, although not necessarily from a perspective of how the money influences policy makers.)

From here, the article mostly devolves into a lot of interpretation from pundits. (Few of the claims are supported and there are a few more context-free stats, like how much Steve Jobs has given to Democrats, sans info about whether/how much he has given to Republicans.)

The sad thing is that if you follow the link to Adam Bonica’s site, he seems to have a good analysis going on. (I’m not an expert in political science, so I can’t judge terribly well. Some of the aspects of the graphs are worrisome, but we’re in very different fields with very different statistical norms.) But you can’t really tell from this story because the statistics aren’t given context.

posted by John Weiss at 19:15  

Sunday, October 31, 2010

Context-Free Statistics

I’ve been noticing this more of late, although I realize it’s been with us for quite a long time. You’ve seen it, too. A new report or an opinion piece will quote some statistic, whether it be a percent of people who watch the Daily Show who lean left or how many cubic kilometers of ice sheet we’ve lost this year. But what they won’t tell you is some context in which to interpret those statistics.

Take the ice sheets: if I told you that Antarctica loses 100 cubic kilometers of ice annually, what would you make of that number? Large? Small? Cause of worry? Or just an interesting datum? Honestly, by itself it’s impossible to tell. What you need to know is how many cubic kilometers of ice are in Antarctica, for example. Or how much that melt will raise sea levels. Or whatever else context will let you interpret that number appropriately for the story at hand. But by itself, unless you’re an expert in this field or have a particularly good sense of how large ice sheets are (or, at least, how large a cubic kilometer really is), the author might as well have not given you this number at all.

Another example, taken from news of yesterday’s “Rally to Restore Sanity and/or Fear”: some of the media coverage was giving the breakdown of Daily Show fans’ political leanings. It was something like 40% liberal, 38% independent, and 19% conservative. So why is this a problem? Well, it’s more subtle than the last example since we all know what 40% is, but ask yourself: what is the point of these stats? If the only point it to know what Stewart’s audience thinks, politically, they’re fine as they are. But if the author is trying (perhaps surripticiously) to suggest that Stewart and/or his audience is more liberal than normal, we need more information to interpret these statistics. Specifically, how representative is this of the demographic the audience is drawn from. Other studies have shown that the Daily Show audience is younger than normal, so you can’t compare their politics or other habits with the entire adult population and be really fair about it. You need to tell us how this compares with the background population that they more specifically belong to. (Similarly, you never see anyone compare the outcomes of a political survey like this with world-wide leanings because it’s not really helpful to know if an American sub-population is more or less conservative than China or South Africa.)

I suspect that often times, reports fall into this trap unintentionally because they’re not necessarily well-trained in the meaning to numerical data and how to interpret it. But I also suspect (yes, this is me ascribing motivations; take from it what you will) that this is done some of the time to gloss over inconvenient contexts and use numbers of pure shock-and-awe.

posted by John Weiss at 14:05  

Wednesday, October 20, 2010

Conservation of Energy

“No Free Lunch” in physics form. The law of conservation of energy states that in a closed system, energy is never created or destroyed. It can change forms (from chemical potential in gasoline to kinetic in a moving car to heat in our brakes, for example), but it’s always there in the same amount. This is one of the most profound, useful, and often depressing laws in physics.

It’s depressing because it means we can’t get something for nothing. Any energy we use has to come from somewhere and at the expense of something else that would otherwise have that energy. Now, often times this is acceptable. Stealing sunlight from the ground of a desert probably costs little ecologically. Stealing a bit of kinetic energy of water in a hydroelectric plant probably does little harm to the river. But it does come at a price and the price may be higher than we wish.

Conservation of Energy is also a useful principle. It can be used as a shortcut if you know where all of the energy is in a system. (For example, those classic physics problems where the energy is all gravitational potential or kinetic are so much faster to do with Conservation of Energy than the straight up way.) It is widely used by physicists to solve problems that would be a hundred times uglier without it.

And the principle is profound because what it says about the universe. At first blush, there’s no reason why the universe would have had to conserve energy (or anything else). Now, it turns out that if the laws of physics are the same in time (that is, laws like gravitation aren’t changing from what they were a billion years ago) , Conservation of Energy is the natural result. In and of itself this is awfully profound and not just a little bit cool. There’s no a priori reason why this would have been the case, at least not with a casual glance. Time-symmetry isn’t obviously connected to energy until you look under the proverbial hood of the physics engine, be down inside, connected they are.

All that is as may be, but what we most care about here is how you can use the principle of Conservation of Energy as a tool. Here’s how: any time some suggests a source of power for anything humans get up to, ask yourself where the energy comes from. Sometimes it’s well-known and obvious: solar cells use sunlight energy. Gasoline is refined from oil, the by-product of the breakdown of living matter millions of years ago. But what about electric cars? Where do they get their energy? Sure, it’s electricity right before they turn it into kinetic energy and drive off. But where does that energy come from? For a true electric car (not a hybrid), it comes from the same place the electricity in your home does. Depending on where you live, this could be from renewable energy sources, coal, or nuclear. So how “green” are electric cars? It depends a lot on which of those sources supplies the electricity.

How about hydrogen fuel cells? The energy there isn’t electric, at least not when it’s stored. It’s stored as chemical potential energy in the form of hydrogen. Hydrogen loooooves to react with oxygen and make water (a by-product we don’t worry about as much as carbon dioxide). In reacting, it releases energy, which can be harnessed to make electricity (or heat, if you prefer to make that instead). Clean, right? Well, maybe. Where do we get the hydrogen? Remember, hydrogen likes to react with oxygen, that’s what makes it a good fuel. It doesn’t exist here on Earth as hydrogen gas very much. We have to make it from water. Oh, dear, wait: if we start with water and end with water, we’ve put back as much energy into the chemical bonds as we’ve taken out. So where does the energy come from? Answer: not from the hydrogen. We need to put energy into the system to break apart the water. Where does that energy come from? Well, that depends on where you live and who builds the plant. It could be electricity from the grid or solar power, for example. But in any case, the energy has to come from somewhere. (In the end, the hydrogen is basically a form of a battery, storing energy for later use. Then again, in a sense, so is gasoline.)

As our energy consumption drives us to use more and more energy, we worry more about the sources. (This is for all kinds of reasons, ranging from environmental to geopolitical.) Conservation of Energy is a valuable tool to help us citizens navigate the rhetoric (some of it dishonest) about energy policy. While it might be a physicist’s favorite principle, it’s time we shared it with everyone else.

posted by John Weiss at 11:46  

Saturday, October 16, 2010

Free Lunches

“There’s no such thing as a free lunch,” goes an old saying. I’m not sure I can agree that it’s strictly true, but it’s still wise advice. We seldom get something for nothing, but people often let themselves be suckered into thinking that someone, often an anonymous stranger, really is giving them free stuff. It’s true that humans can be and often are altruistic and help each other, but that seldom occurs between strangers or between businesses and people.

The “Free Lunch” flag takes many forms, some of which are kind of subtle, some not so much. For example, there are sites that let you play “free” games on the internet. Now, granted, some of these are amateur games that really are being shared free of charge. Mostly, though, something is driving the business model. Often, it’s advertising for either the games’ creator or a third party. This may be an acceptable price to pay to you, in which case: have at it. But remember that there is a price.

More subtle examples of free lunches abound, however. Consider customer loyalty cards. You get discounts (often) for using them, but is that really something for nothing? Nope. Generally, stores are collecting data on your shopping in exchange for the discounts. Again, it might be worth it (the data is usually used only in aggregate, they don’t care about your purchasing patterns particularly), but it’s a price none-the-less.

In the end, the free-lunch problem doesn’t mean that we should refuse offers that look good. But it does mean that anytime we hear about something that sounds too good to be true, we should think about the details and what the (often hidden) costs might be.

posted by John Weiss at 18:33  

Friday, August 20, 2010

Useless Statistic, Caught in the Wild

Schweyen and city traffic engineer Gary Shannon said a traffic signal is no panacea. A city study of pedestrian accidents between 2002 and 2007 recorded only one accident at Broadway and Third Street, in 2005, Shannon said. Several signalized intersections had many more pedestrian accidents in the study period — including six at South Broadway and Fourth Street and five at South Broadway and Second Street, Shannon said.

— From a Rochester Post-Bulletin newspaper article

Presumably, this article paragraph is saying that a signal may not be needed on Third Street because some other intersections with signals have more accidents than this one. But hold up: does this matter? Is this statistic even meaningful?

An intersection can have a high rate of accidents for a variety of reasons, ranging from usage, to bad lines of sight, to user error. A traffic light can help with some problems, but a very busy intersection will still have accidents.

In fact, this is a sort of a false dichotomy. The choice being made isn’t where to spend money between these specific intersections, it’s simply whether to add a light on one of them. Whether other intersections that already have lights have more accidents or not isn’t really a useful datum.

What we really ought to be told is how much the accident rate has improved (we hope it’s improved) at those intersections after they added the signals and how much it would help (projected) to add one on Third Street. Or, comparing apples to apples, what other intersections are problems where that same money might be spent to alleviate accidents. (Either by adding or by upgrading lights.)

posted by John Weiss at 13:23  

Tuesday, June 8, 2010

Weasel Words, Caught in the Wild

But like it or not, lots and lots and lots of Americans need large vehicles for their jobs, their families, and their lives.

— From an article on gas mileage in cars

Here’s a great example of weasel words in action. “Lots and lots”? How many is that? Or, a better question: what percentage of vehicles? Sure, a million vehicles (to pick a number) is a lot by almost anyone’s standards, but that’s less than 1% of Americans. So, since the author is arguing the case for people having big cars, what’s the percentage and how does it compare to how many large vehicles are actually out there? I don’t know the answer and, in fact, I don’t even know how to adequately quantify “need”. I’ll bet the author doesn’t, either, but it’s not stopping him from using this would-be datum to make a point.

posted by John Weiss at 21:25  

Tuesday, June 8, 2010

Weasel Words

When we’re writing or — especially — speaking, it’s so much easier for us to avoid looking up actual statistics we’re talking about. Instead, we often (see how I’m not quoting a frequency?) just use words like “a lot”, “often”, most”, “many”, and so forth. By itself, this isn’t a bad thing: we’d never have enough time in the day to constantly look up all of the statistics we’d need. But they do cause problems.

You see, it’s easy to abuse this words as weasel words, phrases we throw in not to simplify our lives but to give a mistaken impression while maintaining dependability. If I say something like, “a lot of people want X,” you would probably walk away thinking there’s a large percentage of people who want that. But really, what does “a lot” mean? More than 5? 10? 100? 1000? A thousand people is, by most standards, “a lot”, after all. I wouldn’t want that many in my class, for example. But for national politics, it’s a tiny number. Of course, if you called me on my statement, I could just hide behind the ambiguity of the phrase, “a lot”.

I’m not saying you should question the exact stats every time you encounter these phrases. But certainly when reading (or listening to) formal communication, be it a corporate memo or an op-ed, it’s worth asking what the values actually are and why the writer/speaker didn’t give them to you. Did he not know/couldn’t be bothered to check? Did she want to hide them? Or did the stats just not exist? (In which case, how appropriate is the quantifier at all?)

posted by John Weiss at 21:14  

Sunday, March 7, 2010

Magnetic Fields

Magnets are probably one of the most amazing things we ever get to play with. Watch a kid play with some standard magnets sometime. You’ll remember how wonderful magnets really are. How do they attract and repel each other? How do they disrupt your TV screen? (Assuming you have one of those rapidly disappearing CRTs.) How does a compass know which way north is all the time? Even Einstein as a child was delighted and perplexed by this force. (Of course, he went on to create Special and General Relativity. I went on… to play with magnets well into adulthood. Look, we can’t all be geniuses, OK?)

Unfortunately, the very wonder and mystery of magnets also makes them a popular tool for charlatans. It also causes no end of the confuse among well-meaning people who don’t have good intuition for magnetic fields. (And why would anyone? They’re not particularly part of our natural everyday experience — we’ll see why in a minute — so our ancestors never needed to know it. Evolution, you work too slowly!) So I figured I’d spend a little time talking about what magnetic fields do and don’t do.

First of all, what is a magnetic field?

No, I refuse to answer that even remotely fully. We could spend an entire course on that topic and just barely get into it. So let’s do the abbreviated version.

A magnetic field is a field that interacts with charged matter. Magnetic fields tend to deflect charges, altering their paths. (A time-varying magnetic field can also generate an electric potential, driving current down a wire. This is the principle behind an electric dynamo.) The fact that magnetic fields only act on charged matter in important. Most matter in the universe isn’t very charged, as it turns out. Sure, matter is made up of those electrons (negative charge), protons (positive charge) and neutrons (little buggers have no charge and are aloof regarding magnetic fields; snobs). But as it happens, positive and negative charges respond precisely the opposite way to a given field. So if you bind a proton and an electron together (making a hydrogen atom, in this case), they behave as if there is no charge there at all and proceed merrily on their way1.

So you and I and this desk and even our planet are mostly uncharged because almost all of our electrons have a matching proton to live with. OK, it’s true you can give yourself a net charge by scuffing you feet on the carpet. Go ahead and try it. (We’ll all wait while you find a friend or, better, a relative to zap. Just, please, don’t zap a pet. They can’t zap you back and fair’s fair.) But as it turns out, that charge is small. “Small compared to what?” you ask. Well, first of all, small compared to the number of protons and electrons in your body. You can show that even a minuscule charge imbalance, say one extra electron in about 10 billion, would cause you to explode violently thanks to the electrostatic repulsion of like-charged particles. In fact, any time you do carry a net charge, you’re basically encouraging electrons to either join you or leave you to make you neutral again.

Another good comparison is the ratio of your charge to your mass. Electrons, being small and charged have a high charge-to-mass ratio. They react strongly to magnetic fields. Protons, being nearly 2000 times more massive (and equally charged) also have a pretty high charge-to-mass ratio, but smaller than the electron. They react well to magnetic fields, but they are more sluggish and don’t react as well as electrons. An electron moving through a magnetic field make a circular orbit around the field lines. (If the field is uniform and the velocity is perpendicular to the field anyway. That’s the simplest case, so let’s run with it.) A proton will also gyrate around the field line (albeit in the other direction thanks to its opposite charge), but with a much larger radius. Two thousand times larger, in fact.

So you or I, who have only a small number of extra electrons or protons, barely have any physical response to magnetic fields at all. In fact, if you or I had one extra eletron for every 10 billion in our bodies and walked along at a 1 m/s stroll (about 2 miles/hr), the force of Earth’s magnetic field on one of us would be about 100,000 times weaker than the air pressure from walking.

The conclusion here is that most macroscopic matter (that is, stuff you can see, starting from dinner and yourself up to and including planets and stars) are basically invisible to magnetic fields and do no react to them. Humans are not magnetic creatures. Intuitively, this is kind of obvious, I think. Humans invented the compass precisely because we aren’t magnetic and we need help finding magnetic north.

Given all of this, how effective would you expect magnetic fields to be at curing diseases? Or causing cancers?

Not very.

1 — In the interest of full disclosure, the hydrogen atom doesn’t behave exactly as if there were no charges. The energies that the electron can have as it “orbits” the proton are changed, for example. But that’s a minor effect and one that we can reasonably ignore. It certainly doesn’t alter the atom’s bulk movements, for example.

posted by John Weiss at 12:28  

Saturday, February 20, 2010

Argument by Innuendo

I’m adding this one because I just heard it on the radio yesterday. You actually hear this sort of thing a lot, but what really disturbed me was that I heard this argument on NPR during All Things Considered.

What is “Argument by Innuendo”? It’s an argument made by vaguely referencing some perceived lack of reliability in someone or some group. It’s generally not stated outright, probably because that might tip you off.

Here’s an example: the speaker on NPR, arguing for why the federal government shouldn’t regulate something (never mind what) said that they shouldn’t, “because they’re politicians”. This isn’t an argument for not doing something. What about politicians make them unsuited to regulate? There may, in fact, be reasons and those reasons may make a valid (or even convincing) argument. But that’s not presented. We’re just to assume that our prejudices against politicians make them unsuitable to regulate this.

What makes this argument particularly nasty is that it plays on prejudices most of us have. Most of us don’t trust politicians. Or lawyers. Or (insert political party here). And, of course, it doesn’t outright state the prejudice, so we can each fill in our own personal interpretation. It therefore goes with the grain of our views and subtly wins us over without every making a case. Do not be fooled!

posted by John Weiss at 16:21  
« Previous PageNext Page »

Powered by WordPress