Garbage In Garbage Out and the Desire to Cover Our Own Ass is Ruining the World
Tuesday, September 25th, 2007Anyone who has worked with statistics, models, or data of any form has probably run into the situation where someone wants a measure for something regardless of how useful that measure is.
Maybe it’s some version of the conjunction fallacy or other decision making error: A number gives us confidence regardless of how useful the number is.
Or maybe, like valuing the status quo over innovation, it’s a liability issue. Workers and managers want to be able to cover their own ass if something goes wrong. In such cases it’s always better to have some numbers, regardless of how applicable they are.
My bet is on the second reason, and it’s ruining the world.
Take an abysmal measure of inflation: the CPI. Granted inflation is a difficult thing to measure, but there are several bad problems that only make things worse.
The same goes for GDP, unemployment rates (or here or wikipedia) and basically any other measure of economic activity.
But maybe other disciplines aren’t like that? Don’t bet on it.
A friend of mine who is an accountant says that there is enough error and flexibility in accounting rules to make most balance sheets suspect. Consider the value of goodwill and brand value (look at that equation).
Measures of worker efficiency, political polls and statistics in scientific research all have regular problems.
Earlier I was talking about ourreference and discuss our tendency to slap a measure on everything, but the effect is that it causes us to shift our focus to maximizing the measure rather than what we really want.
As a policy analyst, I run into the ‘a garbage number is better than no number’ mind frame regularly. Government officials and big companies, the havens of bureaucracy need a number to justify their decisions. They need something to protect themselves if something goes wrong.
The problem isn’t really that we want to rely on numbers. It’s that we’re making big, important decisions based on numbers that have high measures of error at best or are just plain irrelevant at worst.
Our desire to cover our own ass is causing us to make bad decisions. In complex, important decisions the situation is always more nuanced than the reduction to measures indicates. But we can’t cover our ass if we make a decision based on a judgment call after considering all the complexities of the situation. That is an opinion and is open to dispute.
Yet what do we miss by relying on approximations so heavily? We miss all the subtle implications of the situation that can’t be measured easily or that get lost in the aggregation of data.
We often forego the time and effort of gaining a deeper understanding of a decision in the name of cost and efficiency. For small decisions it might be worth it to search less and ignore more information. But for large policy decisions it is almost certainly worth the effort of getting more information.
So to conclude this long rant (my apologies), lets reflect a bit on the importance of considering things that can’t be quantified, interactions that can’t be written down in a list, and factors that may only be known to us subconsciously.
Our reliance on numbers as the ultimate truth is ultimately misguided.
-zot
Support The Decision Strategist.Popularity: 17%









