Consider the Source: Why Credibility Matters as Much as Raw Data

Even the most serious of truth seekers must rely heavily on the analyses of others.

The more I learn, the more I'm amazed by people who claim to base their views on "the data alone."  It's a noble dream, but consider the following.

Carefully studying data is enormously time consuming and question-specific.  If you really hold yourself to this standard, you won't be able to responsibly hold opinions on more than a handful of questions. As a result, people who claim to base their views on "the data alone" end up outsourcing most of their data work to others.  There's nothing fundamentally wrong with this, but note the bait-and-switch: People who claim to rely on "the data" are in fact relying on their own judgments about which sources are credible. Data analysis is great, but knowing who's credible is even better

Data analysis and credibility assessment are radically different skills.  Yes, you could carefully audit an expert's data work on one issue, then broadly trust whoever performs well.  But that's only one approach.  We're at least as likely to evaluate credibility based on demeanor, word choice, and mood.  And this seems reasonable.  I delete spam because the senders radiate crazy, not because I carefully audit any of their statements.

Furthermore, both direct data analysis and credibility judgments rest on a more fundamental faculty: memory.  Even people who conduct their own data analysis don't re-do their work before speaking.  Instead, they rely on what they remember about their data analysis.  The same goes, of course, for credibility.  When we decide to trust someone, we rely almost exclusively on our memory of their trustworthiness.

The big lesson: Truth-seekers spend far too little time assessing the tools that underlie almost all of their judgments.  Data analysis is great, but knowing who's credible is even better - and measuring the reliability of your own memory is paramount.

Betting simplifies this seemingly Sisyphean process in two keys ways. First, betting is a good, clean way to evaluate credibility.  Second, betting constrains selective memory: If you want to objectively judge a person's reliability - including your own reliability! - don't go with your gut.  Check out his life-long betting track record.  It's not the best possible measure, but it's probably the best feasible measure.

A version of this article first appeared at EconLog.

Further Reading

{{article.Title}}

{{article.BodyText}}