Science & Fun - Jonathan Graehl

From In Praise of Appreciative Thinking:

I presented three computational models for how attention could lead to consciousness. Based on the papers we assigned, one of them had strong support from the data (Taylor’s CODAM), one of them had medium data-based support (Wang’s LEABRA, small amount of data given via Figure 3 here), and one had little to no support from data (Cavanna’s figure 1 here, the “ascending reticular activating system”). We asked the thirteen other students to rank the models at the beginning of the class, with 1 being the best. Taylor’s model was ranked last, with an average rank of 2.21, Wang’s was second best, with an average rank of 2.07, and Cavanna’s was considered the best, with an average rank of 1.71. So, there was an inverse relationship between the amount of data presented between a model and how much people preferred it.

I’m sure the students preferred the storytelling of the least-supported theory.  But I agree they should prefer the story that is a proven model for actual data, even if it’s less pleasing aesthetically.

 It’s probably what everyone has told them to do their whole lives: be critical of everything you read, don’t trust statistics, etc. The better advice would be to simply try to gauge the veracity and utility of any individual claim based on the data you are given and your prior beliefs about the possible bias of the data source.

Interpreting data is hard.  Most students probably didn’t try to do so.

If people are critical of every new idea they encounter, all that will do is bias them towards favoring the status quo or the null hypothesis. I blame the market failure on our social norms: being critical is too often viewed as automatically synonymous with correct.

Lazy skepticism is a necessary filter; too much “research” is being produced and publicized.  However, if a trusted expert recommends a paper, I’d be happy to learn about the evidence in it.  I’d prefer that they summarize for me the data and the inferences they think should be drawn from it, because I’ll rarely bother to sift through a paper aimed at the clique of academics who specialize in the topic.