Or, at least, that’s what Kenneth P. Green and Hiwa Alaghebandian of the American Enterprise Institute want us to think. See, they did some research into how science is talked about in news sources. They conducted an experiment, which ran roughly as follows:
1.) They picked a bunch of “Authoritarian” sounding phrases.
2.) They counted up how often those phrases appeared in Lexis Nexis by year.
3.) Then they made a pretty graph.
No, seriously, that was about the extent of their argument:
To see if our suspicions were correct, we decided to do a bit of informal research, checking Lexis Nexis for growth in the use of what we would categorize as “authoritarian” phrasing when it comes to scientific findings. We searched Nexis for the following phrases to see how their use has changed over the last 30 years: “science says we must,” “science says we should,” “science tells us we must,” “science tells us we should,” “science commands,” “science requires,” “science dictates,” and “science compels.”
There are a few problems with this “research”. (To be fair, they did admit that it was “informal”, but that didn’t stop them from drawing some far-reaching conclusions from it.) The main one, however, is that the thing they are measuring isn’t correlated with what they’re trying to prove. They’re measuring number of references of certain “authoritarian” phrases as a function of time, but they’re trying to prove that the way science is talked about is changing. All that the authors really succeeded in proving is that science is being talked about more
The problem is that the number of references of these “authoritarian” phrases is not an independent variable. It is, in fact, linked very closely with another important number that the authors entirely ignore: the amount of text in Lexus Nexus by year. Sure, there are more mentions of the phrases the researchers chose now than their were 30 years ago. But I’m pretty sure that that’s true of any reasonably common, generic phrase.
I’m sure the number of times that the phrases “the investigation is ongoing” and “alcohol may have been involved” have also increased in the past three decades. Does that mean that investigations or alcohol consumption have suddenly skyrocketed? No. It means that people are writing a hell of a lot more for a wide variety of reasons. (I’m reasonably sure it’s significant that the start of the positive trend in the number of references correlates roughly with the rise of the Internet and the World Wide Web.)
In order for their numbers to mean something even close to what the authors claim they do, they would need to be measuring something like “authoritarian references per column inch” or “authoritarian references per article”. Something that gives us a sense of the “density” of references in the larger (and changing) volume of media. Unless they can provide such a “density” style measure, we really don’t know if the tone of scientific discourse is changing (to be fair, it might be) or if people are writing more, or writing about science more, or if Lexus Nexus is just covers more sources for recent years than for previous ones, etc.
Now, to be fair to the authors, their argument might be entirely true. If it is, then their research is insufficient to show that. To be honest, I imagine that the way science is covered by the media probably has changed quite a bit in the past 30 years. I do genuinely think that part of that change revolves around writers trying to cast descriptive science in a proscriptive fashion. But if we’re going to try to reform the way in which science is seen by both the media and the public, it behooves us to be accurate in both our experiments (informal or no) and also our arguments.