There is no shortage of academics complaining about the detrimental impact of the REF on scientific integrity and freedom. Similarly, the undesirability of publication bias is a well-trodden trope among academic circles. Witness then, the audacity of this blog in taking these well known ideas, linking them tenuously together, and declaring the result novel.
So, we all know that REF is the mechanism through which the four higher education funding bodies allocate funding to higher education institutions. It is meant to reward those institutions doing the best work with a greater proportion of the funding pot with which to do a greater volume of work in the hope that the quality will be unaffected. So far, so meritocratic.
Meanwhile publication bias is that mechanism by which certain types of studies are systematically more likely to be published, typically studies with positive results. Since negative findings are as important as positive ones, this is a bad thing as it leads to a distorted picture of reality in published literature. It means we only see selected parts of the truth and are missing whole other crucial sections, much unlike looking at a cat’s paw and seeing a teddy bear, rather than looking at the whole cat and seeing it for what it is (a miniature tiger). I digress.
It does seem though that the argument that REF’s impact criterion might unintentionally worsen publication bias hasn’t been explored as much as it might be. The simple idea being that it is much more difficult (impossible?) to demonstrate impact from a negative study than from a positive one. A well conducted study that shows an intervention doesn’t work is incredibly valuable, but how would one show that the work had influenced or altered practice elsewhere? Would an absence of further studies investigating a discredited intervention be evidence of impact? Impact from a positive finding is difficult enough to evidence. How hard would it be to show that something *not* happening was a direct result of your work?
And more importantly, if I was struggling to make myself REF returnable, would I be prioritising a study with a positive outcome, or one with a negative one?
Impact in REF’s terms can only be easily addressed by a positive study and in many ways is a step backward for science. Even the stated criteria to assess impact are “reach” and “significance”. The second of those terms is a remarkably poor choice if what they mean is “importance” (I have written about the use of the term significance previously).
Is it reasonable then, for REF to make explicit that impact statements need only be returned for studies that demonstrate positive findings, while they are optional for negative studies? Would such a change make clear that funding councils value all science, regardless of whether they attain some arbitrary significance threshold?
I should say in “researching” this post i found this article which skirts the issue, but perhaps doesn’t point the finger as firmly at REF as it could have. As ever, i am deeply resentful of comments so please keep your opinions to yourselves (read: please comment on my blog).