A clash between two major figures in cognitive neuroscience came to a head yesterday, and it got a bit ugly. The dispute centers around a publicly posted peer review of a publicly posted manuscript. Although seemingly aligned with Open Science ideals, the public review prompted worries about ulterior motives and power dynamics in some researchers. In addition to being juicy drama, the events reveal that Open Science requires trust, tact, and integrity. Some things I’ve learned from this:
- Pre-prints are not just scoop insurance: they also explicitly invite open peer review.
- Ad-hominem attacks are never acceptable; only ever rebut arguments or call out actions.
- Open reviews require everyone to operate in good faith, without ulterior motives or grudges.
- Part of the previous point, but worth a independent mention: Be aware of your power and privilege when writing reviews and responses.
- Always assume fellow scientists operate with good intentions, even if they are negative about your work.
- There are complex and unresolved issues surrounding consent to and potential career damage from open peer review.
For non-scientists, and even for those who aren’t familiar with the particulars of cognitive neuroscience, this whole thing will seem quite confusing. Hence, I will be relying on an analogy borrowed from the field of Celebrity Gossip.
In 2008, singer Taylor Swift released the single “You Belong With Me” as part of the album “Fearless”. With this album, Swift tried to advance the field of country music in a new and exciting direction. Not everyone appreciated her efforts: At the MTV Music Video Awards, where the accompanying music video won the award for “Best Female Video”, fellow artist Kanye West decided to provide a public peer review of Taylor Swift’s song. In the middle of Swift’s acceptance speech, West took the microphone and commented that although Swift’s work had some merits, it lags behind the state of the field. Furthermore, West alleged that Beyoncé et al. were doing much more important things. In an interview, Swift explained that she was hurt by West’s very public remarks, but she later also seemed to make bank by using the controversy.
A somewhat similar thing happened in cognitive neuroscience yesterday. In October 2018, Professor Bradley Love and colleagues of University College London published a manuscript as a pre-print, and also submitted it to a journal. In the words of Prof. Love, the manuscript tried “to answer fundamental questions the field has overlooked”. The journal asked Professor Nikolaus Kriegeskorte of New York University to review the manuscript. Prof. Kriegeskorte has a personal policy of openly publishing his peer reviews if an open version of the manuscript exists, and published his review of Prof. Love’s manuscript yesterday. In response, Prof. Love posted an open review of Niko Kriegeskorte to his own blog, alleging that Prof. Kriegeskorte “completely missed the main point”, that his review was a “cheap self-aggrandizing blog posts“, that its publication was “self-serving”, and at an inappropriate time.
What's the best measure of representational dissimilarity? Open Review of Bobadilla-Suarez, Ahlheim, Mehrotra, Panos, & Love (pp2018) [I3R3] https://t.co/KPViGIAoxh pic.twitter.com/P6bYpl91xW
— Kriegeskorte Lab (@KriegeskorteLab) January 9, 2019
Open is good right? In response to @KriegeskorteLab's open review of our work on neural similarity, I offer you "An open review of Niko Kriegeskorte", which is a less tedious read that touches on how difficult it is do something novel in this field. https://t.co/1flxz1Yu1t
— Bradley Love (@ProfData) January 9, 2019
I do not know either scientist on a personal level. I have met and briefly interacted with Prof. Kriegeskorte in a pub in Cambridge, and he seemed lovely. I have not had the pleasure of meeting Prof. Love yet, but we did briefly interact on Twitter yesterday (all in public tweets), and he also seems very nice and reasonable. Hence, it is my assumption that both scientists have the best intentions towards their work and their colleagues.
As I operate within a different sub-field, I do not think I have the expertise to critically evaluate the arguments put forward by either scientist. I encourage everyone with that expertise to read Prof. Love and colleagues’ manuscript, and Prof. Kriegeskorte’s review.
This post is written in the first person (grammatical irony intended), and the views expressed here are mine. As such, they are likely to be wrong.
What can we learn about Open Science?
Prof. Love raised important points in his post (both explicitly and implicitly), as did many people in the discussion that ensued on Twitter following the publication of his post. Below, I have tried to summarise the most important points highlighted in this discussion. Before moving on to those, let’s quickly summarise how scientific work is usually published, and how Open Science would change this.
How is academic publishing different under “Open Science”?
For the uninitiated: Journals are where academics publish articles that present their work. These are not usually glam journals with flashy photos and over-the-top claims (although some are, but let’s discuss that another time), but instead full of methodological information and statistics. Teams of scientists (usually PhD students and postdocs in the service of a single senior scientist) can work for years to get the data for a publication, write up their findings, and submit their manuscript to a journal. The journal then sends this manuscript two a few scientists who work within the same field, but are not associated with the submitting team. This is called peer-review, and it is considered to be an important check. The upside is that mistakes and downright wrong information are less likely to be published. The downside is that it is a slow process, and that not everyone always reviews with the best intentions (some people are very mean, and some actively try to block work that disagrees with their own).
Open Science is an unstructured movement that aims to, among other things, improve scientific publishing. One popular open initiative is the introduction of “pre-prints”, which are manuscripts that appear on websites without being peer review. Ideally, pre-prints allow scientists a sneak preview of the field, and allow anyone to provide an open peer review.
Another idea is open peer review, which journals like eLife and F1000 have already adopted, and pre-print servers explicitly invite via comment sections. Some people think that open peer review would prevent peer-reviewers from being mean or suppressing others’ work, or at least make it obvious when it does happen.
After peer review, whether open or closed, the authors of the original manuscript usually write a response. In this, they indicate how they addressed the reviewers’ concerns, or how they disagree with the reviewer. (In the latter case, the Editor of a journal usually decides who is right, and whether the manuscript should be published.) When peer reviews are open, so should responses to them be.
Ad-hominem is not OK
Regardless of your argument, you lose the very moment you try to discredit a person rather than rebut their arguments. In my opinion, Prof. Love crossed this line by being dismissive and quite condescending towards Prof. Kriegeskorte and like-minded scientists. I don’t mean to discredit Prof. Love or his other arguments with this; I just think that the way he handled it was mean.
It is completely understandable that Prof. Love is upset, and it should be pointed out that (privately) cursing out reviewers is a rather common thing to do among scientists. However, this rarely happens so openly. Sometimes reviews are nasty, sometimes they missed the point, and sometimes it just hurts when people criticise work that you care about (even if they happen to be right). I think being angry and releasing some steam is a good coping mechanism, but only when this occurs among friends. It becomes problematic when anger is directed directly at individuals (e.g. via threatening emails), or results in personal attacks in the public sphere (e.g. in a mean blog post).
We should be more aware of power structures
It’s slightly ridiculous that someone as renowned and well-established as Prof. Love (full professor at UCL and fellow at the prestigious Alan Turing Institute) complained that fellow well-established scientist Prof. Kriegeskorte (former project leader at an institute affiliated with the University of Cambridge, and current full professor at Columbia University’s Zuckerman Institute) is trying to suppress his views from a position of power, but that does not invalidate the argument that it is important to be aware of power structures and potential abuse of power.
Very many PhD candidates, postdoctoral scientists, and even assistant professors on temporary contracts are very likely to be terribly shafted when their work or reputation is brought into question, even if it is for the wrong reasons. Powerful people like Profs. Love and Kriegeskorte have an impact, and can quite literally break the careers of early-career scientists. In a field where jobs are scarce and contracts last only 2-4 years, junior people are at the mercy of the well-established scientists that rule the hiring process.
This. I think people advocating radical openness do need to grapple with this issue. I don't like the current peer review system at all, but the thought of a public free for all is deeply aversive. It's easy to abuse, and I don't like seeing people handwave away the concerns https://t.co/ZytpyXi0r4
— Danielle Navarro (@djnavarro) January 9, 2019
In addition to the hiring process, well-established scientists usually function as editors and reviewers for prominent journals. As such, they can act as gate-keepers, and block or stall work that does not agree with their ideas.
Open availability of peer reviews would make it impossible to suppress work without others being able to see it. Hence, it would require a much more concerted effort among multiple powerful people to block new ideas or competing theories. Some people think that reviews being public would make reviewers act kinder and more fair.
In the traditional system, reviews are hidden, and manuscripts only come out after potential flaws are picked up on. I have definitely made stupid mistakes in manuscripts, and I am very glad the reviews for those papers are not available. Quite frankly, it’s embarrassing when someone points out your mistakes, and especially among scientists there is a perception that your work has to be perfect from the start. This is especially true for early-career researchers, who do not have a large body of accepted work to fall back on. Thus, open reviews could negatively impact especially early-careers.
Scientists are conscious and conscientious beings
In his blog, Prof. Love calls people who agree with Prof. Kriegeskorte “acolytes”. He argues that others have rejected his manuscript referencing Prof. Kriegeskorte’s public review, and that this “poisoned the well” for the project. Although it is most definitely true that the opinions of senior academics like Pros. Love and Kriegeskorte carry weight, each individual scientist is also responsible for their own opinions and conduct.
Scientists are not blind followers; they are highly trained in critical evaluation of evidence, and thus should be able to make up our own minds about topics. People who agree with Prof. Kriegeskorte’s review of Prof. Love’s work are not necessarily blind followers of Prof. Kriegeskorte. They should be trusted as independent scientists, even if they take inspiration from a blog post that not everyone might agree is fair.
Speaking of trusting scientists: I think it is incredibly important that we start from an assumption that our peers act in good faith. It is entirely possible for someone who is simply doing their best and acting on their expertise to misunderstand a paper and to write a review that misses the point. I don’t know whether this is what happened here, but I think this assumption should prevail over the assumption that a fellow scientist is deliberately trying to kill a paper.
Pre-prints are not just scoop insurance
Posting a pre-print, to some, serves as scoop-prevention. It circumvents the possibility that your manuscript is held up during the sluggish process of peer review, because your work is already publicly available. The current drama highlights that pre-prints are also invitations for other people to review your work in the public sphere.
Repositories are set up specifically to allow for comments. You could argue that those comments should occur within the same venue, i.e. as comments embedded on the pre-print repository. However, open reviews posted as comments are equally public as those posted in blog post.
That said, it is easier to miss important context from a separate blog than it is from an embedded comment with the manuscript. It could also be argued that one should read a manuscript before reading its reviews, which is in the format provided by most pre-print websites.
would the commentary not be just as available to reviewers as a heuristics had it been left underneath the preprint as a comment? is the objection to the fact that people are likely to actually read Niko's blog, and not comments?
— Tal Yarkoni (@talyarkoni) January 9, 2019
Basically, at the end of the day, there is a system for leaving comments on preprint servers, it should be used. Because it is truly open: preserves context; allows authors to reply/engage in dialogue with criticism; and improve their manuscript.
— Olivia Guest | Ολίβια Γκεστ (@o_guest) January 9, 2019
open means you make things available, and people can comment on them and use them however they like. it doesn't mean "it's a preprint, but you still need to leave your comments on the same page as the original, and you can only post at certain times that are convenient for me"
— Tal Yarkoni (@talyarkoni) January 10, 2019
Open peer review is served by some rules
Importantly, Prof. Love alleges that Prof. Kriegeskorte published his review at a strategic time: Not after finishing it, but only after the manuscript was rejected. This prevented Prof. Love’s team from responding, while it did allow the world to see what Prof. Kriegeskorte thinks of the work. Thus, argues Prof. Love, his work received negative publicity without a chance to respond. In addition, it was just unpleasant for his team to receive a rejection of the journal, as well as public negativity.
Here, the old system clashes with the new. If Prof. Kriegeskorte were to have only done a review in response to Prof. Love’s pre-print, there would not have been an issue. It would clearly be invited by the existence of the pre-print, although we could argue about what the appropriate venue would have been (blog or embedded comment – see previous section). Instead, Prof. Kriegeskorte reviewed the paper as part of a journal’s publication process, and decided to publish it only after said journal had rejected the manuscript.
The timing of the open review has been hotly debated. On the one hand, Prof. Love argues that it should not have been posted only after rejection of the manuscript at the journal that it was submitted to. On the other hand, people have argued that publication before the journal’s decision could have affected other reviewers or the editor. (This is now also true for future submissions at other journals, although those can now be accompanied by Prof. Love’s comments.)
EDIT (2018-01-11, 10:55): Prof. Kriegeskorte has clarified the timeline in a comment below. He says that he generally aims to both submit (to a journal that invited a review) and publicly post reviews at the same time. In this case, he happened to submit the review a bit earlier than the post, partly due to time constraints, and partly to allow for edits. He also says that he reached out to Prof. Love on Monday, before posting his review on Tuesday. (It looks like the open review was posted on Wednesday in UK time, as the blog post is dated 9 January.) Importantly, Prof. Kriegeskorte refutes the statement by Prof. Love that he waited for an editorial decision at the journal, and says that he was in fact not aware of the other reviews or the decision at the time of posting.
You stated (here on twitter?!) that the time of publication was wrong and mainly serves self-promotion. This raises the question as to what time would have been better? Posting before the editorial decision may have been perceived as trying to bias the other reviewers.
— Tim Kietzmann (@TimKietzmann) January 9, 2019
Finally, Prof. Love has brought up the issue of consent. He and his colleagues did not expect the reviews from their submission to a journal to appear publicly, as this is usually against journal policy. They argue that they did not consent to a review being published, and argue that Prof. Kriegeskorte only used the existence of their pre-print as an excuse to publish his review openly. Prof. Kriegeskorte has a personal policy of publishing open reviews of papers he reviews for journals, but only if they have been published as a pre-print. Prof. Love argued that the review would have been different if it wasn’t for this blogging policy.
Yeah, it's not really open if the reviewer unilaterally decides what, where, when and how.
— Bradley Love (@ProfData) January 9, 2019
Yep, plus hard to pretend these are honest reviews when they are ultimately intended as blog posts to promote a certain brand of research. That is an inherent conflict of interest. One can't wear all these hats at once, especially when wielding power.
— Bradley Love (@ProfData) January 9, 2019
In my opinion, Prof. Love and team invited open reviews by pre-printing their manuscript. However, I do also see how it wasn’t tactful of Prof. Kriegerskorte to openly publish a review that was requested by a closed-review journal directly upon rejection there. The more courteous way would have been to keep these two review streams separate: Either publish an open review of a pre-print, or privately allow the authors to write a response, and publish these at the same time.
There are good arguments for wanting to share the reviews you wrote for journals, but this comes with an ethical obligation to be mindful of the content’s sensitive nature and the high stakes surrounding academic publishing.
In sum, Prof. Kriegeskorte should probably have been more sensitive about openly publishing what would otherwise have been a closed review, and Prof. Love should probably not have responded with a personal attack piece. Much more importantly, this clash between two very senior academics highlights issues with open science, and in particular with open reviews: They can be harmful for researchers, especially if they are in the early stages of their career. As usual in science: It’s the postdocs and PhDs that get bitten while the Big Dogs fight it out.
UPDATE 2018-01-10, 21:30
It seems that in the time between starting and finishing this blog (interrupted by some meetings), Dr. Sam Schwarzkopf wrote an open review of open reviewing, and Dr. Sebastian Bobadilla-Suarez (postdoc in Prof. Loves group, and lead author on the manuscript in question) has published his view on events. I highly recommend reading both!
That’s a thoughtful summary of the events and arguments.
I’d like to clarify my general peer review policy and the order of events in this particular case.
I do not do secret peer review at all anymore, only open review. All my reviews are published on my blog as soon as I finish them. This entails that I can only review published papers (preprints or journal publications).
To clarify the timeline: I submitted the review a bit hurriedly on Friday. My general policy is to post immediately. But in this case some final edits seemed desirable. I did these on Monday and emailed Brad the review as soon as I finished it to soften the blow and give him a chance to respond before I would post it. (This is not my general policy, but seemed appropriate here.) I posted the review on Tuesday. Contrary to Brad’s description, I did not know of any editorial decision and had not seen other reviews when I posted the review.
Thanks for the clarifications! I’ll update the relevant bits of the blog as soon as I’m near a computer.