Things that I have learned by being a journal editor

I have just completed a 3.75 year tenure as an associate editor of the American Political Science Review. Over that time, as one of six associate editors, I have managed the review process for 742 manuscripts, about 200 per year. Here are some departing thoughts on the job in no particular order and with no overarching thesis.

1. I wish the new editorial team well. Being an editor is a constant, low-level source of stress. There is always a queue of editorial activity that you could be doing. Even though it is always something that you could put off until tomorrow if there is something more urgent, the fact that it never done is a constant source of nagging stress. The manuscript in the queue that is currently longest in review is always one that has been under review for too long. When you add to that the fact that you are constantly having to make difficult decisions (on which careers may depend) it is fundamentally not a fun job.
   
2. The ability of editors to make sound decisions is fundamentally dependent on reviewers taking their task seriously, and the vast majority of reviewers do so. While many journals struggle to get reviewers, the rate at which reviewers accepted our invitations was consistently high. I had remarkably few cases of intemperate or inappropriate reviews, although of course review quality does vary! The best reviews are remarkably detailed and thoughtful, but more importantly, most reviews were good enough to make me comfortable about the vast majority of the decisions. I am grateful for all the reviews that I received, many of them from political scientists I have never met.
3. Unfortunately, if you are handling the volume of manuscripts that the APSR handles, with the very low acceptance rate that the journal maintains, the only thing one can be very confident of is that some of the decisions would not have come out the same way if we re-ran the review process. Even beyond the judgment calls by reviewers and editors that might have come out another way, there are decisions which are simply mistakes, one way or the other, because the reviewers and editors err. Some of these become apparent, some do not. I had a few difficult exchanges with authors and reviewers about close calls, but remarkably few.
4. The experience of being an editor has been useful for improving how I present my own research. I learned things about the field and about the peer review process. I have had less time to do research for the last four years, but have also become far more efficient.
   
5. Most of my compensation from APSA went into supporting my research, primarily running survey experiments. Optimising for less time but more money with which to do research has contributed to shifting my research in new directions that I am pleased with.
6. Editorial work did not remain intellectually engaging for the entire term that I was editor. The hope is that you will get to read and think about lots of the latest and best research in the field. Unfortunately, given the submission rate, actually spending that much time thinking about each manuscript would require spending far too much time overall, so there is a strong incentive to be the most efficient classifier that you can be. Is this an APSR paper or not? At a certain point, like a machine learning classifier, you start to lose the ability to articulate how it is that you know the answer to this question. This is bad for two reasons. First, it makes writing the decision letters more difficult. Second, if you stop carefully thinking through why you are making the decisions you are making, it is very likely that you will let your potential biases go unreflected upon.
7. I would make the same decision to become an APSR editor again, knowing what I know now. Nonetheless, I have no intention to agree to be a journal editor again. From a reciprocity perspective I continue to believe that being an editor at some point was appropriate given all the editorial work that I have created and will create for others through my own submissions. But I have done my bit now, and then some. The amount of editorial work that I have done is a very healthy multiple of the amount I will generate for others to do over my career.
   
8. As a matter of personal taste, I would have preferred to be an editor for a less “prestigious” journal where more of the focus would have been on correctness and less on importance. The most difficult part of being an editor (or a reviewer) for a journal like the APSR is adjudicating the ineffable quality of being important enough to be in the APSR. A few of the papers that get submitted clearly meet this criterion, but not nearly enough to fill the journal. The most difficult decisions involve trading importance off against other qualities of papers (see item 15 below).
9. It is better to desk reject more, but it is “easier” to send papers out (at least in an immediate sense). Each paper you send out today takes 3 reviewers out of the pool of those who are available for the next four months (I generally tried not to ask for reviews from the same reviewer within 3 months of receiving the last one, excluding second round reviews). The set of reviewers that you have not recently asked for reviews is a precious resource because reviewers’ time is a precious resource. It is depressingly common that when you identify the ideal reviewers for a manuscript, most of them are unavailable because they are currently reviewing or have just reviewed another manuscript for the journal.
   
10. It is difficult to maintain a consistent standard for desk review that is not very low or very unevenly applied across areas of research. I desk rejected 23% of the manuscripts that made it to me from the main editor, who had already desk rejected a proportion. Maybe this sounds like a lot to you, but keep in mind that the APSR publishes something like 5% of initial submissions. The decision to desk reject is just a statement that you are sure the submission is not in the top 5%, not that you are precisely sure that it is in the bottom X%, where X is the proportion you are desk rejecting. The problem is that it is easier to quickly identify which manuscripts have a chance in the research areas that you are most familiar with, which makes it possible to be more aggressive at desk rejecting manuscripts in those areas. This is “unfair” if you view desk reject versus reject with reviews as an outcome that we should worry about with respect to fairness. This proposition is itself debatable. The tradeoff between time under review and feedback from reviews makes it unclear who is really better off (and of course much depends on the usefulness of the reviews).
    
11. I would have been a better editor if I were a better professional networker. I do not particularly enjoy going to conferences or otherwise engaging in networking activities. This means I know fewer people than I might, which brought a penalty when it came to selecting reviewers. That said, having been an editor, I have a much broader knowledge of scholars in the field, even the ones who never submitted a manuscript, but I discovered while looking for reviewers.
12. There are many manuscripts that are submitted to the APSR after being first rejected at the AJPS, and I often discovered this when invited reviewers told me that they had already reviewed the paper. Some reviewers refuse to review the same manuscript multiple times for different journals because they think it is unfair to the authors. I tried to fight against this as a reason to refuse to review, with middling success. If there is a problem with the manuscript, especially one that might not be obvious to an editor but which is clear to someone with expertise on that specific question, it is (imho) insane for that information to be withheld from the editor of the second journal that the authors submit to in order to ensure a “fair” peer review lottery. What if only one person has noticed some fatal flaw in a paper? Arguably the most important function of peer review is to minimise the number of published papers whose core claims are incorrect. Authors are not entitled to disjoint random draws from the pool of potential reviewers.
13. It is very common for invited reviewers to decline to review when they are on sabbatical. While I certainly have some sympathy for wanting to carve out time for research, I think that if you are on a research-active leave (that is, not on parental leave) during which you would submit journal articles or book manuscripts for review by other scholars, you should probably also be available to review. Of course the usual degree of discretion applies: no one is under any obligation to review any particular manuscript they are invited to review. I understand that circumstances vary and sometimes reviewing is not possible for various reasons, but the reciprocity argument–if you are making work for other reviewers you should be available to review–looms large for me. I should note here that I did not argue with anyone about their decision to decline, since I am not entitled to justifications for these decisions and many people will have had further constraints beyond the ones that they quickly typed into the text field that popped up when they declined. Perhaps for transparency I should note here that I only reviewed 11 manuscripts for other journals while I was an editor, which is a very light load over nearly 4 years! Mostly I did not get very many invites (editor solidarity!), but I did apply a higher bar for fit than I previously applied and I declined nearly all invites from journals to which I have never submitted articles.
14. The ordinal prompt to reviewers that has long been used by the APSR and many other journals is badly designed. The choices of “Accept”, “Minor Revision”, “Major Revision” and “Reject” are not good at extracting the relevant pieces of information from reviewers. This becomes immediately clear when you compare these responses to the review texts. Among other problems, there are two distinct questions that are being flattened into one. (1) Should the best version of this paper be published in this journal? (2) How far from that ideal version of the paper is the submitted manuscript? In my experience, “Minor Revision” is used by reviewers to mean everything from “This is a superb manuscript that needs a few minor edits that you should definitely publish in the APSR” to “This is ok scholarship and there is not much that could be done to improve it, but it definitely does not belong in the APSR”. “Major Revision” is used by reviewers to mean everything from “This is potentially groundbreaking work but it needs some significant revision before you should publish it in the APSR” to “This manuscript is a complete mess, and even if the author fixed it, it would not belong in the APSR”. You can usually figure out which the reviewer means by reading the review, but it would make sense to just ask the two questions separately.
15. In making decisions, I tried to keep in mind three criteria for publication: novelty, validity, and importance. Is this new? Is this true? How many “people” (heavily weighted towards political scientists) will care about this? To be published in the APSR, a manuscript needs to have a non-zero value on all three of these, significant strength in two of three, and an honest statement of all three. That is, a paper that does something very new and brings very strong evidence for its claims can get away with being on a less important question. A paper that is very new and very important can get away with somewhat weaker evidence (if the limitations are clearly stated and better evidence is not available). A paper that brings very strong evidence on a very important question can get away with being only modestly novel. This leaves room for lots of different kinds of papers, on different kinds of questions, with different kinds of methods.
    
16. The common “where is the theory?” reviewer criticism is often an “importance” concern in disguise. Why should I care about in particularly? The answer is often that they speak to some broader puzzle in the field. There are certainly some reviewers that think that an APSR paper needs a section called “Theory” and a claim that a new theory is being presented, but most reviewers just want an engaging articulation of what is at stake and where it fits into what we already know.

Finally, a broad point about the APSR and the role of journals like it in political science. My conclusion at the end of being an editor of this journal is that the field might be better off without journals like the APSR, but I am honestly not sure. Many of the obvious problems with the current political science journal system are driven by the fact that having “top journals”, and more generally a status hierarchy thereof, means a tremendous amount of scholarly energy is directed through the peer review process towards distributing professional rewards to researchers. I am personally sympathetic to the view that this distracts from what I take to be the more fundamental function of peer review, which is quality control. A status hierarchy in journals creates incentives for overclaiming in manuscripts and also for authors to submit their manuscripts to all the “top journals” sequentially, which creates substantial costs that are then distributed across authors, reviewers and editors.

Of course it is easy to identify things that seem bad, it is more difficult to design institutions that fix the problems, particularly without creating other ones. I have spent idle moments thinking about the potential for something like arxiv.org with a peer review system to supplant journals, and I think there is some merit in the idea, but I can also see plenty of problems too. The career stakes that currently induce people to spend a lot of time trying to optimise journal placement do not disappear if you change the way that we peer review and publish our work. Those career stakes might simply move their outlet from optimising journal placement to shameless advertising of research to other researchers, which would be more annoying and wouldn’t even have the current system’s virtue of improving research quality (however inefficiently it does so).

Finally, I will just note that whatever my mixed feelings about the current journal system, believing that system-level change might be good doesn’t negate the need to contribute to the system we have now. If you are submitting to journals, you should be reviewing proportionately, and occasionally editing. Just because the system of social cooperation we have now might not be the one you think is best doesn’t mean you can opt-out of the bits you don’t like (reviewing and editing) while still taking advantage of the bits you need for your career (submitting).

• « Previous 2019 UK General Election MRP Post-mortem