Home

Awesome

Reviewing academic papers

If you work in academia you will be asked to referee papers. Your first review will almost certainly be something you are asked to do by your advisor. But as your career goes on and you become recognized as an expert in one (or many!) areas, you will have the ahem...opportunity to review a lot more.

No one ever gave me formal guidelines for writing reviews. I think this experience is pretty similar to most graduate students. The format, tone, and content of these documents is usually learned via an apprenticeship model (if at all).

This document serves to describe the way that you will write reviews while a member of the Leek group. The goal is to avoid becoming that jerk reviewer, while simultaneously fulfilling your role to the academic community. It is a responsibility that must be taken seriously, but almost everyone wishes other people did better.

The key players in peer review

There are a few different players in the peer review process.

The first is the editor of the journal, who will do some vetting of papers at the beginning - mostly to screen out really crazy stuff that gets submitted (you'd be amazed at what gets submitted). At most journals, the editor is a senior scientist with a broad knowledge of the field who has a pretty good intuition about what is likely to be interesting to the readership of the journal and what is likely real science. Papers that are uninteresting or obviously wrong usually won't get past the editor.

The editor usually assigns the paper to an associate editor who has more expertise in the topics covered in the paper. The associate editor is usually a mid-level faculty member (senior assistant to associate professor). Again, papers that are obviously flawed or make wild claims often don't make it past the associate editor.

If a paper passes these hurdles it does not mean that it is correct or the claims are justified. It only means that on a quick read the paper seems interesting and not outlandish. The associate editor then makes an effort to find referees who work in the specific area the paper focuses on but do not have strong conflicts or collaborations with the authors of the paper. As you can imagine, in some areas of science it is hard to find referees that aren't in conflict one way or the other and have time to review. So sometimes they have to find people whose expertise is close, but not perfectly aligned with the theme of the paper.

What is your job in peer review?

A scientific paper can be distilled into four parts:

  1. A set of methodologies
  2. A description of data
  3. A set of results
  4. A set of claims

When you (or anyone else) writes a paper the goal is to communicate clearly items 1-3 so that they can justify the set of claims they are making. In the current peer review system there are three tasks you are responsible for as a peer reviewer:

  1. Evaluating the quality and accuracy of the methods, data, and results.
  2. Determining whether the methods, data and results justify the claims.
  3. Determining how important the claims are and whether they belong in the journal the paper was submitted to.

Ideally you would be able to verify every single claim in the paper and test every result. Given the time constraints of the review process and the remuneration you get for reviewing, this is absolutely not feasible. Your goal is instead to do your best to obtain reasonable estimators of each of the three components of the review and "show your work" by providing references, pointing to figures, and putting results in context.

Your prior belief about 1-3 should start with the assumption that the scientists in question are reasonable people who made efforts to be correct, thorough, transparent, and not exaggerate. You may adjust your prior if the paper was submitted to a journal that you and your colleagues have never heard of, or if you are asked to review a paper far outside of your expertise (legit journals try not to do this), or if the claims are so extreme that they would overturn huge areas of science (e.g. a paper claiming to prove evolution isn't true or that vaccines cause autism), or if the journal has the intent of only publishing papers that are groundbreaking.

Structure of a review

Your review will have three parts. The comments to the authors, the comments to the editor, and a recommendation.

Comments to the authors

When you write a review the first part consists of the comments to the authors; it should have the following components:

I think the summary is critical because if you can't distill the ideas down then you haven't really understood the paper. The summary should absolutely not be a restatement of the abstract of the paper, you should find the parts you think are most relevant and include them in the summary.

The major issues should be a bulleted list. Depending on the quality of the paper this list may be longer or shorter. A major issue has to be one of the following: (1) a claim that is not supported by the data, (2) a method or result that appears completely incorrect, (3) a critical missing piece of information, or (4) a paper that is not readable by a person trying their best to understand it. You should point to specific figures, paragraphs, or results for each major issue and be concrete about the problems. Vague criticisms are unacceptable. When possible, use references from previous literature to back up your claims.

Not making all data and code available with a specific link and instructions, is a major issue.

Minor issues should also be a bulleted list. There are a much broader range of minor issues that you may encounter. Some examples include simulations that miss some cases, figures that are missing axis labels, or the paper has extraneous results that aren't relevant to the claims being made.

Typos are not minor issues or major issues. It is not your responsibility to find them. If you do, you should provide them as a bulleted list to the author in the format: "On page x, line z, change ... to ...".

If there are a huge number of typos then that may be stated as a minor issue. If the paper is completely unreadable then that is a major issue. Completely unreadable means you could not follow the paper even after ignoring all typos.

Here are some things that your comments to the authors should not contain:

Remember that this is a professional document. They are typically anonymous (you don't have to sign your name) but the associate editor and editor will see the review and your reputation will be affected by the quality of the work you perform. There is no reason to be rude, competitive, or snarky in a review.

Comments to the editor

If you think you have covered everything in your comments to the authors you may leave this field blank. If you do put any text in, it should be no more than one paragraph. It should not contain any criticism of methods/results that you did not put in your comments to the authors. It may include a statement of how interesting you think the paper is and how appropriate it is for the journal readership if it helps justify your decision. It should be very consistent with the comments for the authors; if you aren't comfortable saying it to the authors directly, you should consider carefully whether it should be said.

Recommendation

You usually have these four options for the decision

Reject if you think that the methods, results, or claims are blatantly false. Reject if you think the paper has major flaws that could not be corrected. Reject if the paper is clearly not an improvement on the current state of the art. This third category is very hard to judge if you don't have a lot of experience in the field. If you are new to reviewing you should consult your advisor.

You should decide major revisions if you think there are serious problems with the paper but that they can be corrected. If you ask for major revisions your default plan should be that if they can/do correct all of the major issues you pointed out, you would be prepared to accept the paper. Sometimes, in the course of performing the corrections, they will show that their method/results/claims are not actually true. Then you should reject.

Do not ask for major revisions if you think the paper is uninteresting and you wouldn't accept it even if they did everything you said. This is the #1 way to be a jerk reviewer.

You should ask for minor revisions if there are only minor issues with the paper that you are pretty sure the authors can correct and you would be prepared to accept if the authors address those issues.

Do not ask for minor revisions if you think the paper is uninteresting and you wouldn't accept it even if they did everything you said. This is the #1 way to be a jerk reviewer.

It is very atypical for a reviewer to accept a paper outright. However there will be times when you receive a paper that has only minor issues and those issues are only judgement calls on your part, as opposed to things that need to be fixed to justify the claims or to make methods/results/data clear. It is perfectly acceptable in this case to list the minor issues and to suggest acceptance.

Length of a review

The best reviews are bullet pointed, brief, and point out exactly the key issues and nothing more. It is absolutely not your responsibility to rewrite the paper, change the message of the paper, or make the authors do something that wasn't in the scope of the original work. If you think the paper isn't appropriate for a journal in its current form, you should explain/justify why and choose reject. But you should not make the authors conform to your opinions.

There is a temptation to write really long reviews to show that you read a paper carefully and show off how expert you are. Do not succumb to this temptation. You get no bonus points for being nitpicky, verbose, or long.

You get big bonus points for the following things:

Very good reviews are often 1-2 pages long in bullet pointed format.

Re-review

Unless the paper was outright rejected or accepted, the authors will have a chance to respond to your review. If you have followed the guidelines above, it should make the re-review process more straightforward:

How long should it take you to review?

You should never take more than a month to review. If you accept a review, you should plan to complete it within a month. Ideally you will do the review in less time than that (think 2 weeks). If you are unable to do the review in that time frame you should politely decline and offer some alternative reviewers.

It is inevitable that you will miss some deadlines for reviews. It is an important component of an academic's professional life but it is not a priority above your own work. If you are going to miss a deadline, you should let the associate editor know and give them a time frame for when you will complete the review.

Remember that someone else put a huge amount of work into this paper and their career/livelihood depends on them getting papers published in a reasonable amount of time. If you think the paper should be rejected, do it quickly! If you think it should be accepted, do it quickly!