Length of Your Review

A question posed to me recently asked about the best length for a manuscript review. Let me quote from the email:

“On one side of the spectrum is a group of people that will only comment if something is blatantly wrong. On the other side of the spectrum is the reviewer that will give a lengthy response including all of the changes that could improve the study (or at least as improve the study in the eyes of the reviewer).”

Neither of these extremes is optimal. A very short, highly negative review is particularly bad but close behind is a very short, highly positive review. The more details I have, the better that I can assess the manuscript and weigh the recommendations from the two reviewers. So, in general, I like longer reviews. Reviewers can, actually, go too far, suggesting additional studies and analysis that might be very good to conduct but are not necessary for publication of the submitted manuscript. To me, an optimal review includes praise of the good parts of the paper, identification of what is wrong or unclear, and suggestions for what will make the study publishable.

As I said in one of my first posts on this blog, please be thorough – I like the longer reviews better than the short ones. If you need to write several paragraphs to explain why some aspect of the study is off base, then please do it. That helps me make the right decision about the manuscript.

The main place that I find reviewers being too verbose is with suggested new work. If you write two paragraphs on how the authors should really include another section, then please stop and ask yourself: is this new section that I am asking for necessary to make the submitted manuscript acceptable for publication? If not, then your two paragraphs are really a suggestion for future work. Such suggestions are fine but it should be noted as such. If not noted, then Editors and authors assume that it is a recommendation for acceptability, and authors should either do it or write a thorough rebuttal as to why they didn’t do it.

At the AGU reviewer resources page, the text begins with the main goal of peer review:

“AGU Publications relies on our reviewers to help ensure the standards, quality, and significance of our papers.”

Ensuring the standards, quality, and significance of papers in AGU journals is, I think, best served by identifying those elements of the manuscript that make it fall short of being a “significant contribution to the field.” A terse report, positive or negative, is less than fully helpful to me as an Editor. Similarly, it is useful to expand, at least somewhat, on the positive elements of the manuscript that raise it to the level of acceptability.

Down the page a bit is this definition of a good review:

“In general, the most helpful review is one which first provides an overall summary of the main contribution of the paper and its appropriateness for the journal and summarizes what major items should be addressed in revision.”

I agree. A very brief summary of the paper is always appreciated, not only for the Editor but also for the authors, so that they know that you understood the main point of the paper. Positive comments on the appropriateness for the journal are good, too, especially if the other reviewer finds fault with the same aspect of the manuscript. While many reviews include these elements, a large percentage of reviews do not, instead jumping in to the negative comments. Having these two other elements, a neutral paragraph reiterating the main point of the paper and a positive paragraph highlighting what is good about it, also helps to set a cordial tone to the report, which is always a good thing, in my view.

I like the figure in the Eos article about writing a good review, linked on this page.

peerreviewguide-flowchart

It highlights that the first things a reviewer should write are these two elements, the summary and the positive aspects of the paper. Then move on to the concerns and suggestions.

This page also has a link to another page with the lists of questions and pull-down menus that reviewers will be asked as part of the submission of their report. This is not the full list of all things to consider in your review, but these are the questions that AGU would like to see answered about every manuscript sent out for peer review. Explanations of the answers to these questions should also be in the main text of the review.

So, longer reviews are, in general, better for the Editor. Just take this advice with some caution, however, and think twice about suggestions for new work. Make it clear in the report which suggestions are, in your view, simply possibilities for future investigations and which are recommendations for acceptability.

Advertisements

Preprint Servers: Challenges

A third (and probably final, for now) post on of ESSOAr, AGU’s new preprint server for Earth and space sciences. The first described it, the second touted it, and now this one is the ethical scold of how best to use it.

The biggest point to remember is that preprint servers are not peer-reviewed journals. Yes, there is an editorial board that checks submissions for scientific scope, but there is no vetting of the accuracy of the content. The editorial check takes a day or two, maybe a week max, but it is not a real review process. Yes, content here gets a DOI, but we should all remember that content on preprint servers are essentially just a step above “private communication” in terms of referencing authority. That is, it could be wrong.

We hope that content on ESSOAr, and any other preprint server, will eventually be published in a scientific journal. Researchers are putting their reputation out there with each new post on one of these servers, so the content is, for the most part, respectable. Go ahead and use it to learn what is being done by your colleagues. Because preprint server content has not been through the peer review process, though, it should be replaced with the “final” version of the study from whatever journal it eventually appears in.

To summarize in a graphic:

Caution-preprints

            Peer review should still be the standard for what is accepted as “knowledge” of the subject. Even this can be wrong but at least it has been thoroughly scrutinized by experts. You should be very skeptical of older preprints on the server (say, more than 2 years since original posting) that lack a link to a final published version of the paper. That work either was not submitted or did not pass peer review. If the former, then it is perhaps the case that the authors found a problem with the study and therefore never submitted that version of the paper. If the latter, then perhaps the editor or referees found a problem with the study and declined publication of it. Either way, the study did not reach its “final” form in the literature.

The advice to the community about older preprints can be summed up like this:

  • Authors: use caution when citing an older preprint.
  • Reviewers: pay extra attention to citations of older preprints.
  • Editors: ask reviewers to check the appropriateness of older preprint citations.
  • Societies: set policy about citing older preprints.

I am told that the astrophysics community, which regularly uses the arXiv preprint server, understands this difference in “publication” levels. That is, research communities can learn to use preprint servers and make it their go-to place for the latest content across a number of journals, as I am told that many in astrophysics do. They also know, however, that when it comes time to write your own paper, don’t rely on preprints as your main entries in the reference list. The astrophysics community, I am told, understands the guidelines about preprint servers and only uses it for finding the latest work on a topic.

We, the Earth and space science research community, should adopt this same mentality about preprint servers, not only ESSOAr but any server (and there are several being created). Such servers should be a place to get the latest studies from across a variety of journals, learning about content as the manuscripts are submitted rather than months later when they are accepted and eventually published. We should only use it for the latest work, though. A preprint server is not the place for full literature searches – those should be done in Web of Science, Google Scholar, Scopus, ADS, or other services that scan the published, peer-reviewed literature. And, as an editor, I strongly urge you to please conduct a full literature search, because a recent study by Mark Moldwin and me showed that the more complete your reference is, the more citations your paper will get (on average).

Use ESSOAr, but know its purpose within the hierarchy of scientific publications.

Annotating Manuscripts with Hypothes.is

A few months ago, AGU introduced a new feature in GEMS – annotating the merged PDF of the manuscript. Senior AGU Pubs staff wrote an Eos Editors’ Vox article about it. AGU has partnered with hypothesis.is, an online annotation tool, so that reviewers can highlight text and insert comments. Editors can then add additional comments before making a decision about the paper. The comments are labeled “reviewer 1,” “Editor,” etc., so that the author can identify which of the assessors made the remark. During the revision process, authors can respond to these comments directly in the annotated PDF.

hypothesis_logo

            I have used it a couple of times and I have seen ~10 reviewers use it over the last few months. I think it works really well, so it is it time to publicize this feature and make the community aware of this powerful resource.

When you agree to review a manuscript, you will see this new section on the review page:

hypothesis_entry_button

It’s just below the link to retrieve the paper and the link for submitting your review. When you click on it, you get a new browser window with the manuscript PDF:

hyothesis-screenshot

This page already has several sections of text highlighted with example comments written. There are controls across the top bar for navigating around the document. When you highlight some text, a small pop-up window appears below it with the word “annotate” in it:

hypothesis_annotate

This opens a text box in the right-hand column in which you can type your comment:

hypothesis_textbox

The “You” at the top indicates the originator of the comment, then the highlighted text is repeated, and then a box for writing your comment, including rich text features like inserting hyperlinks, images, and LaTeX-based equations. Along the bottom of the text box is a row of buttons for specifying the type of remark you are making. Is it an overview comment? Pick “Summary.” Do you want to designate it as a “major” or “minor” concern? Go for it. Are you suggesting a small English usage correction? Then pick “Edit.” Are you suggesting a new reference or two, or commenting on a figure? Click that button, then. Finally, there is a “Confidential?” button that you can click if the remark is just to the Editor and not meant for the author. I promise to look through the comments and read these.

Back on the main reviewer page, you can actually see if there are annotations on the “annotated merged PDF.” It should appear as a new link, “Show Summary Table,” like this:

hypothesis_summary-table-link

When you click on this, all of the comments in the hypothes.is PDF are shown:

hypothesis_summary_table

Nice, huh?

Note that you still need to click the link on the main reviewer page to complete the review:

Review_the_manuscript_button

You should answer the pull-down-menu questions and fill in any comments you want in the review text box. It is helpful if you, at the least include a sentence like, “Please see my detailed comments in the online annotated PDF.” This reminds the Editor to go to the annotated PDF and see your comments there. It is also helpful to include a short paragraph summary of your review there. In fact, you can make your review a hybrid of the two, with major comments in the review text box and specific comments embedded in the annotated PDF.

In addition to the Eos article and this blog, there are also more detailed author instructions, reviewer instructions, and even editor instructions at the AGU website. The hypothes.is website also has a really good tutorial. Also, one caveat: it is an interactive web-based tool, so you have to be online to use it.

Also, this whole thing is optional. You don’t have to use it. So far, I’d say that most reviewers do not use it. But most reviewers could be using it, so please consider it. Many reviews include line-identified comments, and this new feature should be easier than typing the location coordinates into your review.

DEI Pledge

As most readers of this blog probably know, JGR Space Physics is a journal of the American Geophysical Union. AGU strives to be a society for Earth and space scientists across the world, but one country is right there in the name…America. In light of recent comments by the President of the United States, I feel compelled to respond with a post. On Martin Luther King Jr. Day here in America, it is a perfect time to push back against racism and other forms of bigotry. I think that silence makes us complicit and I, for one, detest our president’s position.

DEI-Pledge

            When tackling a problem, a diverse workforce brings together many perspectives and makes for a better solution. The USA prides itself on being the world’s melting pot, accepting immigrants from everywhere. A diverse population has helped to make America “Great.” This is exact what the inscription on the Statue of Liberty promotes. Yet, there has always been an undercurrent of racist, bigoted, prejudiced, and/or sexist attitudes in America, with some of the “already privileged” being skeptical and scared of the rise of the “under-privileged.” These feelings are based on ignorance, though. Each time, the rise of an under-privileged group works out well for America. Scientific research, in particular, loves diversity in the workforce.

As a white male of Norwegian descent, I know that I have led a privileged life. I am sure that, who knows how many times, I have benefited from the racism and sexism of others. While I cannot change my past life trajectory, I can steer the future.

So, I pledge to mindfully apply myself towards being a strong supporter of implementing practices at my work and in my daily life that increase diversity, equity, and inclusion. That last phrase is often shortened as DEI. Perhaps you’ve heard that acronym. It’s a good one to know, and I strongly encourage you to adopt a DEI mindset.

For JGR Space Physics authors and reviewers, one way to do this is to practice the Platinum Rule in your interactions with each other and suggest a diverse set of potential reviewers. In the workplace, it can include identifying and confronting Bro Culture and sexist microaggressions. Little by little, we can do a lot.

AGU is working on this too. It has been a strong position of AGU Presidents, and AGU has a Diversity and Inclusion Task Force working right now to review current policy and recommend changes.

            Keep the gates open. Challenge racism, sexism, and bigotry. Promote diversity.

More On Plain Language Summaries

For over a year now, AGU has been including the option of a Plain Language Summary with manuscript submissions to any of its journals. This can be about as long as a regular Abstract to your paper, but should be written so that those outside of space physics can understand it. From the AGU text requirements page, the definition goes like this:

“The plain language summary should be written for a broad audience. It should be free of jargon, acronyms, equations and any technical information that would be unknown to the general public. The purpose is to explain the study to the public. A good summary should state the general problem, what was done, and the result.”

This description should be ingrained in all of us, not just those submitting papers in the near future but also anyone reviewing a manuscript for JGR Space Physics or another AGU journal. Yes, if you are asked to review a paper and it has a Plain Language Summary, then please read it and comment on its quality. This should be considered as an essential part of the review process, just like assessing the Key Points and keywords that the authors have provided for the paper.

AGU now has more information about these Plain Language Summaries to help you write a good one. For me, this advice about creating a Plain Language Summary comes down to the final bullet point: take the time to do it right. This is not something that you should crank out during the GEMS submission process. That not only will just be an initial draft of what it could be but also won’t be vetted by your coauthors. Their name is on the paper too, and the Plain Language Summary is published with the paper, right below the official Abstract, so you should definitely include your coauthors in its creation. Please do not just change a few words from your regular Abstract, but instead write it from scratch and edit it to make it appealing to a nonspecialist audience.

Here is the nice graphic from that webpage, by @JoannaScience:

Jargon-Barrier

She did a cartoon for one of my Editors’ Vox articles. This graphic above pretty much sums up how space physics Abstracts are understood by non-space-physicists. Our niche of AGU has to work especially hard at communicating our work to the public; learning how to write a good Plain Language Summary is an excellent start.

AGU has put together a page with some really good Plain Language Summaries. Have a look to see the kind of summary that resonates.

For now, this paragraph is optional, and I have been told that roughly 20% of manuscript submissions include a Plain Language Summary. Writing a good Plain Language Summary, however, greatly increases the chances of your paper being highlighted by AGU in some way. AGU HQ staff read every Plain Language Summary for all accepted papers across all AGU journals. If they come across a good one. At 20%, this is about 5 summaries per day. When they come across a really good one, the paper will, at the very least, receive a social media highlight. They might work with the journal Editor that handled the paper to create an Editors’ Highlight for the paper. Or, it might even be the initial nugget of a Research Spotlight or Editors’ Vox article about the paper. The point is that the paper could be elevated to receive a highlight regardless of what the reviewers and editor thought about its highlight worthiness. If you write a good highlight, then your paper will have an increased chance of receiving special highlight attention from AGU.

While I have not seen stats on whether the various highlighting that AGU does for papers results in more citations, I have seen the stats on page views and full-text downloads, and the link is clear and extremely favorable. Traffic towards the paper is typically greatly enhanced with a highlight. So, it is in your best interest to spend some time on the Plain Language Summary.