Outstanding Reviewers for 2017

Every year, AGU asks the editors of each journal to come up with a list of recipients for the reviewing excellence award. We had 1124 manuscripts submitted to JGR Space Physics in 2017, so the editors could choose up to 11 people to receive this award. This is a nearly impossible task, as we had 1,448 different people serve as reviewers for the journal in 2017. To further complicate it, 11 does not divide 5 ways, so it is not an even split among the editors.

2017RefereeingExcellence

            So, first let me say thank you to all of the 1,448 scientists that provided one or more reviews for JGR Space Physics last year. Every single one of you is vitally important to making this journal what it is. Your name is in print in our thank you editorial, which appeared in the June 2018 issue. The journal could not exist without the collective input of so many members of the research community.

There was an Eos article, also in June, listing the 2017 reviewing excellence award winners. I sometimes remember to write a blog post about these awardees, but I also forget to do this on other years. We select these awardees in March, the decision is a groupwide vote after we all suggest 2 to 5 names, but I have to wait until after the Eos article comes out before I announce anything here. This year, I remembered! So, here it is.

            The 2017 awardees for JGR Space Physics are as follows:

  • Maciej Bzowski, Space Research Center, Polish Academy of Sciences
  • Pascal Demoulin, Observatoire de Paris
  • Robyn Fiori, Geomagnetic Laboratory, Natural Resources Canada
  • Michael Gedalin, Ben Gurion University of the Negev
  • James Hecht, The Aerospace Corporation
  • Erin Lay, Los Alamos National Laboratory
  • Noé Lugaz, University of New Hampshire
  • Robert Marshall, University of Colorado Boulder
  • Evgeny Panov, Space Research Institute, Austrian Academy of Sciences
  • Jack Scudder, University of Iowa
  • Paul Withers, Boston University
Advertisements

Unconscious Bias in Space Physics

I attended the Triennial Earth-Sun Summit meeting a couple of weeks ago, and there was a very good plenary session on unconscious bias in space physics. The presenters were the authors of the Clancy et al. paper in JGR Planets on bias in astronomy and planetary science. They summarized the findings of that paper, which quantified the extent of women and minorities reporting feeling unsafe or encountering a hostile work environment in these science fields. The numbers are not encouraging, with 80% of women experiencing some kind of sexist remark and two-thirds of women-of-color hearing racist remarks in the workplace. Furthermore, over a quarter of women have felt unsafe in their current position because of their gender or race. This is disturbing to me that the numbers are so large in 2018.

Unconcious Bias Plenary Handout title

            Fortunately, the conversation is not ending with the TESS plenary session. The organizers created a handout that was available to everyone at the session and online with the session description. I highly encourage everyone to read this tri-fold pamphlet. They encourage people to take the Harvard implicit bias test and read through the materials at the U of Arizona’s StepUp! by-stander intervention program. The sheet is filled with tips on how to identify and minimize implicit bias. Two of the biggest things that individuals can do immediately: amplify minority voices is group discussions (but don’t he-peat) and avoid making sexual remarks in the work environment.

As for JGR Space Physics, fighting implicit bias can be done in several ways. The first is to be cordial in your correspondence, especially to early career researchers like graduate students, and to apply the Platinum Rule in your interactions with others, thinking about how they want to be treated and considering the interaction from their perspective. Authors, please use gender-neutral pronouns in responses to anonymous reviewers. Reviewers, consider using one of the links in the handout for quantifying gender bias in writing. Finally, I hope that you all make a personal DEI pledge to promote diversity, equity, and inclusion. People leave the field because of sexism in the workplace, and for our discipline, the workplace includes manuscript correspondence. I occasionally hear from advisors whose students have had a bad interaction with a reviewer.

Thanks to the TESS meeting and session organizers and for coordinating this panel discussion. Let’s continue to strive to do better to reduce implicit bias in space physics.

Length of Your Review

A question posed to me recently asked about the best length for a manuscript review. Let me quote from the email:

“On one side of the spectrum is a group of people that will only comment if something is blatantly wrong. On the other side of the spectrum is the reviewer that will give a lengthy response including all of the changes that could improve the study (or at least as improve the study in the eyes of the reviewer).”

Neither of these extremes is optimal. A very short, highly negative review is particularly bad but close behind is a very short, highly positive review. The more details I have, the better that I can assess the manuscript and weigh the recommendations from the two reviewers. So, in general, I like longer reviews. Reviewers can, actually, go too far, suggesting additional studies and analysis that might be very good to conduct but are not necessary for publication of the submitted manuscript. To me, an optimal review includes praise of the good parts of the paper, identification of what is wrong or unclear, and suggestions for what will make the study publishable.

As I said in one of my first posts on this blog, please be thorough – I like the longer reviews better than the short ones. If you need to write several paragraphs to explain why some aspect of the study is off base, then please do it. That helps me make the right decision about the manuscript.

The main place that I find reviewers being too verbose is with suggested new work. If you write two paragraphs on how the authors should really include another section, then please stop and ask yourself: is this new section that I am asking for necessary to make the submitted manuscript acceptable for publication? If not, then your two paragraphs are really a suggestion for future work. Such suggestions are fine but it should be noted as such. If not noted, then Editors and authors assume that it is a recommendation for acceptability, and authors should either do it or write a thorough rebuttal as to why they didn’t do it.

At the AGU reviewer resources page, the text begins with the main goal of peer review:

“AGU Publications relies on our reviewers to help ensure the standards, quality, and significance of our papers.”

Ensuring the standards, quality, and significance of papers in AGU journals is, I think, best served by identifying those elements of the manuscript that make it fall short of being a “significant contribution to the field.” A terse report, positive or negative, is less than fully helpful to me as an Editor. Similarly, it is useful to expand, at least somewhat, on the positive elements of the manuscript that raise it to the level of acceptability.

Down the page a bit is this definition of a good review:

“In general, the most helpful review is one which first provides an overall summary of the main contribution of the paper and its appropriateness for the journal and summarizes what major items should be addressed in revision.”

I agree. A very brief summary of the paper is always appreciated, not only for the Editor but also for the authors, so that they know that you understood the main point of the paper. Positive comments on the appropriateness for the journal are good, too, especially if the other reviewer finds fault with the same aspect of the manuscript. While many reviews include these elements, a large percentage of reviews do not, instead jumping in to the negative comments. Having these two other elements, a neutral paragraph reiterating the main point of the paper and a positive paragraph highlighting what is good about it, also helps to set a cordial tone to the report, which is always a good thing, in my view.

I like the figure in the Eos article about writing a good review, linked on this page.

peerreviewguide-flowchart

It highlights that the first things a reviewer should write are these two elements, the summary and the positive aspects of the paper. Then move on to the concerns and suggestions.

This page also has a link to another page with the lists of questions and pull-down menus that reviewers will be asked as part of the submission of their report. This is not the full list of all things to consider in your review, but these are the questions that AGU would like to see answered about every manuscript sent out for peer review. Explanations of the answers to these questions should also be in the main text of the review.

So, longer reviews are, in general, better for the Editor. Just take this advice with some caution, however, and think twice about suggestions for new work. Make it clear in the report which suggestions are, in your view, simply possibilities for future investigations and which are recommendations for acceptability.

Preprint Servers: Challenges

A third (and probably final, for now) post on of ESSOAr, AGU’s new preprint server for Earth and space sciences. The first described it, the second touted it, and now this one is the ethical scold of how best to use it.

The biggest point to remember is that preprint servers are not peer-reviewed journals. Yes, there is an editorial board that checks submissions for scientific scope, but there is no vetting of the accuracy of the content. The editorial check takes a day or two, maybe a week max, but it is not a real review process. Yes, content here gets a DOI, but we should all remember that content on preprint servers are essentially just a step above “private communication” in terms of referencing authority. That is, it could be wrong.

We hope that content on ESSOAr, and any other preprint server, will eventually be published in a scientific journal. Researchers are putting their reputation out there with each new post on one of these servers, so the content is, for the most part, respectable. Go ahead and use it to learn what is being done by your colleagues. Because preprint server content has not been through the peer review process, though, it should be replaced with the “final” version of the study from whatever journal it eventually appears in.

To summarize in a graphic:

Caution-preprints

            Peer review should still be the standard for what is accepted as “knowledge” of the subject. Even this can be wrong but at least it has been thoroughly scrutinized by experts. You should be very skeptical of older preprints on the server (say, more than 2 years since original posting) that lack a link to a final published version of the paper. That work either was not submitted or did not pass peer review. If the former, then it is perhaps the case that the authors found a problem with the study and therefore never submitted that version of the paper. If the latter, then perhaps the editor or referees found a problem with the study and declined publication of it. Either way, the study did not reach its “final” form in the literature.

The advice to the community about older preprints can be summed up like this:

  • Authors: use caution when citing an older preprint.
  • Reviewers: pay extra attention to citations of older preprints.
  • Editors: ask reviewers to check the appropriateness of older preprint citations.
  • Societies: set policy about citing older preprints.

I am told that the astrophysics community, which regularly uses the arXiv preprint server, understands this difference in “publication” levels. That is, research communities can learn to use preprint servers and make it their go-to place for the latest content across a number of journals, as I am told that many in astrophysics do. They also know, however, that when it comes time to write your own paper, don’t rely on preprints as your main entries in the reference list. The astrophysics community, I am told, understands the guidelines about preprint servers and only uses it for finding the latest work on a topic.

We, the Earth and space science research community, should adopt this same mentality about preprint servers, not only ESSOAr but any server (and there are several being created). Such servers should be a place to get the latest studies from across a variety of journals, learning about content as the manuscripts are submitted rather than months later when they are accepted and eventually published. We should only use it for the latest work, though. A preprint server is not the place for full literature searches – those should be done in Web of Science, Google Scholar, Scopus, ADS, or other services that scan the published, peer-reviewed literature. And, as an editor, I strongly urge you to please conduct a full literature search, because a recent study by Mark Moldwin and me showed that the more complete your reference is, the more citations your paper will get (on average).

Use ESSOAr, but know its purpose within the hierarchy of scientific publications.

Annotating Manuscripts with Hypothes.is

A few months ago, AGU introduced a new feature in GEMS – annotating the merged PDF of the manuscript. Senior AGU Pubs staff wrote an Eos Editors’ Vox article about it. AGU has partnered with hypothesis.is, an online annotation tool, so that reviewers can highlight text and insert comments. Editors can then add additional comments before making a decision about the paper. The comments are labeled “reviewer 1,” “Editor,” etc., so that the author can identify which of the assessors made the remark. During the revision process, authors can respond to these comments directly in the annotated PDF.

hypothesis_logo

            I have used it a couple of times and I have seen ~10 reviewers use it over the last few months. I think it works really well, so it is it time to publicize this feature and make the community aware of this powerful resource.

When you agree to review a manuscript, you will see this new section on the review page:

hypothesis_entry_button

It’s just below the link to retrieve the paper and the link for submitting your review. When you click on it, you get a new browser window with the manuscript PDF:

hyothesis-screenshot

This page already has several sections of text highlighted with example comments written. There are controls across the top bar for navigating around the document. When you highlight some text, a small pop-up window appears below it with the word “annotate” in it:

hypothesis_annotate

This opens a text box in the right-hand column in which you can type your comment:

hypothesis_textbox

The “You” at the top indicates the originator of the comment, then the highlighted text is repeated, and then a box for writing your comment, including rich text features like inserting hyperlinks, images, and LaTeX-based equations. Along the bottom of the text box is a row of buttons for specifying the type of remark you are making. Is it an overview comment? Pick “Summary.” Do you want to designate it as a “major” or “minor” concern? Go for it. Are you suggesting a small English usage correction? Then pick “Edit.” Are you suggesting a new reference or two, or commenting on a figure? Click that button, then. Finally, there is a “Confidential?” button that you can click if the remark is just to the Editor and not meant for the author. I promise to look through the comments and read these.

Back on the main reviewer page, you can actually see if there are annotations on the “annotated merged PDF.” It should appear as a new link, “Show Summary Table,” like this:

hypothesis_summary-table-link

When you click on this, all of the comments in the hypothes.is PDF are shown:

hypothesis_summary_table

Nice, huh?

Note that you still need to click the link on the main reviewer page to complete the review:

Review_the_manuscript_button

You should answer the pull-down-menu questions and fill in any comments you want in the review text box. It is helpful if you, at the least include a sentence like, “Please see my detailed comments in the online annotated PDF.” This reminds the Editor to go to the annotated PDF and see your comments there. It is also helpful to include a short paragraph summary of your review there. In fact, you can make your review a hybrid of the two, with major comments in the review text box and specific comments embedded in the annotated PDF.

In addition to the Eos article and this blog, there are also more detailed author instructions, reviewer instructions, and even editor instructions at the AGU website. The hypothes.is website also has a really good tutorial. Also, one caveat: it is an interactive web-based tool, so you have to be online to use it.

Also, this whole thing is optional. You don’t have to use it. So far, I’d say that most reviewers do not use it. But most reviewers could be using it, so please consider it. Many reviews include line-identified comments, and this new feature should be easier than typing the location coordinates into your review.