Best-Serving The Readers

In the same New York Times op-ed article that I cited last time, Margaret Sullivan boils her analysis of reviewer closeness down to this: “How is the reader best served?” Again, it’s a good question to ask with respect to JGR Space Physics.

One of my recent posts gave the stats on just how many reviewers the journal uses in a typical year. It’s a lot! While a few referees reveal their identity during the review process or click the button to be acknowledged in the printed article, most of these reviewers remain anonymous to the author and the rest of the community. I think that there is a good discussion to have about whether anonymity “best serves” the community. Please feel free to comment on this topic below but I will not really address it in this post.

I think that readers are best served by reviewers objectively, thoughtfully, and courteously identifying the good elements of the paper, pointing out the areas that need to be improved, and offering suggestions for how the paper can achieve a recommendation for publication. There are six phrases in that sentence that I would like to highlight:

  • Objectively: make a strong effort to remove personal bias, positive or negative, towards the topic, methodology, or authors in evaluating the study
  • Thoughtfully: we call on you for your expertise, so please be thorough in your analysis of the paper, going into detail when necessary
  • Courteously: please keep all reviews at a professional and respectful level of discourse, being considerate of how the authors will interpret your comments, and never including ad hominem attacks
  • Identifying the good: an often-forgotten part of the review process, including comments on the strong aspects of the paper will help the editor weigh your review relative to others received for this paper
  • Pointing out improvement: this is what we seem to be best at and what most reviews focus on, which is fine with me as long as the other elements are kept in mind as well
  • Offering suggestions: this is very important as it provides a clear target for the authors to know the “bar” for recommendation towards publication

reviewing-quarterly-performance

            JGR Space Physics is different from books in that, being an obscure technical journal, the journal readership is, for the most part, the journal author pool as well. I think that helps ensure good reviews. We want only high-quality studies to be published in the journal, so we are tough on each other, but reviewers also know that someday they will be submitting papers, so we strive for civility in our correspondence.

Reviewers: How close is too close?

The New York Times had an op-ed piece from Margaret Sullivan, the NYT Public Editor, “For Reviewers, How Close Is Too Close?”  The title caught my attention. Alas, she was not talking about scientific peer reviewing, but rather book reviews. Still, it is a good question to ask for JGR Space Physics.

We apply several filtering levels to rule out potential reviewers. Firstly, we never send a paper to people at the same institution. Secondly, if the assigned editor knows of a close working relationship between the author and a potential reviewer, then that person will also not be considered. Thirdly, I will often look at the author lists of recent papers by the corresponding author to get a feeling for close collaborators. It’s not a perfect filter but it’s pretty good and I am not about to institute the NSF rule of every coauthor identifying every collaborator from the last 48 months. Perhaps I should apply the Facebook “close friend” filter, but I don’t.

close-friends-300

            The definition of “close collaborator” is rather subjective. I tend to think of it as “coauthor on the other person’s first-author paper in the last couple of years.” Even this definition has the qualitative words “last couple of years” in it, which could mean two or four or ten in this context. I tend to lean towards the “two year” definition of “couple of years” but other editors can apply this a bit differently.

My definition above doesn’t pass muster with U-M’s promotion letter writer regulators, though. For them, “close” includes the authorship lists in which both people were coauthors on a third person’s paper, or even presentation…for the last ten years. Oof da. That’s strict. I am not going to apply that definition to JGR Space Physics.

As much as I would love to have a drink with all of you and get to know your stories, I don’t (yet!) know everyone in the field. Plus, I cannot (will not!) spend all of my time doing editorial tasks, so I can’t investigate every possible conflict of interest or potential bias. I usually, although not always, honor your requests for potential reviewers to whom I should not send your manuscript. To a large extent, I have to rely on you, the potential reviewer, to self identify potential conflicts of interest, both positive and negative. I want you to be an objective evaluator of the paper; if you do not think that you can do that, then please just recuse yourself from the task. It’s okay to say no to a review request for which you don’t feel like you can be unbiased.

Of course, whenever you say no, providing a few names of other potential reviewers is always helpful to us.

Gender-Neutral Responses

As Mother’s Day approaches, here is some good advice: author responses to referee comments should be devoid of gender-specific pronouns. While the majority of the community is male (the 2013 AGU annual report lists the society membership as two-thirds male), there are a lot of females in our field. Please do not assume that the reviewer of your paper was a man. Please refer to this person as “the referee” or “the reviewer” instead of assigning a gender.

HeSheThey

            I often see this sentence, or something very similar, in the author responses: “The authors thank the referee for his or her useful comments.” I have never big a big fan of the “his or her” cover-the-bases double specification that we often use in English. The substitutes of he/she or (s)he are no better, in my opinion. I find these to be awkward and I wish that there was a good gender-neutral replacement. Doing an internet search on “gender-neutral pronouns” gets you a long list of pages trying to explain the options out there. Apparently, many alternatives have been invented but, from what I can tell, nothing has caught on in popularity.

One option I have seen in books is to switch gender designations every paragraph or chapter. This doesn’t really work in an author response, because the about the only place a gender-specific pronoun is used is in the first and last paragraphs (at most). Switching gender usage from sentence to sentence is very confusing, in my opinion. The document just isn’t long enough for this technique to work properly.

Another option is to use the grammatically incorrect plural pronoun, “The authors thank the referee for their useful comments.” The Grammar Enforcer inside of me winces in pain but at least it is gender neutral. It seems to be the most popular solution as it mirrors how many of us speak.

Another alternative is to rewrite the sentence to avoid the need for a pronoun. That is, replace “his or her” with another descriptor, like “providing,” “making,” or “these.” Like this, “The authors thank the referee for providing useful comments.” I like this solution the best but it requires extra time and effort to reword the sentence in a way that makes sense.

In any case, please stop using just he/him/his in author responses to reviewer comments. Because AGU keeps the identity of the referee hidden to the authors, the gender of this person is unknown. Using the male pronouns is outdated and sexist. Diction matters.

Happy May Day

Happy May Day, everyone. Even though we don’t celebrate it here in the USA until September, today is apparently Labor Day (or, more precisely, International Workers’ Day) in many countries around the world. I am working, catching up on my editorial duties, which I have let slip for a few days. For your day off, though, here is a bit of humor from PhD Comics that was recently forwarded to me about what all of you really mean in those Acknowledgments thank you statements.

Enjoy!

phd040815s

Original post here:  http://www.phdcomics.com/comics.php?f=1792

NSF Geospace Portfolio Review

I am siting here at the Agency Town Hall at the Triennial Earth-Sun Summit meeting in Indianapolis, listening to Janet Kozyra talk about the NSF Geospace Science program. This reminds me to send out a public service announcement about the Geospace Portfolio Review going on right now. The Portfolio Review Committee is accepting comments until May 31, 2015.

Atmospheric and Geospace Sciences (AGS)

nsf-ags

            Delores Knipp and Bill Lotko wrote an editorial for Space Weather explaining this process, including a call for input from the community to the committee.

There are two big themes that the committee is charged to consider and for which they will develop recommendations. The first is the “critical capabilities” needed for enabling science progress on the objectives in the 2013 Solar and Space Physics Decadal Survey. The second is the balance of investments and if any “rebalance” is needed.

Because it is NSF, not only are intellectual merit considerations being considered by the committee but also considerations regarding broader impacts are within scope of the discussion and recommendations. They are also assuming a flat budget, which means suggestions for new or augmented program initiatives will have to come at the expense of other existing programs. Keep that in mind with your comments to the panel; strong justifications must accompany their recommendations. Finally, comments on not only domestic programs and facilities but also international partnerships are solicited and encouraged.

So, again: the Portfolio Review Committee is accepting community input right now, and this input period is only open until May 31, 2015. To echo the title of the editorial now is the time to be heard! Additional information, including committee membership, is located here:

http://www.nsf.gov/geo/ags/geospace-portfolio-review-2015/

How often should you review?

In my last post, I went over some statistics of reviewing for JGR Space Physics. One of those stats is 3.5, the average number of times each manuscript is assessed by a referee. The basic math, then, answering the question of how often a researcher should accept a reviewing assignment is 3 or 4 (including re-reviews) per new manuscript that person submits as corresponding author (not counting resubmissions of the same paper).

review-def

I’d like to suggest that the accounting is a little more complicated than that. For starters, we rarely ask students to serve as referees. If we do, then it is usually in the later years of their PhD and probably (hopefully) one or two requests. Therefore, the student’s advisor should pick up this burden, and their reviewing load should increase according to the number of times their students submit papers as corresponding author.

There are some faculty, though, that can’t or won’t take on that many assignments. I’m in this category. As EiC of JGR Space Physics, I rarely review papers for other journals. According to the expectation I just gave above, I don’t officially review enough to cover my own submissions, let alone my students’ papers. There are other faculty that have administrative roles (e.g., department chair, lab director, or associate dean) that prevent them from doing very many reviewing assignments. Others are overwhelmed with a new course development or a high teaching load, and doing a large number of reviews is difficult. So, while it would be ideal for faculty to pick up the extra workload of reviewing student papers, this isn’t always possible. The rest of the community needs to cover this.

A reservoir of referees that I regularly tap is the senior members of the research community. I really value and appreciate their expertise. Such people could be very active in writing new papers and submitting manuscripts to the journal, but even if they are not, I still send them requests to review. I am glad that so many senior researchers agree to take on a disproportionately large refereeing role.

There are also “the usual suspects” that get asked to review a lot. These are the names that people enter with their manuscript submission, and my anecdotal evidence is that certain names appear frequently for certain topics. When I look in the system, I see that these potential reviewers are often already doing one or just turned one in.

Another type of person that gets a lot of requests are Associate Editors. At JGR Space Physics, we use them as “super reviewers.” That is, we’ll send them a paper for which we’ve already had a large number of declines to review, knowing that they will most likely say yes. We send them the tough papers for arbitration (although not just them; others of you also get such requests). We ask them to do other service tasks for the journal, too, like shepherding special sections and helping to promote the journal. With regard to refereeing, though, they to a lot, and again I am very glad that there are people in the community willing to serve in this role for the journal.

So, think about your “appropriate” workload. If you think we’re not sending you enough manuscripts to review, then let me know. We can fix that!

Reviewer Statistics for 2014

Here are a few factoid statistics about the JGR Space Physics referees for 2014. We had exactly 1000 papers last year that were sent out for review. We send each of these to two reviewers for the initial round. Some papers even get a third reviewer for the first round, if the two reviews warrant it. On each subsequent submission of a manuscript, papers might go to zero, one, or two reviewers, and again another referee might be pulled into the mix during the process. So, given all of this, the total number of reviews conducted for JGR Space Physics in 2014 is…3,495. I’ll let you do the math for the average.

We sent out 7,355 requests to review. That’s a 48% acceptance rate, which I think is very good. Remember, JGR Space Physics has an editorial practice of sending out requests to four people (usually) in order to secure the initial two referees. As people decline, we send out additional requests, keeping the number of requests pending in the two-to-four range until two reviewers agree. Therefore, I am not surprised at the 48% acceptance rate; this number makes perfect sense given our procedures.

Note that there were 221 times when we designated a potential reviewer with the “no response” label. This is given a day or two after the third contact email is sent to you. You can still accept after that, but we usually stop sending you chasers at that point (unless I really want you as a reviewer).

A number that I really like is the accept-to-decline ratio. There were 1666 declines registered in 2014, so this ratio is 2.1. This means that over two-thirds of the recorded responses to our requests were acceptances. You people say yes a lot more than you say no.

As I said in my previous post about outstanding reviewers, we had 1409 people serve as referees for JGR Space Physics in 2014. The average is 2.5, a number that includes “re-reviews” of revised manuscripts. We had 164 people complete 5 or more reviews last year; if you were in this category, then you were one of our workhorse referees and I greatly appreciate your service to the journal. A truly outstanding number is that there are 609 of you that have 100% acceptance rates in 2014, clicking yes every time you were asked (i.e., never declining and never being in the “not needed” category). Awesome job!

The above statistics are part of the reason why I think you are spectacular people. The space physics research community proactively agrees to serve, giving up some of their time in the form of high-quality, objective assessments of other people’s work. Thank you very much for all of your hard work to make JGR Space Physics what it is!

C364e_Kapp-20150422133218

Outstanding Reviewers for JGR Space Physics

Every year, the editors of JGR Space Physics (and every other AGU journal) select a few people from the research community to receive a 2014 Editor’s Citation for Excellence in Scientific Refereeing. We are allowed to select a number of reviewers equal to 1% of the total number of new manuscript submissions that year. JGR Space Physics had 1099 new submissions, so we selected 11 people for special recognition.

It was a tough selection because this is such a small number. We used 1409 different people for reviewing tasks in 2014, so picking just 11 people is nearly impossible. In the end we had a long discussion and a vote. The 11 people that we selected to receive this award were (alphabetical order):

  • Chris Crabtree
  • Nirvikar Dashora
  • Michael Denton
  • Mark Engebretson
  • Lynn Kistler
  • Didier Mourenas
  • David Nunn
  • Ian Richardson
  • Craig Rodger
  • Anatoly Streltsov
  • Bruce Tsurutani

The Editors of JGR Space Physics are very grateful for their service to the journal. There will be a formal announcement in Eos very soon, but we submitted these names a month ago and I wanted to start the recognition process.

As I said, it was a very tough decision and there were many worthy candidates for this honor. I would like to thank all of our 1409 reviewers for their time and effort to maintain the high quality of research included in JGR Space Physics.

Note that we cannot pick Associate Editors for this award, even though one of their primary functions for the journal is to serve as “super reviewers,” doing more than an average share and being arbitrators for difficult papers. We also thank all of our AEs for their dedication and service to the journal and the research community.

Dual Affiliations

This is a clarification for all of you out there that have more than one affiliation.

If you are in need of multiple affiliations on papers, then you probably have noticed that the GEMS system only allows for one primary work address. You can designate a “temporary work address” but you must specify dates for which this other address is your primary contact information, during which GEMS will use this address instead of your primary one for contacting you about any submission or reviewing tasks. This seems awkward for those with multiple affiliations. How, then, does one indicate connection to more than one institution on a JGR Space Physics paper?

The answer is that the work address in GEMS is not used for the affiliation address on a paper, it is only used for contacting you for GEMS business. Thus, you only need one active address and, more specifically in the electronic age, one active email address. In fact, because AGU so rarely mails a hardcopy of anything regarding manuscript reviews and decisions, if you will still have access to your primary work email address while on temporary assignment somewhere else, then there really is little need to switch to the temporary work address in the GEMS system.

For the affiliations on the “in print” version of the paper, Wiley fills in these fields from the manuscript itself. So, if you have multiple affiliations, then please make sure you indicate that in the manuscript author list, as in the graphic below.

MultipleAffiliations

In Word, this is easy. In TeX, it takes a bit more work, but there are examples around to help you get the coding correct, such as the site that I got the graphic.

Spotting Bad Science

Andy Brunning has a chemistry-oriented website, but he had a broader-interest post last year with the “Rough Guide to Spotting Bad Science.” It floated around Facebook for a while last summer.

Spotting-Bad-Science

When I teach 100-level classes here at Michigan, and sometimes when I teach higher-level courses, I often include a day introducing the basics of good versus bad science. As the post linked above says, most people get their science from other people or media reports. My reading of this guide is that it focuses mainly on conducting the science and the write-up in the obscure journal (like JGR Space Physics). It includes just a few points (the first two?) related to the equally important issue of “bad science” at the later steps in the process: the media presentation of the research and the individual’s interpretation and usage of that media report. That topic deserves its own post. Here, we’ll stick to the bad science at the researcher level.

What are some key elements for our field from this list? I’ll highlight a few. Correlation and causation (#4) is sometimes an issue for space science as we look for meaning between quantities that don’t have a plausible physical connection. Sample size too small (#6) is another that can easily get us because we are often limited by the observations available or the computing constraints of running a big code. I hope we are not plagued by “Cherry-picked” results (#10), but the temptation is always there to exclude results that do not support a preferred hypothesis. Probably the biggest for us is Unreplicable results (#11). First off, we rely on nature to conduct the experiments and we get whatever the data we can from whatever observatories are available at that time. Second, and more to the point here, is the issue of “closed” data sets, data processing techniques, and computer code. For others to independently verify the findings and build consensus around a hypothesis, everything that went into that result should be available for others to use.