Yesterday the JGR Space Physics editors had their quarterly telecon and we talked a bit about the new Journal Impact Factor (JIF) that was just released. We want the journal to be very high quality but we do not want to be metrics manipulators. We agreed to monitor it for the next few years.
The topic of metric reliability is on the minds of many journal editors. Martyn Clark, the Editor in Chief of AGU’s journal Water Resources Research, just published an Editorial entitled, “The citation impact of hydrology journals,” coauthored by Brooks Hanson, AGU’s Director of Publications. It analyzes several metrics for 6 hydrology-related peer-reviewed journals for the past ~20 years. It’s a very nice examination of journal metrics for a geophysics field. I encourage you to read it.
Let me summarize the key findings. They show that all of the journals have the same temporal trend in their metrics, with the JIF steadily rising, in general, for all hydrology journals over the last 15 years. They also see significant variability in the JIF of smaller journals (i.e., those that publish < 200 articles per year) as a few highly-cited papers skew the JIF upwards for a year or two, quantified by resampling the articles to create a uncertainty spread on the metric. All of the journals had Lost Papers with zero citations and Super Papers with >100 citations. They find hydrology papers taking a relatively long time to “mature” and reach full influence on the field, a similar trend as in space physics, as evidenced by most citations occurring after the 2-year window of the JIF (compare their Figures 6 and 7 with a similar plot for JGR Space Physics here). The main finding of the article is that journal metrics, in particular the JIF, are temporally variable, have relatively large spreads of uncertainty, and are not representative of the influence of a specific paper on its research field.
The JIF is reported to 4 significant digits, but this Editorial clearly demonstrates that this level of precision is overkill. Here is a plot of the spread of JIF values for 3 of the journals:
JHM is the smaller of these 3 and the uncertainty in its JIF is > 0.5. The other two journals publish 500-800 articles per year, so their uncertainties are lower, but they are still several tenths of a point.
They bring up a fantastic point that I want to repeat here: citations to a paper do not necessarily measure the quality of the paper, but rather represent the utility of the paper. Citations show that others are building on the findings of the paper but the number of citations does not capture the robustness of the analysis within the paper. I don’t think that we have a good measure for that yet.
If you look at the Acknowledgments, Jennifer Satten at Wiley provided the bibliometrics data for this article. She has given me much of the same information for the field of space physics. I could work up a similar article for our discipline. It’s on my to-do list. Maybe I will, or perhaps I’ll just show some plots in this blog as I make them.