Should the impact factor of the year of publication or the last available one be used when evaluating scientists?

  1. Slafer, Gustavo A. 1
  2. Savin, Roxana 1
  1. 1 University of Lleida-AGROTECNIO Center, Dept. of Crop and Forest Sciences. Av. R. Roure 191, 25198 Lleida
Revista:
Spanish journal of agricultural research

ISSN: 1695-971X,  2171-9292

Año de publicación: 2020

Volumen: 18

Número: 3

Tipo: Artículo

DOI: 10.5424/SJAR/2020183-16399 DIALNET GOOGLE SCHOLAR lock_openDialnet editor

Otras publicaciones en: Spanish journal of agricultural research

Objetivos de desarrollo sostenible

Resumen

Aim of study: A common procedure when evaluating scientists is considering the journal’s quartile of impact factors (within a category), many times considering the quartile in the year of publication instead of the last available ranking. We tested whether the extra work involved in considering the quartiles of each particular year is justifiedArea of study: EuropeMaterial and methods: we retrieved information from all papers published in 2008-2012 by researchers of AGROTECNIO, a centre focused in a range of agri-food subjects. Then, we validated the results observed for AGROTECNIO against five other European independent research centres: Technical University of Madrid (UPM) and the Universities of Nottingham (UK), Copenhagen (Denmark), Helsinki (Finland), and Bologna (Italy).Main results: The relationship between the actual impact of the papers and the impact factor quartile of a journal within its category was not clear, although for evaluations based on recently published papers there might not be much better indicators. We found unnecessary to determine the rank of the journal for the year of publication as the outcome of the evaluation using the last available rank was virtually the same.Research highlights: We confirmed that the journal quality reflects only vaguely the quality of the papers, and reported for the first time evidences that using the journal rank from the particular year that papers were published represents an unnecessary effort and therefore evaluation should be done simply considering the last available rank.

Referencias bibliográficas

  • Abramo G, D'Angelo CA, Di Costa F, 2010. Citations versus journal impact factor as proxy of quality: Could the latter ever be preferable? Scientometrics 84: 821-833. https://doi.org/10.1007/s11192-010-0200-1
  • Abramo G, D'Angelo CA, Felici G, 2019. Predicting publication long-term impact through a combination of early citations and journal impact factor. J Informetrics 1: 32-49. https://doi.org/10.1016/j.joi.2018.11.003
  • Albarrán P, Crespo J, Ortuño I, Ruiz-Castillo J, 2011. The skewness of science in 219 sub-fields and a number of aggregates. Scientometrics 88: 385-397. https://doi.org/10.1007/s11192-011-0407-9
  • Ancaiani A, Anfossi AF, Barbara A, Benedetto S, Blasi B, Carletti V, Cicero T, Ciolfi A, Costa F, Colizza G, et al. 2015. Evaluating scientific research in Italy: The 2004-10 research evaluation exercise. Res Eval 24: 242-255. https://doi.org/10.1093/reseval/rvv008
  • Bartneck C, Kokkelmans S, 2011. Detecting h-index manipulation through self-citation analysis. Scientometrics 87: 85-98. https://doi.org/10.1007/s11192-010-0306-5
  • Baveye PC, Trevors JT, 2011. How can we encourage peer-reviewing? Water Air Soil Pollut 214: 1-3. https://doi.org/10.1007/s11270-010-0355-7
  • Bornmann L, Leydesdorff L, 2017. Skewness of citation impact data and covariates of citation distributions: A large-scale empirical analysis based on Web of Science data. J Informetrics 11: 164-175. https://doi.org/10.1016/j.joi.2016.12.001
  • Bornmann L, Leydesdorff L, Wang J, 2013. Which percentile-based approach should be preferred for calculating normalized citation impact values? An empirical comparison of five approaches including a newly developed citation-rank approach (p100). J Informetrics 7: 933-944. https://doi.org/10.1016/j.joi.2013.09.003
  • Bradford SC, 1934. Sources of information on specific subjects. Engineering 137: 85-86.
  • Butler L, 2002. A list of published papers is no measure of value. Nature 419: 877. https://doi.org/10.1038/419877a
  • Chavarro D, Ràfols I, Tang P, 2018. To what extent is inclusion in the Web of Science an indicator of journal 'quality'? Res Eval 27: 106-118. https://doi.org/10.1093/reseval/rvy001
  • Cooke S, Donaldson M, 2014. Self-citation by researchers: Narcissism or an inevitable outcome of a cohesive and sustained research program? Ideas Ecol Evol 7: 1-2. https://doi.org/10.4033/iee2014.7.1.e
  • Didegah F, Thelwall M, 2013. Determinants of research citation impact in nanoscience and nanotechnology. J Am Soc Inform Sci Technol 64: 1055-1064. https://doi.org/10.1002/asi.22806
  • Egghe L, 2011. A disadvantage of h-type indices for comparing the citation impact of two researchers. Res Eval 20: 341-346. https://doi.org/10.3152/095820211X13164389670356
  • Finardi U, 2013. Correlation between journal impact factor and citation performance: an experimental study. J Informetrics 7: 357-370. https://doi.org/10.1016/j.joi.2012.12.004
  • Fox CW, 2017. Difficulty of recruiting reviewers predicts review scores and editorial decisions at six journals of ecology and evolution. Scientometrics 113: 465-477. https://doi.org/10.1007/s11192-017-2489-5
  • Frank M, 2003. IFs: arbiter of excellence? J Medical Library Assoc 91: 4-6.
  • Hirsch JE, 2005. An index to quantify an individual's scientific research output. P Nat Acad Sci USA 102: 16569-16572. https://doi.org/10.1073/pnas.0507655102
  • Huang DW, 2016. Positive correlation between quality and quantity in academic journals, J Informetrics 10: 329-335. https://doi.org/10.1016/j.joi.2016.02.002
  • Ioannidis JPA, 2015. A generalized view of self-citation: Direct, co-author, collaborative, and coercive induced self-citation. J Psychosomatic Res 78: 7-11. https://doi.org/10.1016/j.jpsychores.2014.11.008
  • Kacem A, Flatt JW, Mayr P, 2020. Traking self-citations in academic publishing. Scientometrics 123: 1157-1165. https://doi.org/10.1007/s11192-020-03413-9
  • Kreiman G, Maunsell JH, 2011. Nine criteria for a measure of scientific output. Front Comput Neurosci 5: 48. https://doi.org/10.3389/fncom.2011.00048
  • Langfeldt L, Bloch C, Sivertsen G, 2015. Options and limitations in measuring the impact of research grants - evidence from Denmark and Norway. Res Eval 24: 256-270. https://doi.org/10.1093/reseval/rvv012
  • Levitt JM, Thelwall M, 2008. Patterns of annual citation of highly cited articles and the prediction of their citation ranking: A comparison across subjects. Scientometrics 77: 41-60. https://doi.org/10.1007/s11192-007-1946-y
  • Leydesdorff L, 2008. Caveats for the use of citation indicators in research and journal evaluations. J Am Soc Inform Sci Technol 59: 278-287. https://doi.org/10.1002/asi.20743
  • Liu XL, Gai SS, Zhang SL, Wang P, 2015. An analysis of peer-reviewed scores and impact factors with different citation time windows: A case study of 28 ophthalmologic journals. PLoS ONE 10 (8): e0135583. https://doi.org/10.1371/journal.pone.0135583
  • Logan JM, Bean SB, Myers AE, 2017. Author contributions to ecological publications: What does it mean to be an author in modern ecological research? PLoS ONE 12 (6): e0179956. https://doi.org/10.1371/journal.pone.0179956
  • Moed HF, 2008. UK research assessment exercises: Informed judgments on research quality or quantity? Scientometrics 74: 153-161. https://doi.org/10.1007/s11192-008-0108-1
  • Mutz R, Daniel HD, 2012. Skewed citation distribution and bias factor: Solutions to two core problems with the journal impact factor. J Informetrics 6: 169-176. https://doi.org/10.1016/j.joi.2011.12.006
  • Owlia P, Vasei M, Goliaei B, Nassiri I, 2011. Normalized impact factor (NIF): An adjusted method for calculating the citation rate of biomedical journals. J Biomed Informatics 44: 216-220. https://doi.org/10.1016/j.jbi.2010.11.002
  • Pajić D, 2015. On the stability of citation-based journal rankings. J Informetrics 9: 990-1006. https://doi.org/10.1016/j.joi.2015.08.005
  • Rajasekaran S, Shan RLP, Finnoff JT, 2014. Honorary authorship: Frequency and associated factors in physical medicine and rehabilitation research articles. Archiv Phys Med Rehabil 95: 418-428. https://doi.org/10.1016/j.apmr.2013.09.024
  • Sahel JA, 2011. Quality versus quantity: Assessing individual research performance. Sci Transl Med 3: 84cm13. https://doi.org/10.1126/scitranslmed.3002249
  • Seglen PO, 1992. The skewness of science. J Am Soc Inform Sci 43: 628-638. https://doi.org/10.1002/(SICI)1097-4571(199210)43:9<628::AID-ASI5>3.0.CO;2-0
  • Seglen PO, 1997. Why the IF of journals should not be used for evaluating research. Brit Med J 314: 497-502. https://doi.org/10.1136/bmj.314.7079.497
  • Slafer GA, 2005. Multiple authorship of crop science papers: are there too many co-authors? Field Crops Res 94: 272-276. https://doi.org/10.1016/j.fcr.2004.11.011
  • Slafer GA, 2008. Should crop scientists consider a journal's impact factor in deciding where to publish? Eur J Agron 29: 208-212. https://doi.org/10.1016/j.eja.2008.07.001
  • Stegehuis C, Litvak N, Waltman L, 2015. Predicting the long-term citation impact of recent publications. J Informetrics 9: 642-657. https://doi.org/10.1016/j.joi.2015.06.005
  • van Raan AFJ, 2013. Universities scale like cities. PLoS ONE 8: e59384. https://doi.org/10.1371/journal.pone.0059384
  • Vanclay JK, 2013. Factors affecting citation rates in environmental science. J Informetrics 7: 265-271. https://doi.org/10.1016/j.joi.2012.11.009
  • Waltman L, 2016. A review of the literature on citation impact indicators. J Informetrics 10: 365-391. https://doi.org/10.1016/j.joi.2016.02.007
Fuente de los datos: Dialnet