The citations of papers with conflicting reviews and confident reviewers

Jiangen He, Chaomei Chen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Disagreement is essential for knowledge growth in science. However, disagreement in peer review is usually regarded as a sign of unreliability in existing studies. The predictive role of disagreement for potentially impact ful discoveries was rarely explored. Reviewer expertise has been proved to have effects on review strictness and outcomes, but reviewers' added-value on paper quality remains unclear. In this paper, we examined the predictive effects of disagreement and confidence of reviewers on citations of reviewed papers. Using a dataset of 489 papers submitted ICRL 2017, we found that predictive effects of disagreement and confidence on citations. Among accepted papers, the papers with higher review ratings tend to receive more citations and the disagreement cannot predict citations. However, among rejected papers with more than ten citations, the ratings they received cannot predict their citations, but the disagreement level of their reviewers does. Similar opposite findings were also seen regarding confidence. Accepted papers reviewed by confident reviewer tend to have more citations but rejected papers might not benefit from confident reviewers.

Original languageEnglish
Title of host publication17th International Conference on Scientometrics and Informetrics, ISSI 2019 - Proceedings
EditorsGiuseppe Catalano, Cinzia Daraio, Martina Gregori, Henk F. Moed, Giancarlo Ruocco
PublisherInternational Society for Scientometrics and Informetrics
Pages2411-2417
Number of pages7
ISBN (Electronic)9788833811185
Publication statusPublished - 2019 Jan 1
Event17th International Conference on Scientometrics and Informetrics, ISSI 2019 - Rome, Italy
Duration: 2019 Sep 22019 Sep 5

Publication series

Name17th International Conference on Scientometrics and Informetrics, ISSI 2019 - Proceedings
Volume2

Conference

Conference17th International Conference on Scientometrics and Informetrics, ISSI 2019
CountryItaly
CityRome
Period19/9/219/9/5

Fingerprint

Citations
Confidence
Tend
Peer Review
Predict
Review
Expertise

All Science Journal Classification (ASJC) codes

  • Statistics and Probability
  • Computer Science Applications
  • Management Science and Operations Research
  • Applied Mathematics
  • Modelling and Simulation

Cite this

He, J., & Chen, C. (2019). The citations of papers with conflicting reviews and confident reviewers. In G. Catalano, C. Daraio, M. Gregori, H. F. Moed, & G. Ruocco (Eds.), 17th International Conference on Scientometrics and Informetrics, ISSI 2019 - Proceedings (pp. 2411-2417). (17th International Conference on Scientometrics and Informetrics, ISSI 2019 - Proceedings; Vol. 2). International Society for Scientometrics and Informetrics.
He, Jiangen ; Chen, Chaomei. / The citations of papers with conflicting reviews and confident reviewers. 17th International Conference on Scientometrics and Informetrics, ISSI 2019 - Proceedings. editor / Giuseppe Catalano ; Cinzia Daraio ; Martina Gregori ; Henk F. Moed ; Giancarlo Ruocco. International Society for Scientometrics and Informetrics, 2019. pp. 2411-2417 (17th International Conference on Scientometrics and Informetrics, ISSI 2019 - Proceedings).
@inproceedings{13e965fc4e344b0dad0bd77f1e852205,
title = "The citations of papers with conflicting reviews and confident reviewers",
abstract = "Disagreement is essential for knowledge growth in science. However, disagreement in peer review is usually regarded as a sign of unreliability in existing studies. The predictive role of disagreement for potentially impact ful discoveries was rarely explored. Reviewer expertise has been proved to have effects on review strictness and outcomes, but reviewers' added-value on paper quality remains unclear. In this paper, we examined the predictive effects of disagreement and confidence of reviewers on citations of reviewed papers. Using a dataset of 489 papers submitted ICRL 2017, we found that predictive effects of disagreement and confidence on citations. Among accepted papers, the papers with higher review ratings tend to receive more citations and the disagreement cannot predict citations. However, among rejected papers with more than ten citations, the ratings they received cannot predict their citations, but the disagreement level of their reviewers does. Similar opposite findings were also seen regarding confidence. Accepted papers reviewed by confident reviewer tend to have more citations but rejected papers might not benefit from confident reviewers.",
author = "Jiangen He and Chaomei Chen",
year = "2019",
month = "1",
day = "1",
language = "English",
series = "17th International Conference on Scientometrics and Informetrics, ISSI 2019 - Proceedings",
publisher = "International Society for Scientometrics and Informetrics",
pages = "2411--2417",
editor = "Giuseppe Catalano and Cinzia Daraio and Martina Gregori and Moed, {Henk F.} and Giancarlo Ruocco",
booktitle = "17th International Conference on Scientometrics and Informetrics, ISSI 2019 - Proceedings",

}

He, J & Chen, C 2019, The citations of papers with conflicting reviews and confident reviewers. in G Catalano, C Daraio, M Gregori, HF Moed & G Ruocco (eds), 17th International Conference on Scientometrics and Informetrics, ISSI 2019 - Proceedings. 17th International Conference on Scientometrics and Informetrics, ISSI 2019 - Proceedings, vol. 2, International Society for Scientometrics and Informetrics, pp. 2411-2417, 17th International Conference on Scientometrics and Informetrics, ISSI 2019, Rome, Italy, 19/9/2.

The citations of papers with conflicting reviews and confident reviewers. / He, Jiangen; Chen, Chaomei.

17th International Conference on Scientometrics and Informetrics, ISSI 2019 - Proceedings. ed. / Giuseppe Catalano; Cinzia Daraio; Martina Gregori; Henk F. Moed; Giancarlo Ruocco. International Society for Scientometrics and Informetrics, 2019. p. 2411-2417 (17th International Conference on Scientometrics and Informetrics, ISSI 2019 - Proceedings; Vol. 2).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - The citations of papers with conflicting reviews and confident reviewers

AU - He, Jiangen

AU - Chen, Chaomei

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Disagreement is essential for knowledge growth in science. However, disagreement in peer review is usually regarded as a sign of unreliability in existing studies. The predictive role of disagreement for potentially impact ful discoveries was rarely explored. Reviewer expertise has been proved to have effects on review strictness and outcomes, but reviewers' added-value on paper quality remains unclear. In this paper, we examined the predictive effects of disagreement and confidence of reviewers on citations of reviewed papers. Using a dataset of 489 papers submitted ICRL 2017, we found that predictive effects of disagreement and confidence on citations. Among accepted papers, the papers with higher review ratings tend to receive more citations and the disagreement cannot predict citations. However, among rejected papers with more than ten citations, the ratings they received cannot predict their citations, but the disagreement level of their reviewers does. Similar opposite findings were also seen regarding confidence. Accepted papers reviewed by confident reviewer tend to have more citations but rejected papers might not benefit from confident reviewers.

AB - Disagreement is essential for knowledge growth in science. However, disagreement in peer review is usually regarded as a sign of unreliability in existing studies. The predictive role of disagreement for potentially impact ful discoveries was rarely explored. Reviewer expertise has been proved to have effects on review strictness and outcomes, but reviewers' added-value on paper quality remains unclear. In this paper, we examined the predictive effects of disagreement and confidence of reviewers on citations of reviewed papers. Using a dataset of 489 papers submitted ICRL 2017, we found that predictive effects of disagreement and confidence on citations. Among accepted papers, the papers with higher review ratings tend to receive more citations and the disagreement cannot predict citations. However, among rejected papers with more than ten citations, the ratings they received cannot predict their citations, but the disagreement level of their reviewers does. Similar opposite findings were also seen regarding confidence. Accepted papers reviewed by confident reviewer tend to have more citations but rejected papers might not benefit from confident reviewers.

UR - http://www.scopus.com/inward/record.url?scp=85073877844&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85073877844&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85073877844

T3 - 17th International Conference on Scientometrics and Informetrics, ISSI 2019 - Proceedings

SP - 2411

EP - 2417

BT - 17th International Conference on Scientometrics and Informetrics, ISSI 2019 - Proceedings

A2 - Catalano, Giuseppe

A2 - Daraio, Cinzia

A2 - Gregori, Martina

A2 - Moed, Henk F.

A2 - Ruocco, Giancarlo

PB - International Society for Scientometrics and Informetrics

ER -

He J, Chen C. The citations of papers with conflicting reviews and confident reviewers. In Catalano G, Daraio C, Gregori M, Moed HF, Ruocco G, editors, 17th International Conference on Scientometrics and Informetrics, ISSI 2019 - Proceedings. International Society for Scientometrics and Informetrics. 2019. p. 2411-2417. (17th International Conference on Scientometrics and Informetrics, ISSI 2019 - Proceedings).