Evidence Standards Improve the Reliability of Scholarly Peer Review - Gazette

Thursday, Aug. 17, 10-11:30 a.m.

BN-4031

Please join the Faculty of Business Administration for our next research “Munch and Learn” on August 17 from 10:00-11:30am in BN4031. Dr. Paul Ralph will present his paper “Evidence Standards Improve the Reliability of Scholarly Peer Review.”

Please RSVP to mgulliver@mun.ca by August 16.

Evidence Standards Improve the Reliability of Scholarly Peer Review 

Background. Scholarly peer review is “the lynchpin about which the whole business of science is pivoted” (Ziman 1968). Most researchers believe peer review is effective (Ware 2008), but empirical research consistently shows that reviewers cannot reliable distinguish methodologically sound from fundamentally flawed studies (Cole 1981; Peters & Ceci 1982; Lock 1991; Rothwell and Martyn 2000; Price 2014; Ralph 2016). Consequently, Ralph et al. (2020) created comprehensive evidence standards and tools to improve peer review in software engineering and related fields.

Objective. The objective of this study is to investigate the impact of evidence standards on scholarly peer review.

Method. A randomized controlled experiment was conducted at an A-ranked software engineering conference. The program committee was randomly divided into two groups: one using a typical conference review process; the other using a standardized process based on the ACM SIGSOFT Empirical Standards for Software Engineering Research.

Results. Evidence standards significantly improve inter-reviewer reliability without harming authors’ or reviewers’ attitudes toward the review process.

Discussion. Asking reviewers to write qualitative comments about a paper and score it on a 6-point scale from strong reject to strong accept produces data statistically indistinguishable from random noise. This means that decisions are determined entirely by reviewer selection, not the merits of the research. Conventional review processes are therefore scientifically and morally indefensible. While evidence standards are not a silver bullet, standards-based review significantly improves reliability, and the data collected in this study facilitates further refinement of the standards and tooling toward still greater reliability. Adjacent research communities such as management information systems, management science, human-computer interaction and CS education should investigate adapting evidence standards to improve their peer review processes.

Bio. Paul Ralph, PhD (British Columbia), is an award-winning scientist, author, consultant, and Professor of Software Engineering at Dalhousie University. Dr. Ralph’s research intersects software engineering, sustainable development, human-computer interaction, and project management. He has published more than 80 peer-reviewed articles in premier venues including IEEE Transactions on Software Engineering and the ACM/IEEE International Conference on Software Engineering. Dr. Ralph is editor-in-chief of the SIGSOFT Empirical Standards for Software Engineering Research. In 2022 he was the #1 ranked computer scientist in Atlantic Canada according to csrankings.org.

Presented by Faculty of Business Administration

Adblock test (Why?)

https://news.google.com/rss/articles/CBMiYmh0dHBzOi8vZ2F6ZXR0ZS5tdW4uY2EvZXZlbnRzL2V2aWRlbmNlLXN0YW5kYXJkcy1pbXByb3ZlLXRoZS1yZWxpYWJpbGl0eS1vZi1zY2hvbGFybHktcGVlci1yZXZpZXcv0gEA?oc=5

2023-08-14 00:42:14Z
CBMiYmh0dHBzOi8vZ2F6ZXR0ZS5tdW4uY2EvZXZlbnRzL2V2aWRlbmNlLXN0YW5kYXJkcy1pbXByb3ZlLXRoZS1yZWxpYWJpbGl0eS1vZi1zY2hvbGFybHktcGVlci1yZXZpZXcv0gEA