再议论文同行评议的可靠性——来自元分析的新证据

A Further Discussion on the Reliability of Manuscript Peer Review: New Evidence Based on Meta-Analysis

摘要:

论文同行评议的可靠性是科学界长期探讨却又悬而未决的议题。对此，选取目前广泛使用的两种评价者间信度指标，即组内相关系数（ICC）和卡帕系数（κ）来测度论文同行评议的可靠性，并通过元分析方法针对49项发表自1974至2022年间有关论文同行评议场景下评价者间信度的实证研究进行量化归纳，继而开展亚组分析以考察情境和程序两大类共10种因素对论文同行评议可靠性的影响及其测度指标差异，以期拓展对论文同行评议质量的认识与理解。研究结果显示：论文同行评议的可靠性总体上不甚理想（ICC=0.3611，κ=0.1947）；此外，论文同行评议的可靠性受评价对象、学科门类和录用难度（3种情境因素）以及盲法形式（单一程序因素）影响显著，而且当使用不同的测度指标对论文同行评议场景下的评价者间信度加以计量时，同一种因素所表现出来的影响模式可能大相径庭。

Abstract:

The reliability of manuscript peer review (MPR) has long been a concerning issue for the scientific community. In this paper, two widely-used inter-rater reliability indicators, namely the intra-class correlation coefficient (ICC) and Cohen’s kappa coefficient (κ), were selected to measure the reliability of MPR. A meta-analysis was conducted to provide a quantitative summary of 49 empirical studies published between the years 1974 and 2022 focusing on inter-rater reliability in the context of MPR, in order to expand the knowledge and understanding of the quality of MPR. After that, a series of subgroup analyses were carried out to investigate the effects of ten factors belonging to two categories, namely situational and procedural factors, on the reliability of MPR as well as how these effects differ between both indicators. The analytical results show that on the whole, the reliability of MPR is far from satisfactory (ICC=0.361, κ=0.195). More importantly, the reliability of MPR is significantly affected by the evaluation subjects, academic disciplines, acceptance rates (i.e., three situational factors), and blind policies (i.e., one procedural factor). In addition, when different indicators are used to measure the reliability of MPR, the effect pattern exhibited by the same factor can be quite different.