技术评估中专家一致性评价方法研究综述

A Review of Expert Consistency Evaluation Methods in Technology Assessment

摘要:

为系统把握技术评估中有关专家一致性评价方法，对已有相关研究进行梳理、综合和评价。依据评价问题的类别，将技术评估中专家评价问题分为指标评价和方案排序两类。在指标评价中，单指标下的评价方法主要依据离散程度统计量进行分析，模型简单、计算难度小，解释直观、容易理解，常被应用于细化分析专家群体对评估对象在各个评价指标下评分的一致性程度；多指标下的评价方法则主要依据相关关系、显著性检验等方法进行分析，前者既可用于定量数据也可用于定性数据，已被广泛应用于解决专家整体一致性评价问题，后者主要从专家个体与群体、方案和属性等多角度对专家一致性进行检验，能够快速找出与其他意见有差异的评价信息，计算量较小、可操作性强。在方案排序评价中，基于判断矩阵的评价方法常依据相似性度量、距离测度、主成分分析等方法进行分析，具有考虑不同专家重要性的影响、有效利用专家权重信息，可以解决多种含有不确定信息和模糊信息的专家一致性评价问题，以及更有效地剔除不一致的专家意见、使得评价结果更为合理的优点；基于序数值的评价方法主要运用Kendall协调系数解决序数值类型的专家一致性评价问题，具有实现过程直观简洁、计算简便，鲁棒性强、易于掌握的特点。由于提出背景不同、模型复杂程度以及应用领域范围不同，每种专家一致性评价方法都有其适用性和局限性，而且目前缺乏一个评估框架帮助研究人员依据不同问题选择相适应的专家一致性评价方法，因此，在判断选用哪一种评价方法最为恰当时，应充分考虑进行专家一致性测量的目的。

Abstract:

To systematically summarize the expert consistency evaluation methods in technology assessment, this paper analyzes, summarizes, and evaluates the existing relevant literature. Based on the type of evaluation questions, the questions for expert evaluation in technology assessment are categorized as indicator evaluation and alternative ranking. In the indicator evaluation, the evaluation methods under a single indicator are mainly based on discrete degree statistics, which have the advantages of a simple model, less computational difficulty, intuitive interpretation, and understandability. These methods are often applied to measure the consistency of the expert groups' assessment of the subjects’ values under the various evaluation indicators. The evaluation methods under multiple indicators are mainly based on the correlation and significance test. The former can be used for both quantitative and qualitative data and has been widely used to solve the problem of expert consistency, and the latter mainly assesses the consistency of experts from multiple perspectives, including individuals and groups, alternatives, and attributes, which can quickly find out the opinion that differs from others, and has the advantages of less calculation and greater operability. In the alternative ranking evaluation, the evaluation methods based on judgment matrices are often realized based on similarity measures, distance measures, principal component analysis, etc., which have the advantages of considering the influence of the different experts' importance, effectively applying the experts' weight information, solving various experts consistency issues that contain uncertainty and fuzzy information, and effectively eliminating inconsistent experts to make the evaluation results more reasonable. The evaluation method based on ordinal values mainly uses Kendall's coordination coefficient to solve the expert consistency problem containing ordinal values, which has the characteristics of a simple and intuitive implementation process, convenient calculation, strong robustness, and understandability. Due to the different backgrounds, model complexity, and application areas, each expert consistency evaluation method has its own suitability and limitations. Currently, there lack of an evaluation framework to help researchers choose the appropriate method according to different issues, so, the purpose of expert consistency measures should be fully considered when determining the most appropriate evaluation method to use.