基于solrcloud的分布式科技项目查重系统

A solrcloud-based distributed system checking for duplicate Science technology projects

摘要:

从海量科技项目文件库检测出与待检测科技项目的相似文本,设计能够支持实时响应的可扩展的科技项目查重系统。首先采用分布式计算,对科技项目文件进行预处理,建立全文倒排索引；然后在分布式solrcloud系统上执行相似性计算,采用标题段落语句模型(TPSM)计算待检测科技项目文本与全库科技项目文本的相似度。搜狗实验室约10万篇规模文本集上的测试结果表明：所提出的标题段落语句模型(TPSM)调和平均值比全文向量空间模型(FVSM)高出约15% ,比段落空间模型(PVSM)高出8%左右；在检测性能上,一篇1000字左右的待检测论文,检测时间约为10秒,达到实时检测需求。实验结果同时表明基于solrcloud的分布式系统的扩展性,容错性均能满足实际需求。

Abstract:

To check the duplicated science and technolog projects from the massive documents library, a solrcloud-based distributed system was designed developed, it was scalable could respond in real time. The system first processed projects using distributed computing, established a full-text inverted index, then performed similarity computation on a distributed solrcloud platform using the Title-Paragraph-Sentence model (TPSM). Test on the Sogou laboratory dataset including about 100 thousand documents, the proposed Title-Paragraph-Sentence model (TPSM)’s harmonic mean value was about 15% higher than that of the Full text Vector Dpace Model (FVSM), and was about 8% higher than the paragraph space model (PVSM). In the speed testing on 1000 documents, the system running time was about 10 seconds, satisfying the user’s real-time running needs. The experimental results showed that the solrcloud-based distributed system could well meet the planned requirements.