Detecção de réplicas de sítios Web em máquinas de busca usando aprendizado de máquina

Abstract

Earlier work estimate that at least 30% of all content available on the Web is replicated content, posing serious challenges to search engines, such as waste of computational resources and decrease in the search effectiveness. Thus, detection of replicated websites is currently a prerequisite for the development of modern search engines. However, the large amount of dynamic content and the huge amount of potential replicas make the detection of replicated website an extremely difficult task. This work focus on the development of new techniques for detecting replicated websites. Intrinsic features of replicated sites were studied and, depending on the value assumed by these features, we can discriminate pairs of replicated and nonreplicated sites. However, (i) there is great difficulty in finding the optimal values that lead to the best discriminative performance, and (ii) there is a lot of features to becombined in order to obtain the best discrimination. Therefore, we evaluated the use of machine learning techniques for detecting replicated websites and we propose an algorithm called DREAM (Detecção de RÉplicas usando Aprendizado de Máquina), which automatically combines and weights the proposed discriminative features.The algorithm DREAM was evaluated through the use of a real collection of sites, collected from the Brazilian Web. Four machine learning algorithms were evaluated: decision trees (C4.5), lazy associative classification (LAC), combinations of trees (random forests) and support vector machine (SVM). Experimental results show that the algorithm DREAM outperforms by 35.1% the best method in the literature in terms of F1 measure. The experimental results indicate that the use of machinelearning techniques lead to superior results comparing to the known algorithms in the literature.