3DR-Indexing: um método para identificação automática dos melhores atributos de indexação em deduplicação de dados

Abstract

Data deduplication aims to find and remove duplicate records in databases. Duplicate records are data instances that represent the same object in the real world. Usually, the deduplication has three steps: indexing (which assigns a block key value for each record), clustering (which groups the records with similar block key) and classification (which compares the records within the same group). Our study focuses on the indexing step, which creates block key structures to group similar records. Indexing must be effective (as to better distinguish the values) and efficient (as to allow faster deduplication runtime). Thus, in this step, an attribute is chosen and its value is encoded by a function to produce the block key value. Currently, the indexing attributes are chosen by expert users, which takes time and increases the process total cost. Therefore, we present the method 3DR-Indexing, which automatically selects the best attributes for the indexing step. Furthermore, we analyze the impact of the indexing attribute over data deduplication steps. Finally, we evaluate the indexing attribute and the proposed method over 13 distinct datasets, that is, with different domains, number of duplicate records and the total of instances. Our results indicate the indexing attribute has highest impact over deduplication process. For instance, the best indexing attribute differs from the worst one by an average of 44% in terms of F-Measure (considering all datasets). Moreover, the 3DR Indexing has significant results, because it identifies the best indexing attribute in 10 out of 13 datasets.