Abordagens para avaliação automática de conferências científicas: um estudo de caso em ciência da computação

Abstract

Assessing the quality of scientific conferences is an important and useful service that can be providedby digital libraries and similar systems, mainly for dynamic fields such as Computer Science and ElectricEngineering. However, the majority of the existing approaches has been proposed for measuring the quality of journals. In this MSc dissertation we propose two distinct approaches to automatically assess the quality of conferences. In the first one, we depart from a deep analysis of the deficiencies of citation-based metrics to assess the quality of journals and propose a new set of quality metrics specially designed to capture intrinsic and important aspects related to conferences such as longevity, popularity, prestige, and periodicity. To demonstrate the effectiveness of our proposed metrics, we have conducted two sets of experiments. In the first one, our metrics were used to rank a set of Computer Science conferences and the results were contrasted against a 'gold standard' produced by a large group of specialists. Then, we used our metrics to classify these conferences with respect to some pre-established quality levels, also according to the gold standard. Our metrics obtained gains up to 8.4% in ranking similarity and 7.8% in classification accuracy when compared to standard journal quality metrics.In the second approach, we characterize a large number of features (e.g., citations, tradition, submission and acceptance rates, reputation of the program committee members) that can be used as criteria to assess the quality of scientific conference and study how these features can be automatically combined using machine learning techniques to effectively perform this task. Among our several findings, we can cite that: (1) separating high quality conferences from medium and low quality ones can be performed quite effectively, but separating the last two types is a much harder task; and (2) citation features followed by those associated with the tradition of the conference are the most important ones for the task.Thus, in summary, the major contributions of this MSc dissertation are: (i) a study about the relative performance of existing journal metrics in assessing the quality of scientific conferences; (ii) the proposal of a set of new metrics based on bibliographic citations specifically designed to evaluate the conference, which capture intrinsic and important aspects related to conferences that are not considered by existing metrics (for journals); (iii) the characterization of a large number of features that can be used as criteria to assess the quality of scientific conferences; (iv) a study of how these several features can be combined by means of machine learning techniques to automatically and effectively classify conferences; and (v) a deep analysis and discussion about the relative difficulty of the problem.