A benchmark-based method to derive metric thresholds

Abstract

With software-intensive systems growing in size and complexity, better support is required for measuring and controlling the software quality. Software metrics are the practical means for assessing different quality attributes and characteristics, such as size, complexity, maintainability, and usability. In spite of that, only the values of metrics are not enough. The effective measurement of software systems is directly dependenton the definition of appropriate thresholds. Thresholds allow to objectively characterize or to classify each component according to one of the software metrics. The definition of appropriate thresholds needs to be tailored to each metric. Aiming to investigate this topic, we first performed a literature review of methods to derive thresholds. In this review, we analyzed the evolution of such methods and realized that researchers and practitioners do not have a consensus about the characteristics ofthese methods. In fact, many methods have been proposed and have been used in the lasts years. After the literature review, we present a detailed comparison of three recently proposed methods to derive metric thresholds (Alvess, Ferreiras, and Oliveiras methods). This comparison highlights the main characteristics of each method and, as lessons learned, we present eight desirable points for this kind of method based on our theoretical and practical knowledge. Trying to fit all desirable points and getting the best of each method, we propose our own method to derive metric thresholds, named Vales method. We explain our method, justifying each of its steps, and develop a tool to support the method, called TDTool. In the total we provide thresholds for 8metrics. To evaluate Vales method, we (i) analyzed the derived thresholds individually and using a metric-based detection strategy, (ii) analyzed the results using two different types of benchmarks, and (iii) provided an overview of the method compared to other methods in the literature. In summary, all the compared methods seem to be fair to derive metric thresholds, but our method fared better in the evaluations.