Assessing the effectiveness of large language models in code comment generation

Abstract

Source code comments are essential for ensuring software readability and maintainability, significantly reducing future development effort. However, developers often overlook this task, viewing it as tedious and time-consuming. Large Language Models (LLMs) have been applied to a variety of software engineering tasks, including automatic source code comment generation. Some recent studies have investigated this potential; however, few have systematically evaluated the quality of the documentation produced by these tools. In most cases, evaluations rely exclusively on quantitative metrics, which fail to capture the semantic adequacy and practical usefulness of the generated comments. When qualitative analyses are included, they often employ simplified criteria, overlooking more nuanced aspects of quality. Furthermore, many studies are conducted on small datasets, limiting the generalizability of their findings. Another relevant point is that plenty of the existing work focuses on code summarization rather than structured documentation. This dissertation presents results from a comprehensive evaluation of code documentation generated by three widely used LLMs: OpenAI GPT-3.5 Turbo, GPT-4o, and DeepSeek-V3. Using 415 code snippets extracted from popular GitHub repositories, we generated a total of 1,245 Javadocs. Our methodology combines automated textual similarity metrics (BLEU, ROUGE-L, and METEOR) with human evaluation, and further explores how intrinsic code properties influence the quality of the generated documentation. Results indicate that 58.8% of the generated comments match the quality of the original ones, while 27.7% surpass them. More recent models, such as GPT-4o and DeepSeek-V3, produce notably more complete and informative documentation. Interestingly, although attributes such as code size, complexity, and dependency count are typically associated with poorer code quality, our findings reveal a weak yet positive correlation between these factors and the quality of LLM-generated comments—suggesting that additional contextual information may actually support and enhance the automation of code documentation.