Obtaining ground-truth software architectures

TLDR: This paper's experience in recovering the ground-truth architectures of four open-source systems is presented, the primary insights gained in the process are discussed, the characteristics of the obtained ground- Truth architectures are analyzed, and the involvement of the systems' engineers is reflected in a limited but critical fashion.

Abstract: Undocumented evolution of a software system and its underlying architecture drives the need for the architecture's recovery from the system's implementation-level artifacts. While a number of recovery techniques have been proposed, they suffer from known inaccuracies. Furthermore, these techniques are difficult to evaluate due to a lack of “ground-truth” architectures that are known to be accurate. To address this problem, we argue for establishing a suite of ground-truth architectures, using a recovery framework proposed in our recent work. This framework considers domain-, application-, and context-specific information about a system, and addresses an inherent obstacle in establishing a ground-truth architecture - the limited availability of engineers who are closely familiar with the system in question. In this paper, we present our experience in recovering the ground-truth architectures of four open-source systems. We discuss the primary insights gained in the process, analyze the characteristics of the obtained ground-truth architectures, and reflect on the involvement of the systems' engineers in a limited but critical fashion. Our findings suggest the practical feasibility of obtaining ground-truth architectures for large systems and encourage future efforts directed at establishing a large scale repository of such architectures.