Complexity Should Not Be in the Eye of the Beholder: How Representative Complexity Measures Respond to the Commonly-Held Beliefs of the Literature

In an increasingly interconnected & cyber-physical world, the ability to coherently measure and manage complexity is vital for the engineering design and systems engineering community. To this end, numerous measures have been promulgated in the literature, yet these measures differ in terms of their intellectual foundations and perspectives, with limited cross-validation among them. In this paper, we propose a framework for benchmarking the status quo of existing complexity measurement approaches in terms of their alignment with the commonly-held beliefs in the literature. We discover that the literature broadly suggests an understanding of complexity based on a system’s size, number of interconnections, and architectural structure. We adopt a design of experiments approach and synthetically create system architectures to mimic the variation across these dimensions. We then use these architectures as a shared test-bed to document the response of four complexity measures that are representative of the predominant perspectives of the literature. We do this by evaluating the change in measurement of a complexity measure as we incrementally varied the levels of one system architecture property believed to affect complexity while keeping the others constant. We find that none of the measures fully satisfy the commonly-held beliefs of the literature and provide a discussion on the underlying factors that lead to these discrepancies. We note that multiple independent discussions coexist in the literature, with little cohesion and communication across the groups, suggesting that further research is required to understand the interactions and influences among these communities. For this purpose, our rigorous, structured, and grounded in literature benchmarking approach can serve as a testbed for development and verification of future architectural assessment tools and measures.