Do Search-Based Approaches Improve the Design of Self-Adaptive Systems ? A Controlled Experiment

 

Summary

Endowing software systems with self-adaptation capabilities has shown to be quite effective in coping with uncertain and dynamic operational environments as well as managing the complexity generated by non-functional requirements. Nowadays, a large number of approaches tackle the issue of enabling self-adaptive behavior from different perspectives and under diverse assumptions, making it harder for architects to make judicious decisions about design alternatives and quality attributes trade-offs. It has currently been claimed that search-based software design approaches may improve the quality of resulting artifacts and the productivity of design processes, as a consequence of promoting a more comprehensive and systematic design knowledge representation and preventing design bias and false intuition. To the best of our knowledge, no controlled experiments have been performed to provide sound evidence of such claim in the self-adaptive systems domain. In this paper, we report the results of a controlled experiment performed with 24 students of a graduate program in Distributed and Ubiquitous Computing. The experiment evaluated the design of self-adaptive systems using a search-based approach, in contrast to the use of a style-based non-automated approach. The results show that search-based approaches can improve the effectiveness of resulting architectures and reduce design complexity. We found no evidence regarding the method's potential for leveraging the acquisition of distilled design knowledge by novice software architects.

Experiment Goal

The experiment goal was to analyze the design of self-adaptive systems, for the purpose of evaluating the search-based design approach we propose and a design process based on architectural styles catalogs, with respect to the effectiveness and complexity of resulting architectures, as well as the method's potential for leveraging the acquisition of distilled design knowledge by novice self-adaptive systems architects, from the viewpoint of researchers, and in the context of graduate students endowing systems with self-adaptation capabilities.

Research Method

The experiment is characterized as a blocked subject-object study with a paired comparison design. Two UML models representing a web server and a MapReduce distributed architecture are used as experiment objects and two treatments (search-based approach and style-based approach) are considered for the design method factor (independent variable). We are interested in evaluating the impact of the adopted design method on three dependent variables: the effectiveness and complexity of resulting architectures, as well as the method's potential for leveraging the acquisition of distilled design knowledge by novice architects. We use the Generational Distance metric to assess effectiveness in terms of how far the architectures designed by the subjects are from a previously determined Pareto-optimal set of architectures. Design complexity is evaluated by using the Component Point approach while questionnaires with multiple choice questions evaluate the method's potential for leveraging the acquisition of design knowledge.

Experiment Data

• Final report
• Data analysis spreadsheet
• Lectures
• Architectural Styles Catalog for Self-Adaptive Systems - v0