A comparative study of model-driven approaches for scoping and planning experiments

Context: Through the years researchers have proposed several approaches to foster the growth and quality of experiments in Software Engineering. Among these approaches, there are some initiatives that rely on tool support for specifying controlled experiments. Goal: This paper reports results from a study, which aims to organize, analyze and outline the specifications of each initiative through a comparative analysis. Method: Specifications of each initiative have been compared through a comparative analysis, carried out according to eight criteria: (i) standard empirical concepts, (ii) goals and targets, (iii) involved variables, (iv) subject description, (v) design of experiment, (vi) tasks and activities, (vii) instruments and measurements, and (viii) the threats to research validity. Results: The results show that, among the tools currently existing and used in literature, the eSEE (Experimental Software Engineering Environment) is a complete model. However, it is also the most complex. In the other hand, the most flexible one is Experiment DSL. Conclusion: Based on our results, the currently existing solutions have strengths and weaknesses that should address efforts to make improvements in this area. In principal, our general suggestion is to place emphasis on methodological quality âǍŞ more than on method quantity.