A theme I have returned to several times over the years has been how to improve the reproducibility (or replicability) of science; in particular, the software models used to analyse or model data.
Processes for developing software models
Software models, or, as cognitive scientists like to use, computational architectures, are computer programs developed to simulate some behaviour. One of the challenges in reproducing results obtained by a model is in understanding (and trusting) the software on which it has been built.
In these papers we explore how some typical software development techniques, such as the use of test-driven development and agile computing, can improve the process of creating scientific software. The techniques are demonstrated in the accompanying source code.
- P.C.R. Lane and F. Gobet, ‘A theory-driven testing methodology for developing scientific software’, Journal of Experimental and Theoretical Artificial Intelligence, 24:421-56, 2012. Web page, code.
- P.C.R. Lane and F.Gobet, ‘Developing reproducible and comprehensible computational models,’ Artificial Intelligence, 144:251-263, 2003. Web page, code.
Techniques for optimising models
Most models have parameters which need to be set to particular values to get the best results. A model should additionally be verified against multiple sets of data: a model of categorisation should learn to categorise more than one set of objects. Therefore, parameters must be set to work across many different experiments simultaneously.
In this study, we developed an evolutionary algorithm to locate parameters across four competing computational models and twenty-nine different datasets, simultaneously. The algorithm helps solve two typical problems: to locate the full range of acceptable model parameters for a given dataset, and to confirm the consistency of model parameters across different datasets.
Software used in this study is available at: model evolution toolkit.
- P.C.R. Lane and F. Gobet, ‘Evolving non-dominated parameter sets for computational models from multiple experiments’, Journal of Artificial General Intelligence, 4:1-30, 2013. Web page.
Recommendations for practitioners
Often, practitioners within a scientific discipline may need advice on how to conduct and/or report a study in order to improve the likelihood that the study will or can be reproduced.
For example, looking at the area of software defect prediction, relatively few studies are replicated. The reasons for this can often be found in a lack of information in the original publications making it impossible to copy the technique used. One way to improve the likelihood of having a study’s results reproduced is to provide the right kinds of information.
(This work was largely done by Zaheed Mahmood, building out of prior work by David Bowes and Tracy Hall, with some input/supervision from myself.)
- Z. Mahmood, D. Bowes, T. Hall, P.C.R. Lane and J. Petric, ‘Reproducibility and replicability of software defect prediction studies,’ Information and Software Technology, 99:148-63, 2018. Web page.