Using SED-ML for reproducible curation: Verifying BioModels across multiple simulation engines

Abstract

The BioModels Repository contains over 1000 manually curated mechanistic models from published literature, most often encoded in the Systems Biology Markup Language (SBML). This community-based standard formally specifies each model, but does not describe the computational experimental conditions to run a simulation. Therefore, it can be challenging to reproduce any figure or result from a publication with an SBML model alone. The Simulation Experiment Description Markup Language (SED-ML) provides a solution: a standard way to specify exactly how to run an experiment corresponding to a specific figure or result. BioModels was established years before SED-ML, and both systems evolved over time, both in content and acceptance. Hence, only about half of the entries in BioModels contained SED-ML files, and these files reflected the version of SED-ML that was available at the time. Additionally, almost all of these SED-ML files had at least one minor mistake that made them invalid. To make these models and their results more reproducible, we report here on our work updating, correcting and generating new SED-ML files for 1055 curated mechanistic models in BioModels. In addition, because SED-ML is implementation-independent, it can be used for verification, demonstrating that results hold across multiple simulation engines. We tested, corrected, and improved over 450 existing SED-ML files in the BioModels database, and created basic files for the rest of the entries. Then, we used a wrapper architecture for interpreting SED-ML, and report verification results across five different ODE-based biosimulation engines, after further improving the models, the wrappers, and the engines themselves. Our work with SED-ML and the BioModels collection aims to improve the utility of these models by making them more reproducible and credible. Improved reproducibility means these models are now even more fit for re-use, such as in new investigations and as components of multiscale models. Author summaryReproducing computationally-derived scientific results seems like it should be straightforward, but is often elusive. Code is lost, file formats change, and knowledge of what was done is only partially recorded and/or forgotten. Model repositories such as BioModels address this failing in the Systems Biology domain by encoding models in a standard format that can reproduce a figure from the paper from which it was drawn. Here, we delved into the BioModels repository to create and correct the instructions on what to do with every curated model, and then tested those instructions on a variety of simulation platforms, allowing us to find and correct issues with the platforms and the simulators themselves. Not only did this improve the BioModels repository, but also improved the infrastructure necessary to run these verification comparisons in the future, and improved the fitness of the models for re-use by other researchers.