A Structured Evaluation of Genome-Scale Constraint-Based Modeling Tools for Microbial Consortia

Abstract

Harnessing the power of microbial consortia is integral to a diverse range of sectors, from healthcare to biotechnology to environmental remediation. To fully realize this potential, it is critical to understand the mechanisms behind the interactions that structure microbial consortia and determine their functions. Constraint-based reconstruction and analysis (COBRA) approaches, employing genome-scale metabolic models (GEMs), have emerged as the state-of-the-art tool to simulate the behavior of microbial communities from their constituent genomes. In the last decade, many tools have been developed that use COBRA approaches to simulate multi-species consortia, under either steady-state, dynamic, or spatiotemporally varying scenarios. Yet, these tools have not been systematically evaluated regarding their software quality, most suitable application, and predictive power. Hence, it is uncertain which tools users should apply to their system and what are the most urgent directions that developers should take in the future to improve existing capacities. This study conducted a systematic evaluation of COBRA-based tools for microbial communities using datasets from two-member communities as test cases. First, we performed a qualitative assessment in which we evaluated 24 published tools based on a list of FAIR (Findability, Accessibility, Interoperability, and Reusability) features essential for software quality. Next, we quantitatively tested the predictions in a subset of 14 of these tools against experimental data from three different case studies: a) syngas fermentation by C. autoethanogenum and C. kluyveri for the static tools, b) glucose/xylose fermentation with engineered E. coli and S. cerevisiae for the dynamic tools, and c) a Petri dish of E. coli and S. enterica for tools incorporating spatiotemporal variation. Our results show varying performance levels of the best qualitatively assessed tools when examining the different categories of tools. The differences in the mathematical formulation of the approaches and their relation to the results were also discussed. Ultimately, we provide recommendations for refining future GEM microbial modeling tools. Author summaryConstraint-based modeling employing genome-scale reconstructions of microbial species has become one of the most successful approaches for studying, analyzing, and engineering microbial consortia. Over the past decade, many constraint-based modeling tools have been published to examine an immense variety of microbial consortia spanning from the application areas of bioremediation to food and health biotechnology. However, new potential users lack an overview of the quality and performance of existing metabolic modeling tools that would guide their choice. To tackle this issue, we examined 24 tools for genome-scale metabolic modeling of microbial consortia. After an initial qualitative screening, we quantitatively evaluated 14 adequate tools against published experimental data that included different organisms and conditions. We conducted simulations and evaluated model features such as predictive accuracy, computational time, and tractability in capturing critical physiological properties. We found that, generally, more up-to-date, accessible, and documented tools were superior in many important aspects of model quality and performance. Although, in some cases, we observed tradeoffs in older, less elaborate tools that can be more accurate or flexible. This work has broad implications to help researchers navigate the most suitable tools, and suggests to developers opportunities for improvement of the currently existing capabilities for metabolic modeling of multi-species microbial consortia.