UNNT: A novel Utility for comparing Neural Net and Tree-based models

Abstract

Using deep learning in cancer research to tackle scientific challenges is becoming an increasingly popular technique. Advances in enhanced data generation, machine learning algorithms, and compute infrastructure have led to an acceleration in the use of deep learning in various domains of cancer research such as drug response problems. In our study, we explored tree-based models to improve the accuracy of a single drug response model and demonstrate that tree-based models such as XGBoost have advantages over deep learning models, such as a convolutional neural network (CNN), for single drug response problems. However, comparing models is not a trivial task. To make training and comparing CNNs and XGBoost (eXtreme Gradient Boosting) more accessible to users, we developed an open-source library called UNNT (A novel Utility for comparing Neural Net and Tree-based models). The case studies, in this manuscript, focus on cancer drug response datasets however the application can be used on datasets from other domains, such as chemistry. Author summaryAdvancement in data science, machine learning (ML), and artificial intelligence (AI) methods has enabled extraction of meaningful information from large and complex datasets that has assisted in better understanding, diagnosing, and treating cancer. The understanding of the drug response domain in cancer research has been accelerated with developing ML models to aid in predicting the effectiveness of the drugs based on a specific genomic molecular feature. In this study we developed a novel robust framework called UNNT (A novel Utility for comparing Neural Net and Tree-based models) that trains and compares deep learning method such as CNN and tree-based method such as XGBoost on the user input dataset. We applied this software to single drug response problem in cancer to identify the best performing ML method based on the National Cancer Institute 60 (NCI60) dataset. In addition, we studied the computational aspects of training each of these models where our results show that neither is evidently superior on both CPUs and GPUs while training. This shows that when both models have similar error rates for a dataset the hardware available determines the model choice for training as discussed in tables 7 and 11. UNNT: https://github.com/vgutta/UNNT.git O_TBL View this table: org.highwire.dtl.DTLVardef@410860org.highwire.dtl.DTLVardef@18ce417org.highwire.dtl.DTLVardef@d2b43corg.highwire.dtl.DTLVardef@1370c16org.highwire.dtl.DTLVardef@fa7a00_HPS_FORMAT_FIGEXP M_TBL O_FLOATNOTable 7.C_FLOATNO O_TABLECAPTIONFastest training times for CNN and XGBoost on CPU and GPU (all features) C_TABLECAPTION C_TBL O_TBL View this table: org.highwire.dtl.DTLVardef@5f9fbborg.highwire.dtl.DTLVardef@1efecc1org.highwire.dtl.DTLVardef@a0813borg.highwire.dtl.DTLVardef@1845b46org.highwire.dtl.DTLVardef@8463e8_HPS_FORMAT_FIGEXP M_TBL O_FLOATNOTable 11.C_FLOATNO O_TABLECAPTIONFastest training times for CNN and XGBoost on CPU and GPU (FDA model) C_TABLECAPTION C_TBL