Cooper, C. I. | bioRxiv Science

bioRxiv2018

HPCI: A Perl module for writing cluster-portable bioinformatics pipelines

BackgroundMost biocomputing pipelines are run on clusters of computers. Each type of cluster has its own API (application programming interface). That API defines how a program that is to run on the cluster must request the submission, content and monitoring of jobs to be run on the cluster. Sometimes, it is desirable to run the same pipeline on different types of cluster. This can happen in situations including when:\n\nO_LIdifferent labs are collaborating, but they do not use the same type of cluster\nC_LIO_LIa pipeline is released to other labs as open source or commercial software\nC_LIO_LIa lab has access to multiple types of cluster, and wants to choose between them for scaling, cost or other purposes\nC_LIO_LIa lab is migrating their infrastructure from one cluster type to another\nC_LIO_LIduring testing or travelling, it is often desired to run on a single computer\nC_LI\n\nHowever, since each type of cluster has its own API, code that runs jobs on one type of cluster needs to be re-written if it is desired to run that application on a different type of cluster. To resolve this problem, we created a software module to generalize the submission of pipelines across computing environments, including local compute, clouds and clusters.\n\nResultsHPCI (High Performance Computing Interface) is a Perl module that provides the interface to a standardized generic cluster.\n\nWhen the HPCI module is used, it accepts a parameter to specify the cluster type. The HPCI module uses this to load a driver HPCD:: . This is used to translate the abstract HPCI interface to the specific software interface.\n\nSimply by changing the cluster parameter, the same pipeline can be run on a different type of cluster with no other changes.\n\nConclusionThe HPCI module assists in writing Perl programs that can be run in different lab environments, with different site configuration requirements and different types of hardware clusters. Rather than having to re-write portions of the program, it is only necessary to change a configuration file.\n\nUsing HPCI, an application can manage collections of jobs to be runs, specify ordering dependencies, detect success or failure of jobs run and allow automatic retry of failed jobs (allowing for the possibility of a changed configuration such as when the original attempt specified an inadequate memory allotment).

bioinformatics↗

bioRxiv2018

Valection: Design Optimization for Validation and Verification Studies

BackgroundPlatform-specific error profiles necessitate confirmatory studies where predictions made on data generated using one technology are additionally verified by processing the same samples on an orthogonal technology. In disciplines that rely heavily on high-throughput data generation, such as genomics, reducing the impact of false positive and false negative rates in results is a top priority. However, verifying all predictions can be costly and redundant, and testing a subset of findings is often used to estimate the true error profile. To determine how to create subsets of predictions for validation that maximize inference of global error profiles, we developed Valection, a software program that implements multiple strategies for the selection of verification candidates.\n\nResultsTo evaluate these selection strategies, we obtained 261 sets of somatic mutation calls from a single-nucleotide variant caller benchmarking challenge where 21 teams competed on whole-genome sequencing datasets of three computationally-simulated tumours. By using synthetic data, we had complete ground truth of the tumours mutations and, therefore, we were able to accurately determine how estimates from the selected subset of verification candidates compared to the complete prediction set. We found that selection strategy performance depends on several verification study characteristics. In particular the verification budget of the experiment (i.e. how many candidates can be selected) is shown to influence estimates.\n\nConclusionsThe Valection framework is flexible, allowing for the implementation of additional selection algorithms in the future. Its applicability extends to any discipline that relies on experimental verification and will benefit from the optimization of verification candidate selection.

bioinformatics↗