Stephen R Piccolo | bioRxiv Science

bioRxiv2016

A cloud-based workflow to quantify transcript-expression levels in public cancer compendia

Public compendia of raw sequencing data are now measured in petabytes. Accordingly, it is becoming infeasible for individual researchers to transfer these data to local computers. Recently, the National Cancer Institute funded an initiative to explore opportunities and challenges of working with molecular data in cloud-computing environments. With data in the cloud, it becomes possible for scientists to take their tools to the data and thereby avoid large data transfers. It also becomes feasible to scale computing resources to the needs of a given analysis. To evaluate this concept, we quantified transcript-expression levels for 12,307 RNA-Sequencing samples from the Cancer Cell Line Encyclopedia and The Cancer Genome Atlas. We used two cloud-based configurations to process the data and examined the performance and cost profiles of each configuration. Using \"preemptible virtual machines\", we processed the samples for as little as $0.09 (USD) per sample. In total, we processed the TCGA samples (n=11,373) for only $1,065.49 and simultaneously processed thousands of samples at a time. As the samples were being processed, we collected detailed performance metrics, which helped us to track the duration of each processing step and to identify computational resources used at different stages of sample processing. Although the computational demands of reference alignment and expression quantification have decreased considerably, there remains a critical need for researchers to optimize preprocessing steps (e.g., sorting, converting, and trimming sequencing reads). We have created open-source Docker containers that include all the software and scripts necessary to process such data in the cloud and to collect performance metrics. The processed data are available in tabular format and in Google's BigQuery database (see https://osf.io/gqrz9).

Bioinformatics↗

bioRxiv2015

Tools and techniques for computational reproducibility

When reporting research findings, scientists document the steps they followed so that others can verify and build upon the research. When those steps have been described in sufficient detail that others can retrace the steps and obtain similar results, the research is said to be reproducible. Computers play a vital role in many research disciplines and present both opportunities and challenges for reproducibility. Computers can be programmed to execute analysis tasks, and those programs can be repeated and shared with others. Due to the deterministic nature of most computer programs, the same analysis tasks, applied to the same data, will often produce the same outputs. However, in practice, computational findings often cannot be reproduced due to complexities in how software is packaged, installed, and executed--and due to limitations in how scientists document analysis steps. Many tools and techniques are available to help overcome these challenges. Here we describe seven such strategies. With a broad scientific audience in mind, we describe strengths and limitations of each approach, as well as circumstances under which each might be applied. No single strategy is sufficient for every scenario; thus we emphasize that it is often useful to combine approaches.

Bioinformatics↗