Czech, E. A. | bioRxiv Science

bioRxiv2024

Analysis-ready VCF at Biobank scale using Zarr

BackgroundVariant Call Format (VCF) is the standard file format for interchanging genetic variation data and associated quality control metrics. The usual row-wise encoding of the VCF data model (either as text or packed binary) emphasises efficient retrieval of all data for a given variant, but accessing data on a field or sample basis is inefficient. Biobank scale datasets currently available consist of hundreds of thousands of whole genomes and hundreds of terabytes of compressed VCF. Row-wise data storage is fundamentally unsuitable and a more scalable approach is needed. ResultsZarr is a format for storing multi-dimensional data that is widely used across the sciences, and is ideally suited to massively parallel processing. We present the VCF Zarr specification, an encoding of the VCF data model using Zarr, along with fundamental software infrastructure for efficient and reliable conversion at scale. We show how this format is far more efficient than standard VCF based approaches, and competitive with specialised methods for storing genotype data in terms of compression ratios and single-threaded calculation performance. We present case studies on subsets of three large human datasets (Genomics England: n=78,195; Our Future Health: n=651,050; All of Us: n=245,394) along with whole genome datasets for Norway Spruce (n=1,063) and SARS-CoV-2 (n=4,484,157). We demonstrate the potential for VCF Zarr to enable a new generation of high-performance and cost-effective applications via illustrative examples using cloud computing and GPUs. ConclusionsLarge row-encoded VCF files are a major bottleneck for current research, and storing and processing these files incurs a substantial cost. The VCF Zarr specification, building on widely-used, open-source technologies has the potential to greatly reduce these costs, and may enable a diverse ecosystem of next-generation tools for analysing genetic variation data directly from cloud-based object stores, while maintaining compatibility with existing file-oriented workflows. Key PointsO_LIVCF is widely supported, and the underlying data model entrenched in bioinformatics pipelines. C_LIO_LIThe standard row-wise encoding as text (or binary) is inherently inefficient for large-scale data processing. C_LIO_LIThe Zarr format provides an efficient solution, by encoding fields in the VCF separately in chunk-compressed binary format. C_LI

bioinformatics↗

bioRxiv2020

Massively parallel quantification of phenotypic heterogeneity in single cell drug responses

Single cell analysis tools have made significant advances in characterizing genomic heterogeneity, however tools for measuring phenotypic heterogeneity have lagged due to the increased difficulty of handling live biology. Here, we report a single cell phenotyping tool capable of measuring image-based clonal properties at scales approaching 100,000 clones per experiment. These advances are achieved by exploiting a novel flow regime in ladder microfluidic networks that, under appropriate conditions, yield a mathematically perfect cell trap. Machine learning and computer vision tools are used to control the imaging hardware and analyze the cellular phenotypic parameters within these images. Using this platform, we quantified the responses of tens of thousands of single cell-derived acute myeloid leukemia (AML) clones to targeted therapy, identifying rare resistance and morphological phenotypes at frequencies down to 0.05%. This approach can be extended to higher-level cellular architectures such as cell pairs and organoids and on-chip live-cell fluorescence assays.

bioengineering↗

bioRxiv2019

Extracting T Cell Function and Differentiation Characteristics from the Biomedical Literature

The role of many cytokines and transcription factors in the function and development of human T cells has been the subject of extensive research, however much of this work only demonstrates experimental findings for a relatively small portion of the molecular signaling network that enables the plasticity inherent to these cells. We apply recent advancements in methods for weak supervision and transfer learning for natural language models to aid in extracting these individual findings as 283k cell type, cytokine, and transcription factor relations from 64k relevant documents (53k full-text PMC articles and 11k PubMed abstracts). All data, results and source code available at https://github.com/hammerlab/t-cell-relation-extraction.

immunology↗