Sanabria, M. | bioRxiv Science

bioRxiv2023

The human genome's vocabulary as proposed by the DNA language model GROVER

Large Language Models (LLMs) on natural language have achieved a level of performance that allows the generation of coherent and syntactically correct text. DNA sequence of genomes follows rules similar to natural language, but a distinguishing factor is the absence of a concept analogous to words. We established byte-pair tokenization on the human genome and trained a foundation language model called GROVER ("Genome Rules Obtained Via Extracted Representations") to select the optimal vocabulary with a custom fine-tuning task of next-k-mer prediction. We thus defined a dictionary of words/tokens in the human genome that best carries the information content for DNA language models. Analyzing GROVERs learned representations, we observed that token embeddings primarily encode information related to their frequency, sequence content, and length. Some tokens are almost exclusively localized in repeats, while the vast majority widely distributes over the genome. The model also learns context and lexical ambiguity. Average embeddings of genomic regions relate to functional genomics annotation and thus indicate that GROVER has learned these structures purely from the contextual relationships of tokens. That we can extract functional annotations from the genome, purely based on sequence representation to the trained model, highlights the extent of information content encoded by the sequence. This is supported by fine-tuning tasks on genome biology with questions of promoter identity and protein-DNA binding. GROVER learns sequence context, a sense for grammatical structures and language rules in the genome. This knowledge can be extracted and used to compose a grammar book for the code of life.

genomics↗

bioRxiv2023

Distinguishing word identity and sequence context in DNA language models

Transformer-based large language models (LLMs) are very suited for biological sequence data, because the structure of protein and nucleic acid sequences show many analogies to natural language. Complex relationships in biological sequence can be learned, although there may not be a clear concept of words, because they can be generated through tokenization. Training is subsequently performed for masked token prediction. With this strategy, the models learn both the token sequence identity and a larger sequence context. We developed a framework to interrogate what a model learns, which is both relevant for the interpretability of the model and to evaluate its potential for specific tasks. We used a DNA language model, which is trained on the human reference genome with a Bidirectional Encoder Representations from Transformers (BERT) model. In this model, tokens are defined with overlapping k-mers. To gain insight into the models learning, we interrogated how the model performs predictions, extracted token embeddings, and defined a fine-tuning benchmarking task to predict the next tokens of different sizes without overlaps. This task is very suited to evaluate different pretrained DNA language models, also called foundation models, since it does not interrogate specific genome biology, does not depend on the tokenization strategy, the size of the vocabulary, the dictionary, or the number of parameters used to train the model. Lastly, the task performs without leakage of information from token identity into the prediction task, which makes it particularly useful to evaluate the learning of sequence context. Through this assessment we discovered that the model with overlapping k-mers struggles to learn larger sequence context. Instead, the learned embeddings largely represent token sequence. Still, good performance is achieved for genome biology inspired fine-tuning tasks. Models with overlapping tokens may be used for tasks where a larger sequence context is of less relevance, but the token sequence directly represents the desired learning feature. This emphasizes the need to interrogate knowledge representation in biological large language models. Transparency is particularly important for biomedical use cases and an understanding of what the models are learning can be used to match the model to the desired task.

bioinformatics↗