Capturing Context in Organismal Development with a Large Language Model

Abstract

Understanding how regulatory complexity and constraint shape organismal development remains a central challenge in biology. The developmental hourglass framework posits that mid-embryogenesis -the phylotypic stage- is a period of heightened conservation and coordinated regulatory organization. We test this hypothesis using Zebraformer, a transformer-based language model trained on single-cell transcriptomic data from zebrafish embryos. Zebraformer learns context-sensitive representations that capture temporal progression, anatomical identity, and regulatory relationships, yielding gene and cell embeddings that recapitulate the developmental axis and increasing transcriptional divergence over time. In contrast, attention-derived gene networks reveal a transient reorganization of regulatory architecture during the phylotypic stage, marked by tightly coordinated gene modules, reduced cross-module connectivity, and diminished local redundancy. Sensitivity to perturbation emerges specifically when regulatory interaction structure is taken into account, rather than from perturbation magnitude alone, highlighting that constraint during this stage is embedded in network topology rather than representational fragility. These findings are supported by graph-theoretic metrics and gene ontology enrichment analyses. Together, our results refine the hourglass framework by localizing developmental constraint to the architecture of gene regulatory networks and demonstrate that language models can extract interpretable biological structure from high-dimensional single-cell data Significance statementUnderstanding how cells coordinate to build complex organisms remains a central challenge in biology. Development is genetically encoded yet context dependent, shaped by interactions among genes, cells, and time. Here, we use a transformer-based language model, Zebraformer, trained on single-cell gene expression data from zebrafish embryos to investigate how regulatory organization evolves. The model captures key features of organismal formation, including transcriptional divergence, anatomical specificity, and reorganization of gene regulatory structure. Attention-derived networks reveal a transient phase of tightly coordinated gene modules with reduced cross-module connectivity during the phylotypic stage, followed by specialization. These findings refine the hourglass hypothesis by localizing constraint to network-level organization and demonstrate that contextual language models uncover core principles of biological organization directly from data.