bioRxiv Science⌕ Search
Home/Archive/bioRxiv

bioRxiv · 10.1101/2022.08.10.503444

The logical structure of experiments lays the foundation for a theory of reproducibility

Buzbas, E. O.·Devezer, B.·Baumgaertner, B.
View original source ↗Export metadata ↓

Abstract

The scientific reform movement has proposed openness as a potential remedy to the putative reproducibility or replication crisis. However, the conceptual relationship between openness, replication experiments, and results reproducibility has been obscure. We analyze the logical structure of experiments, define the mathematical notion of idealized experiment, and use this notion to advance a theory of reproducibility. Idealized experiments clearly delineate the concepts of replication and results reproducibility, and capture key differences with precision, allowing us to study the relationship among them. We show how results reproducibility varies as a function of: the elements of an idealized experiment, the true data generating mechanism, and the closeness of the replication experiment to an original experiment. We clarify how openness of experiments is related to designing informative replication experiments and to obtaining reproducible results. With formal backing and evidence, we argue that the current "crisis" reflects inadequate attention to a theoretical understanding of results reproducibility.

Explore related subjects

scientific communication and educationPreprint

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Buzbas, E. O., Devezer, B., Baumgaertner, B.. 2022-08-12. The logical structure of experiments lays the foundation for a theory of reproducibility. https://doi.org/10.1101/2022.08.10.503444

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related preprints

bioRxiv2026

Evaluating Large Language Models as Tools to Navigate Researchers in Rapidly Evolving Research Landscapes: A Case Study in Cancer Drug Response Prediction

Large Language Models (LLMs) have emerged as promising tools for assisting researchers in automating and accelerating the synthesis of literature reviews. However, their reliability is a significant concern due to issues like factual inaccuracies and hallucinations. The key question is whether LLMs can reliably provide comprehensive, up-to-date overviews and analyses. This study evaluates the performance of three leading LLMs (OpenAI's ChatGPT, Google's Gemini, and DeepSeek) on the complex task of generating a comprehensive survey paper on deep learning for cancer Drug Response Prediction (DRP). By testing both standard and Deep Research (DR) / Deep Think (DT) modes of LLMs with prompts of varying detail, this paper assesses key academic dimensions, including reference management, content quality, and analytical depth. Key findings reveal that while DR modes of LLMs significantly improve reliability by eliminating hallucinations, performance variations exist across models and prompts. A trade-off between reference quantity and integration quality was observed, and even the best-performing models lacked the analytical depth of human experts, often requiring extensive human supervision. The study concludes that LLMs currently serve as powerful assistive tools but still cannot replace the critical validation and synthesis provided by human researchers. Choosing the best LLM to use depends on the task in hand, while several strategies can be implemented to improve the produced output.

scientific communication and education↗
bioRxiv2022

Nationwide assessment of leadership development for graduate students in the agricultural plant sciences

Leadership development is a universally important goal across the agricultural plant science disciplines. Although previous studies have identified a need for leadership skills, less is known about leadership skill development in graduate programs. To address this, we constructed a mixed-method study to identify the most significant graduate school leadership experiences of scientists in the agricultural plant science disciplines. The survey was deployed to 6,728 people in the U.S. and received 1,086 responses (16.1% response rate). The majority of respondents reported that they were from one of the major agricultural states and employed at one of the agricultural plant science related doctoral universities, industries, or government. Results from this survey suggest that recent graduates were more engaged in graduate school activities that offered leadership development. Key experiences in graduate school were also identified that may be used to develop future leaders. Additionally, respondents reported the greatest barrier to providing leadership development for graduate students was that it is not part of their program curriculum, however current graduate students responded differently, and identifying lack of funding to support experiences as the greatest barrier. This survey also identified the top ranked professional skills considered most important for effective leaders in agricultural plant sciences as well as respondent-driven recommendations on how graduate programs can improve leadership development. Collectively, these results can be used in the future to identify priorities for skill development and opportunities for graduate student training in leadership skills within the plant science disciplines.

scientific communication and education↗
bioRxiv2022

The Australian academic STEMM workplace post-COVID: a picture of disarray

In 2019 we surveyed Australian early career researchers (ECRs) working in STEMM (science, technology, engineering, mathematics and medicine). ECRs almost unanimously declared a "love of research", however, many reported frequent bullying and questionable research practices (QRPs), and that they intended to leave because of poor career stability. We replicated the survey in 2022 to determine the impact of the COVID-19 pandemic and sought more information on bullying and QRPs. Here, we compare data from 2019 (658 respondents) and 2022 (530 respondents), and detail poor professional and research conditions experienced by ECRs. Job satisfaction declined (62% versus 57%), workload concerns increased (48.6% versus 60.6%), more indicated "now is a poor time to commence a research career" (65% versus 76%) from 2019 to 2022, and roughly half reported experiencing bullying. Perhaps conditions could be tolerable if the ecosystem were yielding well-trained scientists and high-quality science. Unfortunately, there are signs of poor supervision and high rates of QRPs. ECRs detailed problems likely worthy of investigation, but few (22.4%) felt that their institute would act on a complaint. We conclude by suggesting strategies for ECR mentorship, training, and workforce considerations intended to maintain research excellence in Australia and improve ECR career stability.

scientific communication and education↗