Biology subjects

Toelch, U.

Publications and source records attributed to Toelch, U..

3 recordsLinked to original sources

bioRxiv2026

Set-up, validation, evaluation, and cost-benefit analysis of an AI-assisted assessment of responsible research practices in a sample of life science publications

The (semi-)automated screening of publications for diverse quality and transparency criteria is at the core of systematic literature assessment. Typically, the assessment process involves two initial reviewers and one additional reviewer for cases that require reconciliation. Here, we explore to what extent this process can be assisted by Large Language Models (LLMs). Specifically, whether LLMs are capable of assessing responsible research practices (RRPs) in scientific papers in a robust way. We employed proprietary LLMs to assess an initial set of 37 papers across ten RRPs. The same papers were also reviewed by three human reviewers. We iteratively redesigned prompts to increase model accuracy compared to human ratings which we treated as the gold standard. The resulting pipeline was validated on an additional set of 15 papers. We show that LLM accuracy is comparable to single human reviewer performance (90% for LLM vs 86% for a single human reviewer). However, performance strongly depended on the specific RRPs with accuracy ranging from 40% to 100%. LLMs exhibited an affirmative bias, making more errors when practices were not reported in the papers. Overall, we show how such an approach potentially replaces one human reviewer, enabling AI-assisted assessment of research papers. We discuss how dataset imbalances, validation procedures, and implementation time limit the broad applicability of such approaches. Through this, we develop initial guidance on the utility of proprietary LLMs in evidence synthesis.

scientific communication and education↗

bioRxiv2022

Increasing discovery rates in preclinical research through optimised statistical decision criteria

The success of scientific discovery in preclinical research is based on the different roles of exploration and confirmation. Exploration involves identifying potential effects (high sensitivity), which are then tested more rigorously during confirmation (high specificity). Here, we examine different experimental strategies and their ability to balance sensitivity and specificity to identify relevant effects. In simulations based on empirical data, we specifically compare a conventional p-value based approach with a method based on an a priori determined smallest effect size of interest (SESOI). Using a SESOI increases transition rates from exploration to confirmation and leads to higher detection rates across the trajectory. In particular, specificity in the SESOI trajectory increases if number of true effects are low. We conclude that employing a SESOI is superior to a p-value based approach in many contexts. Based on our findings, we propose a reconsideration of planning and conducting preclinical experiments, especially when the prior probability of true hypotheses is low.

scientific communication and education↗

bioRxiv2018

I judge you by your profit: Judgments of effort exerted by selfand others are influenced by task contingent rewards

Effort constitutes a major part of cost-benefit calculations underlying decision making. Therefore, estimating the effort someone has spent on a task is a core dimension for evaluating own and others actions. It has been previously shown that self-judgments of effort are influenced by the magnitude of obtained rewards. It is unclear, however, whether the influence of reward on effort estimations is limited to self-judgments or whether reward incorporation represents a general computational principle when judging effort. Here we show that people also integrate reward magnitude when judging the effort exerted by others. Participants (N=48) performed an effortful sensorimotor task interleaved with a partner, while rating either their own or the other persons effort. After each trial but before the effort rating, both participants were informed about the obtained reward. We found that higher rewards led to higher estimations of exerted effort, in self-as well as other-judgments, and this effect was more pronounced for other-judgments. In both types of judgment, computational modelling revealed that reward information and the perceived level of exertion were combined in a Bayes optimal manner to form effort estimates. Remarkably, the extent to which rewards influenced effort judgments was positively correlated with conservative world-views, indicating that the basic computations underlying this behavioural phenomenon might be related to more general beliefs about the association between effort and reward in the society. The integration of reward information into retrospective effort judgments underscores the convergence of multiple information sources that supports adaptive learning and decision making in social contexts.

neuroscience↗