bioRxiv Science⌕ Search
Home/Archive/bioRxiv

bioRxiv · 10.1101/2024.03.25.586710

Have AI-Generated Texts from LLM Infiltrated the Realm of Scientific Writing? A Large-Scale Analysis of Preprint Platforms

Cheng, H.·Sheng, B.·Lee, A.·Chaudhary, V.·Atanasov, A. G.·Liu, N.·Qiu, Y.·Wong, T. Y.·Tham, Y.-C.·Zheng, Y.-F.
View original source ↗Export metadata ↓

Abstract

Since the release of ChatGPT in 2022, AI-generated texts have inevitably permeated various types of writing, sparking debates about the quality and quantity of content produced by such large language models (LLM). This study investigates a critical question: Have AI-generated texts from LLM infiltrated the realm of scientific writing, and if so, to what extent and in what setting? By analyzing a dataset comprised of preprint manuscripts uploaded to arXiv, bioRxiv, and medRxiv over the past two years, we confirmed and quantified the widespread influence of AI-generated texts in scientific publications using the latest LLM-text detection technique, the Binoculars LLM-detector. Further analyses with this tool reveal that: (1) the AI influence correlates with the trend of ChatGPT web searches; (2) it is widespread across many scientific domains but exhibits distinct impacts within them (highest: computer science, engineering sciences); (3) the influence varies with authors who have different language speaking backgrounds and geographic regions according to the location of their affiliations (Italy, China, etc.); (4) AI-generated texts are used in various content types in manuscripts (most significant: hypothesis formulation, conclusion summarization); (5) AI usage has a positive influence on papers impact, measured by its citation numbers. Based on these findings, suggestions about the advantages and regulation of AI-augmented scientific writing are discussed.

Explore related subjects

scientific communication and educationPreprint

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Cheng, H., Sheng, B., Lee, A., Chaudhary, V., Atanasov, A. G., Liu, N., Qiu, Y., Wong, T. Y., Tham, Y.-C., Zheng, Y.-F.. 2024-03-26. Have AI-Generated Texts from LLM Infiltrated the Realm of Scientific Writing? A Large-Scale Analysis of Preprint Platforms. https://doi.org/10.1101/2024.03.25.586710

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related preprints

bioRxiv2026

Perceived Risk and Barriers to Open and Responsible Research Across Fifteen UK Universities

Open research practices are increasingly promoted to improve research transparency, reproducibility, accessibility, and societal impact. Despite growing support from research funders, institutions, and policy initiatives, adoption remains uneven across disciplines and research communities. This study examined perceived risks and barriers associated with 14 FORRT Guideline areas using qualitative responses from the UK Reproducibility Network Open and Transparent Research Practices Survey (N = 2,567), conducted across 15 UK higher education institutions. Free-text responses describing risks and barriers were analysed using inductive thematic analysis. A total of 3,951 relevant comments generated 3,433 coded references to barriers and risks. Three interconnected clusters emerged. Individual concerns included lack of motivation, fear of losing intellectual credit, and concerns about exposing mistakes and criticism. Systemic and institutional barriers included lack of time and resources, inadequate infrastructure, insufficient training and support, lack of incentives and recognition, unclear guidance, disciplinary and methodological challenges, and tensions between open practices and intellectual property requirements. Ethical and quality-related concerns included risks to participant confidentiality and privacy, challenges associated with sensitive data, concerns about inappropriate application of open research practices across different research traditions, perceived impacts on research quality and innovation, and potential effects on public trust in research. Lack of time and resources was the most frequently reported barrier. Across responses, barriers were commonly described as interdependent, with shortcomings in funding, infrastructure, institutional support, incentives, and training reinforcing one another. Respondents largely supported the principles underpinning open research but highlighted substantial practical, professional, and ethical challenges to implementation. These findings suggest that increasing open research adoption requires more than policy mandates or awareness-raising activities. Sustainable uptake will depend on aligned incentives, adequate infrastructure and support, recognition of methodological diversity, and approaches that enable openness to be implemented responsibly across different research contexts.

scientific communication and education↗
bioRxiv2026

Evaluating Large Language Models as Tools to Navigate Researchers in Rapidly Evolving Research Landscapes: A Case Study in Cancer Drug Response Prediction

Large Language Models (LLMs) have emerged as promising tools for assisting researchers in automating and accelerating the synthesis of literature reviews. However, their reliability is a significant concern due to issues like factual inaccuracies and hallucinations. The key question is whether LLMs can reliably provide comprehensive, up-to-date overviews and analyses. This study evaluates the performance of three leading LLMs (OpenAI's ChatGPT, Google's Gemini, and DeepSeek) on the complex task of generating a comprehensive survey paper on deep learning for cancer Drug Response Prediction (DRP). By testing both standard and Deep Research (DR) / Deep Think (DT) modes of LLMs with prompts of varying detail, this paper assesses key academic dimensions, including reference management, content quality, and analytical depth. Key findings reveal that while DR modes of LLMs significantly improve reliability by eliminating hallucinations, performance variations exist across models and prompts. A trade-off between reference quantity and integration quality was observed, and even the best-performing models lacked the analytical depth of human experts, often requiring extensive human supervision. The study concludes that LLMs currently serve as powerful assistive tools but still cannot replace the critical validation and synthesis provided by human researchers. Choosing the best LLM to use depends on the task in hand, while several strategies can be implemented to improve the produced output.

scientific communication and education↗
bioRxiv2026

Undergraduate Biophysical Chemistry Series: Teaching through a Combination of a Purpose-built Textbook, Research-derived Biomolecular Samples and Computer Labs

Biophysics is a rapidly advancing field with an incredible breadth of topics. Thus, undergraduate biophysics instructors have to strategize and decide what topics they will cover in their courses. Educational institutions utilize a variety of biophysics textbooks. A common deficiency of each of the existing texts is that it serves well a given set of topics (theory, illustrations, practice problems) and leaves out other areas. A typical example includes good theory and problems for thermodynamics and kinetics while presenting molecular dynamics and various spectroscopic methods in a lacking or outdated way. The authors of this manuscript teach a capstone Biophysical Chemistry three-quarter series (Western Washington University/WWU, Bellingham, WA) which ideally should resonate with the general and major-specific courses the students take within their major at WWU. To achieve this goal and to enrich the traditional lecture-based delivery, the instructors have developed and brought together key pedagogical elements: purpose-built online textbook with a uniform structure of the academic content and practice problems, a study sample (oligopeptide) of biophysical significance with a growing set of experimental and computational data and student-centric in-class activities including computer labs. Our Biophysical series emphasizes concepts and methods of computational structural biology (Molecular Dynamics) and spectroscopic approaches (IR, UV and NMR). Here we describe the details of our integrative approach, summarize key outcomes and chart ways to advance the biophysical chemistry series further. Our textbook can be found through LibreText.

scientific communication and education↗