Applying Inter-rater Reliability to Improve Consistency in Classifying PhD Career Outcomes

Stayart, C. A.; Brandt, P.; Brown, A.; Hutto, T.; Layton, R.; Petrie, K. A.; Flores-Kim, E. N.; Pena, C. G.; Fuhrmann, C. N.; Monsalve, G. C.

doi:10.1101/370783

bioRxiv · 10.1101/370783

Applying Inter-rater Reliability to Improve Consistency in Classifying PhD Career Outcomes

Stayart, C. A.·Brandt, P.·Brown, A.·Hutto, T.·Layton, R.·Petrie, K. A.·Flores-Kim, E. N.·Pena, C. G.·Fuhrmann, C. N.·Monsalve, G. C.

View original source ↗Export metadata ↓

Abstract

In the past year, there has been an exciting groundswell of national efforts to integrate multiple taxonomies for the transparent dissemination and analysis of PhD career outcomes. In this study, we leveraged the unique resources of the Broadening Experiences in Scientific Training Consortium to examine the reliability of the three-tiered Unified Career Outcomes Taxonomy (UCOT v.2017) that was collaboratively developed at a meeting convened by Rescuing Biomedical Research in August 2017. Using an amended version of the UCOT v.2017 (UCOT v.2017-rev1) and a new Supplementary Guidance document, we categorized over 570 PhD alumni records from three different universities. Utilizing Krippendorffs alpha to measure the interrater reliability from nine different individuals, we determined moderate to robust reproducibility within the first two tiers of the taxonomy (Workforce Sector and Career Type); however, the reliability for the third tier (Job Function) did not meet established standards. The team identified significant sources of error, revised category definitions, improved coder training materials and processes, and tested for improved reliability through coding 219 PhD alumni records using the revised taxonomy, UCOT v.2017-rev2. Our results revealed that the changes introduced in UCOT v.2017-rev2 improved inter-rater reliability in all three tiers, and either met or exceeded the acceptable standards for reliability. A final set of clarifications were made to UCOT v.2017-rev2, resulting in UCOT v.2018 and a Finalized Guidance document. Our findings underscore the importance of carefully developing guidance documents to aid coders in the reliable and consistent categorization of alumni career outcomes. We propose periodic assessment of the UCOT v.2018 to address the natural evolution of PhD careers in the global workforce. Ultimately, we hope that UCOT v.2018 will aid in the classification and dissemination of alumni career outcomes that is essential to educating trainees, institutions, and agencies about the diversity of career options for PhDs, and therein empower all PhDs to pursue the careers of their choice.

Source connections

Published article ↗

Explore related subjects

scientific communication and education Preprint

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Stayart, C. A., Brandt, P., Brown, A., Hutto, T., Layton, R., Petrie, K. A., Flores-Kim, E. N., Pena, C. G., Fuhrmann, C. N., Monsalve, G. C.. 2018-07-19. Applying Inter-rater Reliability to Improve Consistency in Classifying PhD Career Outcomes. https://doi.org/10.1101/370783

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related preprints

bioRxiv2026

From Problems to Solutions: How Self-Questions Drive Strong Scientific Reasoning

Metacognition, or awareness and control of thinking, is essential for problem solving because it helps solvers select strategies for learning and decide when to enact them (planning), check their understanding in the moment (monitoring), and assess effectiveness of solutions and learning strategies (evaluating). Most studies that examine metacognition do so retrospectively, leaving a critical gap in how metacognition is used in-the-moment for problem solving. In this study, we utilized a think-aloud interview approach to capture and characterize the metacognitive skills that senior undergraduate life science students at three institution types in the United States used spontaneously while solving challenging biology problems. We found that the students (n = 39) displayed a wide variety and quantity of monitoring, some evaluating, and nearly no planning skills while creating a solution backed with high-quality reasoning. Students frequently used self-questions to monitor and evaluate their thinking. We found that self-questioning is a critical skill for generating solutions, but it only moves problem solving forward if the student asks and correctly answers the self-question out loud. This study illustrates how self-questioning and other metacognitive regulation skills provide links to bridge the gap between a student's prior knowledge and specific features of a challenging problem. Based on our student data, we conceptualized a model of metacognitive phases during problem solving with high-quality reasoning. Finally, we present a self-question scaffold to facilitate the proposed phases (planning, monitoring, reasoning, and evaluating) so educators can equip students with metacognitive and scientific reasoning tools for effective problem solving.

scientific communication and education↗

bioRxiv2026

Preprint solicitation as a natural experiment in scholarly publishing

Preprinting is a growing and increasingly recognized mechanism for scientific communication and productivity and a way to facilitate global access to science. The interplay of preprinting practices with formal peer-reviewed publication is an open question that is now tractable to targeted analysis. Here, we take advantage of nearly a decade of preprint solicitation at Proceedings B to address this question, which offers a quasi-experimental setting for comparing outcomes for solicited preprints and traditionally submitted manuscripts. We used submission records from 15,368 manuscripts from 2018-2024 to assess whether preprint solicitation affects acceptance rates, journal scope, author geographic diversity, and post-publication uptake. Solicited manuscripts were accepted at nearly twice the rate of traditional submissions and received more downloads and citations after publication. Submission topics were broadly similar between the two routes, though solicited manuscripts showed marginally greater subject diversity. Geographic disparities favoring the Global North persisted, but acceptance rates for solicited manuscripts converged across regions, narrowing the gap seen in traditional submissions. These findings extend earlier work on this initiative and indicate that editor-driven solicitation can identify high-quality preprints, broaden geographic representation, and increase research visibility, offering a replicable model for journals seeking to diversify authorship while maintaining editorial standards.

scientific communication and education↗

bioRxiv2026

Biology Undergraduates Use of AI-Enabled Feedback to Revise Scientific Writing: Recipience and Barriers

Background: Instructors and education researchers are increasingly developing feedback with the assistance of generative AI, which we refer to as AI-enabled feedback, yet we do not know whether established understandings of feedback uptake and effectiveness apply similarly in AI-mediated contexts. This makes the recipience of AI-enabled feedback a central concern to understanding how to best support students in developing their scientific writing skills. This study examined how two teams of undergraduate biology students used AI-enabled feedback to revise their scientific writing, and the extent to which their revisions addressed that feedback. The teams revised their research proposal drafts after receiving AI-enabled feedback developed by their instructor with the aid of a custom AI tool. Results: Our findings show that AI-enabled feedback prompted actionable revision across both teams. Each team addressed approximately half of the feedback comments, indicating comparable levels of uptake despite their differing approaches to revision. We observed that both teams were inconsistent in providing the additional information and reasoning requested in the feedback, and they struggled to address comments requiring self-evaluation. Taken together, we see that students often addressed the more easily actionable parts of AI-enabled feedback while omitting more demanding revisions. Conclusions: These findings provide behavioral evidence of how students used AI-enabled feedback to revise their scientific writing. The observed patterns suggest that the content of AI-enabled feedback supported meaningful revision, while limitations in students agency, strategies, and volition likely shaped their implementation of comments. Although the degree of uptake observed in this study is encouraging, it is unlikely that a single application of generative AI within the feedback process will resolve all the challenges associated with feedback uptake.

scientific communication and education↗