United States

&lt;p&gt;Reese A. K. Richardson,&lt;sup&gt;1,2&lt;/sup&gt; Spencer S. Hong,&lt;sup&gt;1,3&lt;/sup&gt; Jennifer A. Byrne,&lt;sup&gt;4,5&lt;/sup&gt; Thomas Stoeger,&lt;sup&gt;6,7,8&lt;/sup&gt; Luís A. Nunes Amaral&lt;sup&gt;1,2,9,10,11&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Objective &lt;/h4&gt;
&lt;p&gt;Some suggest that the ease of communication provided by the internet and open access publishing has created the conditions for the emergence of entities (including paper mills, brokers, and predatory publishers) that facilitate systematic scientific fraud. However, little is understood about the organization of these entities as well as how they react to and evade science integrity measures. Here, we sought to demonstrate that large networks that produce scientific fraud at scale can be identified by the footprints they have left in the published scientific literature. These footprints can be analyzed to glean insights about their organizational structure and operations.&lt;/p&gt;
&lt;h4&gt;Design &lt;/h4&gt;
&lt;p&gt;Our work consisted of observational case studies making use of article metadata from OpenAlex and the complete corpora of &lt;i&gt;PLOS One &lt;/i&gt;and Hindawi articles; all PubPeer comments made before January 2024; historical indexing data from MEDLINE, Scopus, and Web of Science; and archived webpages obtained with the Internet Archive’s Wayback Machine. Our case studies addressed networks of compromised editors at large mega-journals, analysis of article provenance through observations of interarticle image duplication, a longitudinal study of the operations of a large broker organization, a comparison of revision and retraction rates in closely related biomedical subfields, and an overall assessment of growth rates of systematic scientific fraud. This analysis was performed from June to September 2024.&lt;/p&gt;
&lt;h4&gt;Results &lt;/h4&gt;
&lt;p&gt;We identified 45 editors at &lt;i&gt;PLOS One&lt;/i&gt;, 53 editors at 10 Hindawi journals, and 205 Institute of Electrical and Electronics Engineers (IEEE) conferences that handled articles that were eventually retracted, articles with PubPeer comments, or articles featuring tortured phrases far more often than expected by chance (1-sided Poisson binomial test, Benjamini-Hochberg false discovery rate, &amp;lt;0.05). We characterized a network of image duplication spanning 2213 articles. We catalogued the involvement of a broker organization with an evolving portfolio spanning 188 journals over 7 years. We found similar rates of revision in subfields of RNA biology but retraction rates orders of magnitude apart. We found that while publication rates of scientific articles doubled approximately every 15 years and the publication rates of eventually retracted articles every 3 years, articles of likely paper mill provenance were published at a rate that doubled less than every 2 years.&lt;/p&gt;
&lt;h4&gt;Conclusions &lt;/h4&gt;
&lt;p&gt;Here, we demonstrated through case studies that (1) individuals have colluded to publish problematic papers in a number of journals; (2) brokers can ensure publication in infiltrated journals at scale; and (3) within a field of science, not all subfields are equally targeted for scientific fraud. Our results revealed some of the strategies that enable entities promoting scientific fraud to evade interventions. Our final analysis suggested that this ability to evade interventions is enabling the number of fraudulent publications to grow at a rate far outpacing that of legitimate science.&lt;/p&gt;
&lt;h4&gt;Author Affiliations&lt;/h4&gt;
&lt;sup&gt;1&lt;/sup&gt;Department of Engineering Sciences &amp; Applied Mathematics, Northwestern University, Evanston, IL, US, richardsonr43@gmail.com; &lt;sup&gt;2&lt;/sup&gt;Department of Molecular Biosciences, Northwestern University, Evanston, IL, US; &lt;sup&gt;3&lt;/sup&gt;Department of Chemical &amp; Biological Engineering, Northwestern University, Evanston, IL, US; &lt;sup&gt;4&lt;/sup&gt;School of Medical Sciences, Faculty of Medicine and Health, The University of Sydney, NSW, Australia; &lt;sup&gt;5&lt;/sup&gt;NSW Health Statewide Biobank, NSW Health Pathology, Camperdown, NSW, Australia; &lt;sup&gt;6&lt;/sup&gt;The Potocsnak Longevity Institute, Northwestern University, Chicago, IL, US; &lt;sup&gt;7&lt;/sup&gt;Simpson Querrey Lung Institute for Translational Science, Northwestern University, Chicago, IL, US; &lt;sup&gt;8&lt;/sup&gt;Division of Pulmonary and Critical Care, Northwestern University Feinberg School of Medicine, Chicago, IL, US; &lt;sup&gt;9&lt;/sup&gt;Department of Physics and Astronomy, Northwestern University, Evanston, IL, US; &lt;sup&gt;10&lt;/sup&gt;Northwestern Institute on Complex Systems, Northwestern University, Evanston, IL, US; &lt;sup&gt;11&lt;/sup&gt;NSF-Simons National Institute for Theory and Mathematics in Biology, Chicago, IL, US.
&lt;h4&gt;Conflict of Interest Disclosures&lt;/h4&gt;
Luís A. Nunes Amaral is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.

&lt;h4&gt;Funding/Support&lt;/h4&gt;
Reese A. K. Richardson was supported in part by a National Institutes of Health Training Grant (T32GM008449) through Northwestern University’s Biotechnology Training Program and receives funding from the Dr John N. Nicholson Fellowship from Northwestern University and Moderna. Spencer S. Hong receives support from the Ryan Fellowship and the International Institute for Nanotechnology at Northwestern University. Jennifer A. Byrne receives funding from a National Health and Medical Research Council of Australia Ideas grant (APP1184263) and Rewarding Research Success funding from the Faculty of Medicine and Health at The University of Sydney. Thomas Stoeger receives funding from the National Institute on Aging, Integrative Multi-Scale Systems Analysis of Gene-Expression-Driven Aging Morbidity (R00AG068544), National Institute of Allergy and Infectious Diseases (AI135964), and Successful Clinical Response In Pneumonia Therapy (SCRIPT) Systems Biology Center. Luís A. Nunes Amaral and Thomas Stoeger receive funding from SCISIPBIO: a data-science approach to evaluating the likelihood of fraud and error in published studies (1956338).
&lt;h4&gt;Role of the Funder/Sponsor&lt;/h4&gt;
The funders had no role in the design or execution of this study.



The 10th Peer Review Congress

Scale and Resilience in Organizations Enabling Systematic Scientific Fraud

and scientometrics

informatics

ethics and ethical concerns

editorial and peer review process

bibliometrics

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Reese A. K. Richardson,<sup>1,2</sup> Spencer S. Hong,<sup>1,3</sup> Jennifer A. Byrne,<sup>4,5</sup> Thomas Stoeger,<sup>6,7,8</sup> Luís A. Nunes Amaral<sup>1,2,9,10,11</sup></p>
<h4>Objective </h4>
<p>Some suggest that the ease of communication provided by the internet and open access publishing has created the conditions for the emergence of entities (including paper mills, brokers, and predatory publishers) that facilitate systematic scientific fraud. However, little is understood about the organization of these entities as well as how they react to and evade science integrity measures. Here, we sought to demonstrate that large networks that produce scientific fraud at scale can be identified by the footprints they have left in the published scientific literature. These footprints can be analyzed to glean insights about their organizational structure and operations.</p>
<h4>Design </h4>
<p>Our work consisted of observational case studies making use of article metadata from OpenAlex and the complete corpora of <i>PLOS One </i>and Hindawi articles; all PubPeer comments made before January 2024; historical indexing data from MEDLINE, Scopus, and Web of Science; and archived webpages obtained with the Internet Archive’s Wayback Machine. Our case studies addressed networks of compromised editors at large mega-journals, analysis of article provenance through observations of interarticle image duplication, a longitudinal study of the operations of a large broker organization, a comparison of revision and retraction rates in closely related biomedical subfields, and an overall assessment of growth rates of systematic scientific fraud. This analysis was performed from June to September 2024.</p>
<h4>Results </h4>
<p>We identified 45 editors at <i>PLOS One</i>, 53 editors at 10 Hindawi journals, and 205 Institute of Electrical and Electronics Engineers (IEEE) conferences that handled articles that were eventually retracted, articles with PubPeer comments, or articles featuring tortured phrases far more often than expected by chance (1-sided Poisson binomial test, Benjamini-Hochberg false discovery rate, &lt;0.05). We characterized a network of image duplication spanning 2213 articles. We catalogued the involvement of a broker organization with an evolving portfolio spanning 188 journals over 7 years. We found similar rates of revision in subfields of RNA biology but retraction rates orders of magnitude apart. We found that while publication rates of scientific articles doubled approximately every 15 years and the publication rates of eventually retracted articles every 3 years, articles of likely paper mill provenance were published at a rate that doubled less than every 2 years.</p>
<h4>Conclusions </h4>
<p>Here, we demonstrated through case studies that (1) individuals have colluded to publish problematic papers in a number of journals; (2) brokers can ensure publication in infiltrated journals at scale; and (3) within a field of science, not all subfields are equally targeted for scientific fraud. Our results revealed some of the strategies that enable entities promoting scientific fraud to evade interventions. Our final analysis suggested that this ability to evade interventions is enabling the number of fraudulent publications to grow at a rate far outpacing that of legitimate science.</p>
<h4>Author Affiliations</h4>
<sup>1</sup>Department of Engineering Sciences & Applied Mathematics, Northwestern University, Evanston, IL, US, richardsonr43@gmail.com; <sup>2</sup>Department of Molecular Biosciences, Northwestern University, Evanston, IL, US; <sup>3</sup>Department of Chemical & Biological Engineering, Northwestern University, Evanston, IL, US; <sup>4</sup>School of Medical Sciences, Faculty of Medicine and Health, The University of Sydney, NSW, Australia; <sup>5</sup>NSW Health Statewide Biobank, NSW Health Pathology, Camperdown, NSW, Australia; <sup>6</sup>The Potocsnak Longevity Institute, Northwestern University, Chicago, IL, US; <sup>7</sup>Simpson Querrey Lung Institute for Translational Science, Northwestern University, Chicago, IL, US; <sup>8</sup>Division of Pulmonary and Critical Care, Northwestern University Feinberg School of Medicine, Chicago, IL, US; <sup>9</sup>Department of Physics and Astronomy, Northwestern University, Evanston, IL, US; <sup>10</sup>Northwestern Institute on Complex Systems, Northwestern University, Evanston, IL, US; <sup>11</sup>NSF-Simons National Institute for Theory and Mathematics in Biology, Chicago, IL, US.
<h4>Conflict of Interest Disclosures</h4>
Luís A. Nunes Amaral is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.

<h4>Funding/Support</h4>
Reese A. K. Richardson was supported in part by a National Institutes of Health Training Grant (T32GM008449) through Northwestern University’s Biotechnology Training Program and receives funding from the Dr John N. Nicholson Fellowship from Northwestern University and Moderna. Spencer S. Hong receives support from the Ryan Fellowship and the International Institute for Nanotechnology at Northwestern University. Jennifer A. Byrne receives funding from a National Health and Medical Research Council of Australia Ideas grant (APP1184263) and Rewarding Research Success funding from the Faculty of Medicine and Health at The University of Sydney. Thomas Stoeger receives funding from the National Institute on Aging, Integrative Multi-Scale Systems Analysis of Gene-Expression-Driven Aging Morbidity (R00AG068544), National Institute of Allergy and Infectious Diseases (AI135964), and Successful Clinical Response In Pneumonia Therapy (SCRIPT) Systems Biology Center. Luís A. Nunes Amaral and Thomas Stoeger receive funding from SCISIPBIO: a data-science approach to evaluating the likelihood of fraud and error in published studies (1956338).
<h4>Role of the Funder/Sponsor</h4>
The funders had no role in the design or execution of this study.



technical paper

<p>Anna Abalkina,<sup>1</sup> Svetlana Kleiner<sup>2</sup></p>
<h4>Objective </h4>
<p>Paper mills pose a significant challenge to the scientific community, yet knowledge of paper mills remains fragmented. An estimated 400,000 studies originate from paper mills,<sup>1</sup> although only 56,000 have been retracted or corrected. This study investigated the patterns of paper-mill articles and the challenges of their retraction, using a sample from Tanu.pro,<sup>2</sup> one of Europe’s largest paper mills.</p>
<h4>Design</h4>
<p> Tanu.pro creates unique emails to submit papers, often leading to discrepancies between the country domains and authors’ affiliations. Employing a snowball method, the study identified more than 60 suspicious email domains, such as tanu.pro (which gave the paper mill its title in the literature). To retrieve Tanu.pro articles, we conducted a full-text search in Scopus on these domains on December 14, 2024. Combining both outsider and insider perspectives, we analyzed the retraction challenges, focusing on Tanu.pro papers published in Springer journals. A sample of 77 Springer articles was used to examine scholars’ positions and compare the results with a control group of 77 randomly selected articles without paper-mill hallmarks but with the matching indicators (year, journal, number of coauthors). The ranks and positions of all authors in both groups were examined, where possible, at the time of submission.</p>
<h4>Results </h4>
<p>The study detected 1517 papers published between 2017 and 2025 in 380 journals and coauthored by more than 4500 scholars from 46 countries and over 460 universities. While a problematic email was not definitive proof of malpractice, there was evidence that some Tanu.pro papers were associated with an author number anomaly (<b>Figure 25-1113</b>), poor empirical analysis, irrelevant citations, data fabrication, errors, fake reviewer suggestions, compromised special issues, and translated plagiarism. The Springer articles were initially identified by authors’ emails; an additional selection criterion was the presence of suggested reviewers. Tanu.pro papers had a higher proportion of scholars occupying top university positions, such as professors or department heads, than in the control group (44% vs 30%). A problematic email as the only evidence posed challenges for action by journals when other issues are absent: 20 of the 77 Tanu.pro articles (26%) could not be retracted under current Committee on Publication Ethics (COPE) guidelines. Editors also felt more reluctant to retract when the only red flags were emails and unused suggested reviewers.</p>

![](https://assets.underline.io/markdown_image/1/image/05c9b5fc3034d8ef288f41a58baaaf9c.png)

<h4>Conclusions </h4>
<p>The study has shown that specific paper mill patterns are useful when investigating problematic articles. However, some red flags are insufficient grounds for paper retraction due to limited clarity of COPE guidelines that place some articles in the gray zone if they do not feature strong grounds for retraction. More retractable categories are necessary so that COPE members can mark dubious research more easily, eg, undisclosed third-party submission. Although our recommendations are limited by the analysis of one paper mill, COPE guidelines need regular updating considering patterns of other paper mills.</p>
<h4>References</h4>
<p>1. Van Noorden R. How big is science’s fake-paper problem? <i>Nature</i>. 2023;623(7987):466-467. doi:10.1038/d41586-023-03464-x</p>
<p>2. Bishop D, Abalkina A. Paper mills: a novel form of publishing malpractice affecting psychology. <i>Meta-Psy</i>. March 17, 2024. <a href="https://open.lnu.se/index.php/metapsychology/article/view/3422">https://open.lnu.se/index.php/metapsychology/article/view/3422</a></p>
<h4>Affiliations</h4> <sup>1</sup>Freie Universität Berlin, Berlin, Germany, abalkina@gmail.com; <sup>2</sup>Springer Nature, Dordrecht, the Netherlands. <h4>Conflict of Interest Disclosures</h4> None reported.


Patterns of Paper Mill Papers and Retraction Challenges

<p>Renee V. Hoch,<sup>1</sup> Emily J. Chenette<sup>1</sup></p>
<h4>Objective </h4>
<p><i>PLOS One </i>has experienced a sharp rise in paper mills and poor-quality submissions since 2021 and has taken an agile, multifaceted preventive approach to address these issues at scale. This quality improvement study evaluated trends in large-scale integrity issues (ie, megacases) at <i>PLOS One</i> between 2021 and 2024 and the impacts of preventive initiatives on the journal’s ability to address megacase concerns prior to publication.</p>
<h4>Design</h4>
<p> We analyzed temporal and geographic trends, issue types, and research areas for megacases detected at <i>PLOS One</i> between 2021 and 2024. We then evaluated changes to desk reject rates following specific interventions as well as changes to the pre- vs postpublication impacts of megacases over time.</p>
<h4>Results</h4>
<p> Between 2021 and 2024, <i>PLOS One</i> detected 50 megacases that cumulatively involved more than 7000 submissions and publications, with concerns including paper mills (42%; 21 of 50), authorship (22%; 11 of 50), peer review (52%; 26 of 50), and policy compliance (12%; 6 of 50). Megacases spanned multiple research areas, with the highest representation in the physical sciences and engineering (42%; 21 of 50) and medicine and public health (18%; 9 of 50). Most megacases involved submissions from geographically clustered author groups; megacases were received from 19 countries, with the highest representation from Pakistan (20%; 10 of 50), China (18%; 9 of 50), and India (10%; 5 of 50). The number of megacases detected by PLOS increased from 2 (214 articles) in 2021 to 25 (4310 articles) in 2024, and during this time the journal’s desk reject rate more than doubled, from 13% (5188 of 40,548) in 2021 to 27% (16,160 of 59,425) in 2024. In 2024 alone, we rejected more than 3000 submissions flagged for megacase concerns. Desk reject data in periods before and after specific preventive interventions indicated that each of these efforts increased the journal’s ability to detect and/or reject concerning content (<b>Table 25-0967</b>
	). We quantified pre- vs postpublication megacase impacts using a preventive efficacy metric, defined as the number of submissions flagged for megacase concerns (ie, prepublication impact) divided by the number of affected publications (ie, postpublication impact). <i>PLOS One</i>’s preventive efficacy increased from 0.70 in 2022 to 9.09 in 2024; the cumulative number of affected articles increased substantially during this time, but articles of concern were increasingly rejected prepublication, and the postpublication impact actually decreased (530 affected publications in 2022 vs 427 affected publications in 2024).</p>
	
![](https://assets.underline.io/markdown_image/1/image/b1311984a7ead8b037c781f0545fc4ac.png)

<h4>Conclusions</h4>
<p> Large-scale integrity issues have risen considerably in recent years, affecting a broad array of subject areas and geographies. Through varied approaches, <i>PLOS One</i> has improved prepublication detection and rejection of content linked to megacases. This reduces the volume of unreliable publications, the resources required to address these integrity issues, and the impacts of megacases on peer review. Such scalable preventive approaches are crucial to the long-term sustainability of integrity work in an era of increasing paper mill activity.</p>
<h4>Affiliations</h4> <sup>1</sup>PLOS (Public Library of Science), Cambridge, UK, and San Francisco, CA, US, rhoch@plos.org. <h4>Conflict of Interest Disclosures</h4> Renee V. Hoch is an employee of PLOS; the head of the PLOS Publication Ethics team; a member of the Committee on Publication Ethics (COPE) Council, a United2Act Working Group, and an STM Task & Finish Working Group that developed guidance pertaining to large-scale integrity cases; a former member of the COPE Paper Mills Working Group; and a coauthor of COPE guidance on addressing large-scale paper mill issues. Emily J. Chenette is an employee of PLOS and is the editor in chief of <i>PLOS One</i> and head of PLOS editorial board services. <h4>Acknowledgment</h4> We thank the <i>PLOS One</i> Editorial and Publication Ethics teams for their work identifying and addressing large-scale cases, the PLOS editorial board management team for their help investigating and offboarding editorial board members of concern, and the PLOS Digital team for their help developing data analytics and technology tools needed for this work. Thanks also to Suzanne Farley, Rebecca Kirk, Zena Nyakoojo, and Dena Emmerson for their input on this abstract. <h4>Additional Information</h4> PLOS data and resources were used for this study, and others within PLOS Editorial provided input and contributed to the work underlying the datasets. PLOS was not otherwise involved in the reported study.



Sustainable Approaches to Upholding High Integrity Standards in the Face of Large-Scale Threats: Insights From PLOS One

<p>Malcolm Macleod<sup>1</sup></p>
<h4>Importance </h4>
<p>From the perspective of the researcher, there seems to be too much research, of often questionable quality, in too many journals, published following too little scrutiny.</p>
<h4>Observations </h4>
<p>It has become more difficult to distinguish between intrusive email solicitations from “proper” publishers and those from predatory publishers. While many employed in publishing have the most noble of intentions, the profits of publishing houses—profit drawn largely from our research budgets—continue to soar. At the same time, alternative routes of dissemination, such as preprint servers, now allow research findings to be made public, if not published, at minimal cost. Loud protestations from journals that preprints are of inferior quality because they have not been peer reviewed would carry more weight if the process of peer review was a reliable guarantee of quality and the provenance of the data presented. Despite several decades of opportunity to increase the value of journal articles by investing in these processes, most publishers have not done so. Many in my own community have been complicit in the generation of “research for publication” rather than “research for the advancement of knowledge” (where publication is a part of the process, but not the primary objective). Drivers for these behaviors come from across the research ecosystem, and the focus on journal publication as the cardinal researcher output for evaluation serves as the major enabler of exploitative publisher behaviors.</p>
<h4>Conclusions </h4>
<p>If the journal article is to have a future, this extractive behavior must stop. The added value of peer review needs to be clear and apparent, and the quality of published work must be such that research users can trust the provenance of what is reported; with sufficient information, data, and code provided so that they can check that provenance if they wish. The quality of published work is improving, in some respects, at some journals, and those who have driven these changes, often in the face of internal opposition, should be celebrated. I will close by outlining some further avenues for improvement.</p>
<sup>1</sup>Edinburgh Neuroscience, University of Edinburgh, Edinburgh, Scotland, malcolm.macleod@ed.ac.uk. <h4>Conflict of Interest Disclosures</h4> Malcom Macleod is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.



Does the Journal Article Have a Future?

<p>Jae Il Shin,<sup>1,2,3</sup> Min Seo Kim,<sup>4,5,6,7</sup> Dong Keon Yon,<sup>8</sup> Seung Won Lee,<sup>9</sup> Masoud Rahmati,<sup>10</sup> Marco Solmi,<sup>11,12,13,14,15</sup> Andre F. Carvalho,<sup>16</sup> Ai Koyanagi,<sup>17,18,19</sup> Lee Smith,<sup>20</sup> John P. A. Ioannidis<sup>21,22,23,24,25</sup></p>
<h4>Objective</h4>
<p> Immortal time bias (ITB) is the error in estimating the association between the exposure and the outcome that results from misclassification or exclusion of time intervals. The aim of our study was to estimate the prevalence of ITB in systematic reviews and meta-analyses and to assess the degree to which it contributes to effect size estimates and evidence reversal.</p>
<h4>Design</h4>
<p> We performed a systematic review of systematic reviews with meta-analyses (SRMA) that only underwent detailed analysis based on ITB. We only included SRMA of observational studies including cohort and case-control studies and excluded those without meta-analysis or SRMA of randomized control trials. We searched PubMed/MEDLINE, Embase, and Cochrane Database of Systematic Reviews from database inception to July 31, 2024. Two authors independently extracted data and evaluated the methodological quality of the systematic reviews. Information on ITB judgment and effect sizes with 95% CIs for individual studies in forest plots were extracted to run reanalysis using generic inverse variance fixed- and random-effects methods. After extracting data, we conducted subgroup analyses by the presence of ITB for all available topics and assessed the influence of ITB on the heterogeneity (<i>I</i><sup>2</sup>), changes of evidence, statistical significance of the finding, and altering effect size in favor of intervention/exposure.</p>
<h4>Results</h4>
<p> Among the 12 systematic reviews with relevant data, there were 25 eligible topics (only 21 could be divided by ITB, as 4 included only studies without ITB). The median (IQR) number of studies included for a topic was 6 (4-10). Among 182 studies among 25 topics, 44.0% (80 studies) were affected by ITB. Among the 21 topics where both studies with ITB and studies without ITB were available, 57.1% (12/21) demonstrated discordant results between ITB subgroups (<b>Figure 25-0976</b>). Evidence reversal occurred in 23.8% (5/21), where overall summary results changed from statistically significant to non–statistically significant or vice versa after excluding studies with ITB. The ratio of effect size (effect sizes pooled from studies with ITB relative to those pooled from studies without ITB) was 0.71 (95% CI, 0.66-0.78), suggesting that the effect sizes from studies with ITB were exaggerated by an average of 29% in favor of the intervention/exposure. Excluding studies involving ITB reduced the heterogeneity (<i>I</i><sup>2</sup>) of overall pooled results by 21.4% on average.</p>

![](https://assets.underline.io/markdown_image/1/image/f4e85eb7af8587aaffcff8858c101115.png)

![](https://assets.underline.io/markdown_image/1/image/1a8b75ac4280c10ed9e7883e7ed371a3.png)

<h4>Conclusions</h4>
<p> We quantitatively captured how far ITB has influenced our knowledge and clinical practices. Given the projected high prevalence and nontrivial influence of ITB, ITB should be considered in studies with survival analyses, and improving reporting standards by researchers as well as collective surveillance from readers, reviewers, and editors is warranted. Future studies should address how ITS may also interact with other trial characteristics and biases in affecting treatment effect estimates.</p>
<h4>Affiliations</h4> <sup>1</sup>Department of Pediatrics, Yonsei University College of Medicine, Seoul, Republic of Korea, shinji@yuhs.ac; <sup>2</sup>Severance Underwood Meta-Research Center, Institute of Convergence Science, Yonsei University, Seoul, Republic of Korea; <sup>3</sup>Affiliate in Meta-Research Innovation Center at Stanford, Stanford University, Stanford, CA, US; <sup>4</sup>Samsung Advanced Institute for Health Sciences & Technology, Sungkyunkwan University, Samsung Medical Center, Seoul, Republic of Korea; <sup>5</sup>Medical and Population Genetics and Cardiovascular Disease Initiative, Broad Institute, Cambridge, MA, US; <sup>6</sup>Cardiovascular Research Center, Massachusetts General Hospital, Boston, MA, US; <sup>7</sup>Department of Medicine, Harvard Medical School, Boston, MA, US; <sup>8</sup>Center for Digital Health, Medical Science Research Institute, Kyung Hee University College of Medicine, Seoul, Republic of Korea; <sup>9</sup>Department of Data Science, Sejong University College of Software Convergence, Seoul, Republic of Korea; <sup>10</sup>Department of Physical Education and Sport Sciences, Faculty of Literature and Human Sciences, Lorestan University, Khoramabad, Iran; <sup>11</sup>Department of Psychiatry, University of Ottawa, Ottawa, Ontario, Canada; <sup>12</sup>Department of Mental Health, The Ottawa Hospital, Ottawa, Ontario, Canada; <sup>13</sup>Ottawa Hospital Research Institute (OHRI) Clinical Epidemiology Program University of Ottawa, Ottawa, Ontario, Canada; <sup>14</sup>School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada; <sup>15</sup>Department of Child and Adolescent Psychiatry, Charité Universitätsmedizin, Berlin, Germany; <sup>16</sup>Innovation in Mental and Physical Health and Clinical Treatment, Strategic Research Centre, School of Medicine, Barwon Health, Deakin University, Geelong, VIC, Australia; <sup>17</sup>Research and Development Unit, Parc Sanitari Sant Joan de Déu, Universitat de Barcelona, Fundació Sant Joan de Déu, CIBERSAM, Barcelona, Spain; <sup>18</sup>ICREA, Pg. Lluis Companys 23, Barcelona, Spain; <sup>19</sup>Instituto de Salud Carlos III, Centro de Investigación Biomédica en Red de Salud Mental, CIBERSAM, Madrid, Spain; <sup>20</sup>The Cambridge Centre for Sport and Exercise Sciences, Anglia Ruskin University, Cambridge, United Kingdom; <sup>21</sup>Department of Medicine, Stanford University, Stanford, CA, US; <sup>22</sup>Department of Epidemiology and Population Health, Stanford University, Stanford, CA, US; <sup>23</sup>Department of Biomedical Data Science, Stanford University, Stanford, CA, US; <sup>24</sup>Department of Statistics, Stanford University, Stanford, CA, US; <sup>25</sup>Meta-Research Innovation Center at Stanford, Stanford University, Stanford, CA, US. <h4>Conflict of Interest Disclosures</h4> John P. A. Ioannidis is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. <h4>Funding/Support</h4> The work of John P. A. Ioannidis is supported by an unrestricted gift from Sue and Bob O’Donnell to Stanford University. Jae Il Shin is supported by the Yonsei Faculty Fellowship, funded by Lee Youn Jae. <h4>Role of the Funder/Sponsor</h4> The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. <h4>Additional Information</h4> John P. A. Ioannidis is a cocorresponding author (jioannid@stanford.edu).



Immortal Time Bias Prevalence and Influence on Estimates in Systematic Reviews and Meta-Analyses

<p>Yuanxi Jia,<sup>1</sup> Yiwen Jiang,<sup>2</sup> Karen A. Robinson,<sup>3</sup> Jinling Tang<sup>4</sup></p>
<h4>Objective</h4>
<p> Researchers conducting randomized clinical trials (RCTs) might focus more on primary outcomes than secondary outcomes. Therefore, when the design and conduct of an RCT were manipulated for positive results, the effect size of the primary outcome may be overestimated more than that of the secondary outcomes. Among RCTs with similar design, the discrepant effect estimates of the same outcome between those designating it as a primary outcome (PO-RCTs) and others designating it as a secondary outcome (SO-RCTs) may indicate the impact of bias. This study aimed to compare the effect estimates of the same outcomes between PO-RCTs and SO-RCTs. PO-RCTs were hypothesized to produce higher effect estimates than SO-RCTs.</p>
<h4>Design</h4>
<p> This meta-research study using a matched cohort design included 153 meta-analyses assessing the efficacy/effectiveness of health interventions published in the Cochrane Database of Systematic Reviews between 2021 and 2023. A total of 1073 RCTs with parallel design, individual randomization, superiority tests, recruitment in or after 2006, and negative control (placebo, sham, no intervention, waiting list, minimal intervention, attention control, or standard care) were included. Within each meta-analysis, RCTs designating the meta-analyzed outcome as primary outcome were classified as PO-RCTs (553 RCTs), while those designating the meta-analyzed outcome as secondary outcome were classified as SO-RCTs (520 RCTs). RCTs that did not define a primary outcome were excluded. The PO-RCTs were compared with the SO-RCTs using 2-stage random-effect meta-analyses: the effect estimates from RCTs were transformed into odds ratios (ORs); within each meta-analysis, the ORs from PO-RCTs and SO-RCTs were compared as a ratio of ORs (ROR); and the RORs across meta-analyses were combined. The primary analyses were conducted among all RCTs and prospectively registered RCTs. The risk of bias for blinding was assessed using the Cochrane Risk of Bias Tool.</p>
<h4>Results </h4>
<p>Among 1073 RCTs from 153 meta-analyses, PO-RCTs produced ORs 1.27 (95% CI, 1.14-1.42; <i>I</i><sup>2</sup> = 43.6%) times higher than SO-RCTs. Among 372 prospectively registered RCTs from 122 meta-analyses, PO-RCTs produced ORs 1.21 (95% CI, 1.03-1.41; <i>I</i><sup>2</sup> = 22.3%) times higher than SO-RCTs. When restricted to RCTs with high/unclear risk in performance bias, detection bias, or both types of bias, PO-RCTs produced ORs 1.48 (95% CI, 1.26-1.75), 1.44 (95% CI, 1.21-1.71), and 1.60 (95% CI, 1.30-1.96) times higher than SO-RCTs, respectively. When restricted to RCTs with low risk in performance bias, detection bias, or both types of bias, PO-RCTs produced ORs 1.08 (95% CI, 0.90-1.28), 1.04 (95% CI, 0.87-1.24), and 1.06 (95% CI, 0.90-1.26) times higher than SO-RCTs, respectively.</p>
<h4>Conclusions</h4>
<p> The effect estimate for an outcome could be 27% larger when designated as a primary outcome than a secondary outcome in RCTs, which may be partially attributed to bias. Researchers conducting systematic reviews may need to note whether an outcome was designated as primary or secondary in RCTs, perform sensitivity analyses, and interpret results considering the impact of potential bias.</p>
<sup>1</sup>Yong Loo Lin School of Medicine, National University of Singapore, Singapore, yx.jia@nus.edu.sg; <sup>2</sup>Shenzhen Institute of Advanced Technology, Shenzhen, China; <sup>3</sup>School of Medicine, Johns Hopkins University, Baltimore, US; <sup>4</sup>Shenzhen University of Advanced Technology, Shenzhen, China. <h4>Conflict of Interest Disclosures</h4> None reported. <h4>Funding/Support</h4> This study was supported by the Shenzhen Science and Technology Program (grant KQTD20190929172835662) from the Shenzhen Municipal Government, Guangdong Province, China, and the Outstanding Youth Innovation Fund from the Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences (grant E2G019). <h4>Role of the Funder/Sponsor</h4> The funders had no roles in the design, conduct, or reporting of this study.



Effect Estimates for the Same Outcomes Designated as Primary vs Secondary in Randomized Clinical Trials: A Meta-Research Study

<p>Ian Bulovic,<sup>1</sup> Susmitha Wunnava,<sup>1,2</sup> Wonjin Yoon,<sup>1,3</sup> Adam G. Dunn,<sup>1,4</sup> Florence T. Bourgeois,<sup>1,2,3</sup> Timothy Miller<sup>1,2,3</sup></p>
<h4>Objective </h4>
<p>Selective inclusion and reporting of clinical trial outcomes may result in biases in the medical evidence on benefits and harms of treatments.<sup>1</sup> ClinicalTrials.gov provides the opportunity to identify changes between prespecified outcomes and those eventually reported in publications, but linkage and comparison of these outcomes requires extensive manual curation, precluding studies of more than a few hundred trials.<sup>2,3</sup> ClinicalTrials.gov archives all versions of registration records as changes are made by investigators, providing the basis to monitor trial outcome changes over time if the information could be efficiently processed. Our objective was to use a large language model (LLM) to estimate changes to primary outcomes at scale.</p>
<h4>Design </h4>
<p>OpenAI’s Gpt-4o-2024-08-06 model was prompted (without training) to identify trials with meaningful changes in specified primary outcomes between trial registrations at study start and in the final record (prompts and scripts used for processing data available at <a href="https://github.com/Machine-Learning-for-Medical-Language/outcome-switching">https://github.com/Machine-Learning-for-Medical-Language/outcome-switching</a>. Meaningful changes were defined as addition, removal, or substantial modification to primary outcomes. Substantial modifications included changes to outcome measures, measurement procedures, or measurement time frame. The model performance was evaluated using a set of 100 manually labeled and adjudicated records, showing F1 scores of 0.88 for outcome addition, 0.81 for outcome removal, and 0.67 for outcome modification (F1 scores range from 0 to 1 and balance sensitivity and positive predictive value). The cohort consisted of interventional trials studying US Food and Drug Administration (FDA)–regulated drugs that were prospectively registered and started January 1, 2008, to December 31, 2022 (allowing ≥2 years’ follow-up). Variables of interest (start year, therapeutic area, industry funding, randomization, and enrollment size) were selected a priori and extracted from registration records using structured data available for download from ClinicalTrials.gov. Univariate analyses and multivariate logistic regression were performed to assess for associations between outcome changes and variables of interest.</p>
<h4>Results </h4>
<p>Among 27,227 trials studying FDA-regulated drugs, 11,078 (40.7%) had changes to primary outcomes after trial start. This included 3850 trials (14.1%) with addition of an outcome, 3123 (11.5%) with removal of an outcome, and 9105 (33.4%) with substantial outcome modifications. Trials with industry funding were significantly more likely to have an outcome change (47.7% vs 32.9%; <i>P</i> &lt; .001) (<b>Table 25-1101</b>). This difference was observed for each of the outcome change types, including addition (17.4% vs 10.4%; <i>P</i> &lt; .001), removal (12.9% vs 9.9%; <i>P</i> &lt; .001), and modification (39.7% vs 26.4%; <i>P</i> &lt; .001). Multivariate analysis demonstrated reduction in outcome changes over time (<i>P</i> &lt; .001) and a positive association with industry funding (<i>P</i> &lt; .001).</p>

![](https://assets.underline.io/markdown_image/1/image/baee5d16656423f10f95e3df2f048727.png)

<h4>Conclusions </h4>
<p>LLMs can be used to monitor for changes in clinical trial outcomes efficiently and at scale using information available in ClinicalTrials.gov. Outcome changes in trials of FDA-regulated drugs are common and associated with industry funding but have decreased over time. Future work should explore use of LLMs to identify outcome changes between trial registrations and published articles.</p>
<h4>References</h4>
<p>1. Turner EH, Cipriani A, Furukawa TA, Salanti G, de Vries YA. Selective publication of antidepressant trials and its influence on apparent efficacy: updated comparisons and meta-analyses of newer versus older trials. <i>PLoS Med</i>. 2022;19(1):e1003886. doi:10.1371/journal.pmed.10038862</p>
<p>2. Wang A, Menon R, Li T, et al. Has the degree of outcome reporting bias in surgical randomized trials changed? a meta-regression analysis. <i>ANZ J Surg</i>. 2023;93:76-82. doi:10.1111/ans.182733</p>
<p>3. Hinkel J, Heneghan C, Bankhead C. Selective outcome reporting in cancer studies: a scoping review. <i>medRxiv</i>. 2024:07.02.24309826. doi:10.1101/2024.07.02.24309826</p>
<h4>Affiliations</h4> <sup>1</sup>Computational Health Informatics Program, Boston Children’s Hospital, Boston, MA, US, florence_bourgeois@hms.harvard.edu; <sup>2</sup>Harvard-MIT Center for Regulatory Science, Harvard Medical School, Boston, MA, US; <sup>3</sup>Department of Pediatrics, Harvard Medical School, Boston, MA, US; <sup>4</sup>Biomedical Informatics and Digital Health, Faculty of Medicine and Health, University of Sydney, Sydney, Australia. <h4>Conflict of Interest Disclosures</h4> None reported. <h4>Funding/Support</h4> National Library of Medicine, National Institutes of Health (R01LM012976, R01LM012973). <h4>Role of the Funder/Sponsor</h4> The funder played no role in the design or reporting of the study. <h4>Additional Information</h4> Florence T. Bourgeois and Timothy Miller are co–senior authors.


Detection and Monitoring of Outcome Reporting Changes Using a Large Language Model: Application to FDA-Regulated Drug Trials

<p>Yulin Yu,<sup>1</sup> Yong-Yeol Ahn,<sup>2</sup> Daniel Romero<sup>1,2,3,4</sup></p>
<p><h4>Objective</h4>Datasets are critical in artificial intelligence (AI) research, serving as the foundation for training, evaluating, and developing models. As interest in accelerating AI-driven discoveries grows, it becomes important to examine how strategically repurposing datasets, using them for research topics different from their original context, relates to scientific outcomes. Previous research suggests that combining existing knowledge in novel ways, or recombinant novelty, is associated with greater scientific influence.<sup>1,2</sup> While recent studies have shown that combining datasets in unexpected ways correlates with impact, they have not examined how datasets interact with the topics they are used to study.<sup>3</sup> To address this, we draw on transformational creativity theory, which modifies the accepted conceptual space by altering or removing existing dimensions, such as drug repurposing. Similarly, AI datasets can be repurposed for new research topics. This study addressed 2 questions: (1) are published articles that repurpose datasets associated with higher scientific disruption or recognition? and (2) do repurposed uses of datasets influence how future articles apply the data, and is this repurpose adoption associated with greater scientific outcomes? We conceptualized scientific impact along 2 dimensions: (1) the extent to which an article is recognized and immediately used by research communities, which can be measured by citations, and (2) the degree to which the research article disrupts existing research paradigms. </p>
<p><h4>Design</h4>We analyzed 13,637 published AI articles collected from Papers With Code and linked to the OpenAlex database from 2015 to 2021. We quantified data repurposing by comparing the semantic content of each article to previous articles using the same datasets. For each article, the number of other articles using the same dataset varied (though we set a minimum threshold of at least 1 prior reuse). Articles were embedded using sentence-level language models. We assessed the relationship between repurposing and scientific disruption, defined as the extent to which an article with repurposed data is cited instead of its references, using ordinary least squares regression. We used negative binomial regression to analyze citation counts. Control variables included dataset novelty, article novelty, author information (team size and experience), publication year, citation/reference counts, and disciplinary fields. These variables help account for potential confounders; for example, more recent papers or datasets may naturally have more prior reuse by other studies. We measured repurpose adoption by comparing each article’s similarity to prior vs future works using the same datasets. A positive score suggests that future articles adopted the new topical use.</p>
<h4>Results</h4>
<p> Greater data repurposing was significantly associated with higher disruption scores, corresponding to a 10.2%-SD increase (<i>P</i> &lt; .001) (<b>Figure 25-1139</b>, A). However, articles with repurposed data were slightly negatively associated with citation counts compared with articles without repurposed data, corresponding to a 1.7%-SD decrease, but this was not statistically significant (<i>P</i> &gt; .05) (<b>Figure 25-1139</b>, B). Articles with 1-SD higher repurpose adoption were modestly more disruptive (+2%; <i>P</i> &lt; .001) and showed a stronger positive association with future citations (<i>P</i> &lt; .001). This study is limited by reliance on the OpenAlex database, which may have incomplete metadata.</p>

![](https://assets.underline.io/markdown_image/1/image/2bc5a75630407e6300c494e42d20f8b3.png)

<h4>Conclusions </h4>
<p>In this study, repurposing datasets was associated with greater disruption, though it may have initially reduced recognition. However, when novel data were adopted for reuse in future research, they were associated with both disruption and increased citations. These findings have implications for data sharing and research training aimed at promoting innovative reuse of data.</p>
<h4>References</h4>
<p>1. Uzzi B, Mukherjee S, Stringer M, Jones B. Atypical combinations and scientific impact. <i>Science</i>. 2013;342(6157):468-472. doi:10.1126/science.1240474</p>
<p>2. Leahey E, Lee J, Funk RJ. What types of novelty are most disruptive? <i>Am Sociol Rev</i>. 2023;88(3):562-597. doi:10.1177/00031224231168074</p>
<p>3. Yu Y, Romero DM. Does the use of unusual combinations of datasets contribute to greater scientific impact? <i>Proc Natl Acad Sci U S A</i>. 2024;121(15):e2318482121. doi:10.1073/pnas.2402802121 </p>
<sup>1</sup>School of Information, University of Michigan, Ann Arbor, US, yulinyu@umich.edu; <sup>2</sup>Luddy School of Informatics, Computing, and Engineering, Indiana University, Bloomington, US; <sup>3</sup>Center for the Study of Complex Systems, University of Michigan, Ann Arbor, US; <sup>4</sup>Computer Science and Engineering Division, University of Michigan, Ann Arbor, US. <h4>Conflict of Interest Disclosures</h4> None reported.



Data Repurpose in AI Studies and Scientific Outcomes

<p>Nicola Di Girolamo,<sup>1,2</sup> Reint Meursinge Reynders,<sup>3,4</sup> Ugo Di Girolamo<sup>5</sup></p>
<h4>Objective </h4>
<p>Statements like “to our knowledge, this is the first time that” are ambiguous,<sup>1</sup> but are often used when describing a potential novel aspect of an article, often in lieu of a literature search.<sup>2</sup> Although seemingly innocuous, similar paraphrases lack reproducibility and accountability. This study assessed the prevalence of the statement “to our knowledge” (TOK) in the biomedical literature and in leading medical journals in the past 75 years as well as before and after implementing an intervention to reduce such prevalence in 2 journals.</p>
<h4>Design</h4>
<p> Full texts of articles available in PubMed Central (PMC) were searched using PMC’s phrase index for paraphrases of the statement TOK. Prevalence was measured in articles available in PMC published by 6 leading medical journals (<i>The Lancet</i>,<i> NEJM</i>,<i> The BMJ</i>,<i> JAMA</i>,<i> Ann Int Med</i>,<i> PLOS Medicine</i>) and in all articles in PMC in 2 time periods: January 1950 to December 2019 and January 2020 to December 2024 by 1 author (N.D.G.). To evaluate the search strategy, all articles published in 2024 in <i>The Lancet</i> with full text available in PMC (N = 43) were manually screened for TOK and compared with results of the engine search. To evaluate how often TOK was reported, all <i>The Lancet</i> research articles published in 2024 (N = 166) were manually screened. In 2 journals, <i>Journal of Small Animal Practice</i> (<i>JSAP</i>) and <i>Journal of Exotic Pet Medicine</i> (<i>JEPM</i>), an intervention to advise authors against use of these phrases and in favor of a scoping search of the literature was implemented in 2020, and TOK prevalence was measured in 2018 and 2024.</p>
<h4>Results </h4>
<p>Among 4488 articles available in PMC published in 6 leading medical journals in 2020 to 2024, the prevalence of TOK was 21.7% (974), highest in <i>PLOS Medicine</i> (48.4%) and lowest in <i>NEJM </i>(1.3%) (<b>Table 25-1191</b>). Prevalence among these journals was 5.7% (95% CI, 4.5%-6.9%) higher than all journals included in PMC. Compared with articles published in 1950 to 2019, prevalence of TOK in the last 5 years increased 8.2% (95% CI, 6.9%-9.6%) in leading medical journals and 3.1% (95% CI, 3.1%-3.2%) in all PMC journals. The search strategy had a sensitivity of 66.7% (95% CI, 34.9%-90.1%), a specificity of 100.0% (95% CI, 88.8%-100.0%), and an overall accuracy of 90.7% (95% CI, 77.9%-97.4%). In the manually screened articles, TOK was mentioned 5 times in 1 article (0.6%), 3 times in 6 articles (3.6%), 2 times in 17 articles (10.2%), and 1 time in 57 articles (34.3%). In <i>JSAP</i> and <i>JEPM</i>, TOK prevalence in articles decreased from 37.4% to 7.6% after the intervention.</p>

![](https://assets.underline.io/markdown_image/1/image/be8f320194fc090146374050eee46028.png)

<h4>Conclusions </h4>
<p>Paraphrases of TOK have been highly prevalent in the biomedical literature for the past 75 years and continue to be prevalent, including in leading medical journals. A simple intervention can decrease the use of this statement. This study is limited to only the portion of articles available in PMC, does not include all articles published by the 6 leading medical journals, and does not differentiate between version types (eg, accepted manuscript, published article) or article types (eg, research, review, or opinion).</p>
<h4>References</h4>
<p>1. Vaisrub S. To the best of our knowledge, which is limited at best. <i>JAMA</i>. 1979;241(3):278.</p>
<p>2. Di Girolamo N, Meursinge Reynders R. On “authors’ knowledge” and contrast-enhanced ultrasonography in rabbits. <i>Vet Radiol Ultrasound</i>. 2019;60(4):371.</p>
<h4>Affiliations</h4> <sup>1</sup>Department of Clinical Sciences, College of Veterinary Medicine, Cornell University, Ithaca, NY, US, nd374@cornell.edu; <sup>2</sup>Journal of Small Animal Practice, British Small Animal Veterinary Association, Gloucestershire, UK; <sup>3</sup>Department of Oral and Maxillofacial Surgery, Amsterdam University Medical Center, University of Amsterdam, Amsterdam, the Netherlands; <sup>4</sup>private practice of orthodontics, Milan, Italy; <sup>5</sup>Compass, New York, NY, US. <h4>Conflict of Interest Disclosures</h4> Nicola Di Girolamo was an editor in chief of 2 peer-reviewed journals at the time of submission, 1 published by Elsevier and 1 by Wiley. The other authors declare no competing interests nor conflicts of interest.



Prevalence of the Statement "to Our Knowledge" and Similar Paraphrases in Current and Past Biomedical Literature

<p>Alexander Goldberg,<sup>1</sup> Ivan Stelmakh,<sup>2</sup> Kyunghyun Cho,<sup>3</sup> Alice Oh,<sup>4</sup> Alekh Agarwal,<sup>5</sup> Danielle Belgrave,<sup>6</sup> Nihar B. Shah<sup>1</sup></p>
<h4>Objective </h4>
<p>What biases and errors arise in evaluating the quality of peer reviews? We studied this question, driven by 2 primary motivations: (1) designing incentive systems for high-quality reviewing based on review quality assessments and (2) evaluating experiments within peer review processes.</p>
<h4>Design </h4>
<p>We conducted a large-scale study at the 2022 Conference on Neural Information Processing Systems (NeurIPS), a top-tier venue in machine learning, inviting reviewers, conference chairs, and authors to evaluate reviews. This study was approved by Carnegie Mellon University Institutional Review Board. Evaluators rated reviews (scores and text) on overall quality (7-point Likert scale) and individual criteria (constructiveness, coverage, understanding, and substantiation; 5-point Likert scale). To assess uselessly elongated review bias in randomized controlled trials, we created elongated versions of 10 paper reviews by adding noninformative content (duplicated instructions and copied paper abstracts), increasing length from 200-300 to 600-850 words. We randomly assigned 458 reviewers into 2 equally sized groups: the control group evaluated a short review, and the treatment group evaluated an artificially lengthened review. We used a Mann-Whitney <i>U</i> test with a test statistic that measures the proportion of paper review pairs in which the treatment (long review) was rated higher than the control (short review). To assess author outcome bias, we collected observational data from 3429 authors and 4311 paper reviewers and/or conference organizers evaluating 9870 reviews. We compared author ratings on “accept” vs “reject” reviews of their own papers, controlling for review quality by matching 418 review pairs with similar nonauthor ratings. A Mann-Whitney <i>U</i> test was conducted on these pairs. To assess interevaluator (dis)agreement, miscalibration, and subjectivity, we estimated the magnitude of various sources of error using the observational data described above.</p>
<h4>Results </h4>
<p>Evaluators exhibited a significant elongated review bias (<i>P</i> &lt; .001). Elongated reviews received a mean score of 4.29 vs. 3.73 for original reviews. Each individual criterion also showed length bias (<i>P</i> &lt; .05). Authors were positively biased toward reviews recommending acceptance (<span class="CharOverride-4"><b>Figure 24-0809</b></span>). Authors gave “reject” reviews a mean score of 3.24 compared with 4.65 on “accept” reviews. We also found a statistically significant bias in criteria scores (<i>P</i> &lt; .001 for all criteria). The magnitude of error due to inconsistency, miscalibration, and subjectivity were similar for reviews of reviews and for reviews of papers at NeurIPS.</p>

![](https://assets.underline.io/markdown_image/1/image/688e99baaf2057220b6c8658dfb72e64.png)

<h4>Conclusions </h4>
<p>Our results suggest that the various problems that exist in reviews of papers, most notably bias toward factors like length and review recommendation, also arise in reviewing of reviews. Since our experiment in 2022, our results are increasingly relevant to the evaluation of large language models in peer review settings,<sup>1,2</sup> where the length bias we established through our randomized controlled trial can bias results on the perceived efficacy of large language models.<sup>3</sup></p>
<h4>References</h4>
<p>1. Thakkar N, Yuksekgonul M, Silberg J, et al. Can LLM feedback enhance review quality? a randomized study of 20K reviews at ICLR&#160;2025. <i>arXiv</i>. Preprint posted online April 13, 2025. doi:10.48550/arXiv.2504.09737</p>
<p>2. D’Arcy M, Hope T, Birnbaum L, Downey D. MARG: multi-agent review generation for scientific papers. <i>arXiv</i>. Preprint posted online January 8, 2024. doi:10.48550/arXiv.2401.04259</p>
<p>3. Steyvers M, Tejeda H, Kumar A, et al. What large language models know and what people think they know. <i>Nature Machine Intelligence</i>. 2025;7:221-231. doi:10.1038/s42256-024-00976-7</p>
<sup>1</sup>School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, US, akgoldbe@andrew.cmu.edu; <sup>2</sup>New Economic School, Moscow, Russia; <sup>3</sup>Center for Data Science, New York University, New York, NY, US; <sup>4</sup>School of Computing, KAIST, Daejeon, South Korea; <sup>5</sup>Google Research, New York, NY; <sup>6</sup>GSK, London, UK. <h4>Conflict of Interest Disclosures</h4> Kyunghyun Cho is affiliated with Genentech. Nihar B. Shah is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. <h4>Acknowledgment</h4> We are greatly indebted to the participants of this experiment for providing evaluations of reviews, thereby helping understand the promises and challenges of evaluating review quality, and consequently also shedding light on the design of incentives and experiments in peer review.






Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Assessments

<p>Charvi Rastogi,<sup>1</sup> Xiangchen Song,<sup>2</sup> Zhijing Jin,<sup>3,4</sup> Ivan Stelmakh,<sup>5</sup> Hal Daumé III,<sup>6</sup> Kun Zhang,<sup>2</sup> Nihar B. Shah<sup>2</sup></p>
<h4>Objective </h4>
<p>Many peer-review processes in computer science involve reviewers submitting independent reviews followed by a discussion between reviewers of each article on a typed forum (online discussion board). A key policy question is whether reviewers should remain anonymous to each other. This study investigated 7 research questions (RQs): RQ 1. Do reviewers discuss more when anonymous to each other or not? RQ 2. Are decisions closer to senior or junior reviewers’ opinions across conditions? RQ 3. Are reviewers more polite when not anonymous? RQ 4. Do self-reported reviewer experiences differ? RQ 5. Do reviewers prefer one condition? RQ 6. What factors do reviewers consider important in this policy decision? RQ 7. Have reviewers experienced dishonest behavior when their identity is revealed to other reviewers?</p>
<h4>Design </h4>
<p>A randomized controlled trial was conducted in the Conference on Uncertainty in Artificial Intelligence (UAI) in 2022, where full articles (not abstracts) were reviewed. Reviewers and articles were randomly assigned to either a condition where reviewer identities were hidden from each other or one where they were visible. Reviewers were then matched to articles within each condition using a semiautomated procedure.<sup>1</sup> An anonymous survey of reviewers was also administered. The following measurements were made: RQ 1. Average posts per reviewer-article pairs were compared; test statistic: difference across conditions. RQ 2. Test statistic: difference in the fraction of articles where the reviewer closest to the final decision was senior. RQ 3. Politeness was scored 1 to 5 using a locally deployed large language model<sup>2</sup> with few-shot prompting; scores were averaged across iterations and paraphrased prompts; test statistic: normalized Mann-Whitney U test. RQ 4. Reviewers rated 5 aspects of their experience on a 5-point Likert scale; differences across conditions were tested using a normalized Mann-Whitney U test. RQ 5. Reviewers rated overall preference on a 5-point scale mapped from −2 to 2; test statistic: Cohen <i>d</i>. RQ 6. Reviewers rated the importance of 6 factors in deciding on anonymity policy, each from 1 (least important) to 6. RQ 7. Reviewers reported experience of any dishonest behavior due to reviewer identities being visible to other reviewers, with checkboxes “Yes, in UAI 2022,” “Yes, in another venue,” “Not sure,” and “No.” The test statistics also served as a measure of the effect sizes. <i>P</i> values were computed via permutation testing.</p>
<h4>Results </h4>
<p>Overall, 322 papers were reviewed under the anonymous condition (116 accepted) and 310 papers under nonanonymous (114 accepted), with exactly 289 reviewers in both conditions. There were 611 discussion posts made by reviewers in the anonymous condition and 514 in the nonanonymous condition. The results for the 7 research questions are provided in <b>Table 24-0812</b>.</p>

![](https://assets.underline.io/markdown_image/1/image/bd5b7dcf2758e3af6ff9169164e2cab0.png)

<h4>Conclusions </h4>
<p>Small but significant differences favoring anonymous discussions were found. Subsequent computer science conferences have drawn on these findings for their policy choices, with a greater inclination toward anonymity in reviewer discussions.</p>
<h4>References</h4>
<p>1. Shah N. An overview of challenges, experiments, and computational solutions in peer review (extended version). July 7, 2025. Accessed July 16, 2025. <a href="https://www.cs.cmu.edu/~nihars/preprints/SurveyPeerReview.pdf"><span class="CharOverride-10">https://www.cs.cmu.edu/~nihars/preprints/SurveyPeerReview.pdf</span></a></p>
<p>2. Chiang W, Li Z, Lin Z, et al. Vicuna: an open-source chatbot impressing GPT-4 with 90%* ChatGPT quality. LMSYSORG. March 2023. <a href="https://lmsys.org/blog/2023-03-30-vicuna/"><span class="CharOverride-10">https://lmsys.org/blog/2023-03-30-vicuna/</span></a></p>
<sup>1</sup>Google DeepMind, New York, NY, US; <sup>2</sup>Carnegie Mellon University, Pittsburgh, PA, US, nihars@cs.cmu.edu; <sup>3</sup>ETH Zurich, Zurich, Switzerland; <sup>4</sup>Max Planck Institute, Tübingen, Germany; <sup>5</sup>New Economic School, Moscow, Russia; <sup>6</sup>University of Maryland, Baltimore, MD, US. <h4>Conflict of Interest Disclosures</h4> As per author affiliations above. In addition, Zhijing Jin is going to join the University of Toronto, and Kun Zhang has a partial appointment at Mohamed bin Zayed University of Artificial Intelligence. <h4>Funding/Support</h4> ONR N000142212181, NSF 1942124, 2200410, 2229881, NIH R01HL159805. <h4>Role of Funder/Sponsor</h4> The funders played no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation. <h4>Acknowledgment</h4> This work was conducted when Charvi Rastogi and Ivan St

Downloads

Next from The 10th Peer Review Congress

Patterns of Paper Mill Papers and Retraction Challenges

Similar lecture

Misidentification of Scanning Electron Microscope Instruments in the Peer-Reviewed Materials Science and Engineering Literature

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from The 10th Peer Review Congress

Patterns of Paper Mill Papers and Retraction Challenges

Similar lecture

Misidentification of Scanning Electron Microscope Instruments in the Peer-Reviewed Materials Science and Engineering Literature

Downloads