United States

&lt;p&gt;Fangwen Zhou,&lt;sup&gt;1&lt;/sup&gt; Muhammad Afzal,&lt;sup&gt;2&lt;/sup&gt; Rick Parrish,&lt;sup&gt;1&lt;/sup&gt; Ashirbani Saha,&lt;sup&gt;3&lt;/sup&gt; Wael Abdelkader,&lt;sup&gt;1&lt;/sup&gt; R. Brian Haynes,&lt;sup&gt;1&lt;/sup&gt; Alfonso Iorio,&lt;sup&gt;1,4&lt;/sup&gt; Cynthia Lokker&lt;sup&gt;1&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Objective &lt;/h4&gt;
&lt;p&gt;Systematic reviews are considered one of the strongest levels of evidence, providing essential information for clinical guidelines and bedside practice. However, the broad spectrum of review articles, including narrative reviews and other types, complicates the identification of high-quality systematic reviews.&lt;sup&gt;1&lt;/sup&gt; This study aims to fine-tune and evaluate pretrained encoder transformers to identify high-quality systematic reviews from review articles using a large, reputable dataset.&lt;/p&gt;
&lt;h4&gt;Design &lt;/h4&gt;
&lt;p&gt;Review articles from McMaster’s Premium Literature Service (PLUS) were retrieved.&lt;sup&gt;2&lt;/sup&gt; Articles were considered rigorous if they were systematic reviews that (1) stated the clinical topic; (2) described methods, including databases searched and inclusion criteria; (3) searched ≥1 major database; (4) reported the number of retrieved, reviewed, and included articles; and (5) did not exclude randomized clinical trials for treatment, primary prevention, quality improvement, or economics reviews; or included “inception cohort” for prognosis reviews.&lt;sup&gt;3&lt;/sup&gt; Ground truth was established by research methodology experts using article full texts. Articles from 2003 to 2023 were randomly split 80:10:10 into training, validation, and testing sets. Articles in 2024 were used for external testing. A grid search of 7 pretrained models, 3 learning rates, 5 batch sizes, and 3 random seeds, with or without class weight adjustments, was conducted. Models were trained for ≤10 epochs, with an early stopping patience of 3. Titles and abstracts were used as inputs. Those longer than 512 tokens were truncated, and those shorter were padded. The model that achieved the lowest log loss on the validation set was further evaluated. A threshold of ≥0.50 was used for classification. Bootstrapping of 1000 iterations was used to estimate 95% confidence intervals.&lt;/p&gt;
&lt;h4&gt;Results &lt;/h4&gt;
&lt;p&gt;Among 40,091 reviews, 26,332 (65.7%) were considered rigorous and 31,593 were used for training. Of the 630 fine-tuned models, BioELECTRA with no class weight adjustments, learning rate 5E-5, batch size 64, and seed 2 had the lowest validation log loss of 0.1840. &lt;b&gt;Table 25-0887&lt;/b&gt; details the characteristics of the datasets and the model’s performance, achieving &amp;gt;94% area under the receiver operating characteristic curve and &amp;gt;95% sensitivity. In review articles published in 2024, there was a notable drop in specificity and mild degradation in other metrics.&lt;/p&gt;

![](https://assets.underline.io/markdown_image/1/image/6a45410551b49fc307836396dc72f212.png)

&lt;h4&gt;Conclusions &lt;/h4&gt;
&lt;p&gt;Pretrained encoder transformer models, such as BioELECTRA, demonstrate robust performance in identifying rigorous systematic reviews based on PLUS’s criteria. These models can streamline the identification of high-quality evidence, reducing manual effort and enhancing the efficiency of evidence curation. Performance degradation in articles in 2024 may be due to changes in article structure and content over time. Additionally, generalizability to other datasets is unknown. Future efforts should establish standardized benchmarking datasets for systematic reviews, develop models for tools, such as AMSTAR 2, and account for temporal data drift to better support knowledge translation.&lt;/p&gt;
&lt;h4&gt;References&lt;/h4&gt;
&lt;p&gt;1. Shaheen N, Shaheen A, Ramadan A, et al. Appraising systematic reviews: a comprehensive guide to ensuring validity and reliability. &lt;i&gt;Front Res Metr Anal&lt;/i&gt;. Published online December 21, 2023. doi:10.3389/frma.2023.1268045&lt;/p&gt;
&lt;p&gt;2. Holland J, Haynes RB; McMaster PLUS Team Health Information Research Unit. McMaster Premium Literature Service (PLUS): an evidence-based medicine information service delivered on the Web. &lt;i&gt;AMIA Annu Symp Proc&lt;/i&gt;. 2005;2005:340-344. &lt;a href=&quot;https://www.ncbi.nlm.nih.gov/pubmed/16779058&quot;&gt;&lt;span class=&quot;Hyperlink CharOverride-6&quot;&gt;https://www.ncbi.nlm.nih.gov/pubmed/16779058&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;3. Methodological criteria. McMaster University Health Information Research Unit. Accessed August 19, 2024. &lt;a href=&quot;https://hiruweb.mcmaster.ca/hkr/what-we-do/methodologic-criteria&quot;&gt;&lt;span class=&quot;Hyperlink CharOverride-6&quot;&gt;https://hiruweb.mcmaster.ca/hkr/what-we-do/methodologic-criteria&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;sup&gt;1&lt;/sup&gt;Health Information Research Unit, Department of Health Research Methods, Evidence, and Impact, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada, lokkerc@mcmaster.ca; &lt;sup&gt;2&lt;/sup&gt;Department of Computing, Faculty of Computing, Engineering and the Built Environment, Birmingham City University, Birmingham, UK; &lt;sup&gt;3&lt;/sup&gt;Department of Oncology, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada; &lt;sup&gt;4&lt;/sup&gt;Department of Medicine, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada.&lt;/p&gt;
&lt;h4&gt;Conflict of Interest Disclosures &lt;/h4&gt;
&lt;p&gt;McMaster University, a nonprofit public academic institution, has contracts through the Health Information Research Unit under the supervision of Alfonso Iorio and R. Brian Haynes. These contracts involve professional and commercial publishers to provide newly published studies, which are critically appraised for research methodology and assessed for clinical relevance as part of the McMaster Premium Literature Service (McMaster PLUS). Cynthia Lokker and Rick Parrish receive partial compensation, and R. Brian Haynes is remunerated for supervisory responsibilities and royalties. Ashirbani Saha, Fangwen Zhou, Muhammad Afzal, and Wael Abdelkader are not affiliated with McMaster PLUS.&lt;/p&gt;
&lt;h4&gt;Funding/Support&lt;/h4&gt;
&lt;p&gt; Fangwen Zhou was funded through the Mitacs Business Strategy Internship grant (IT42947) with matching funds from EBSCO Canada.&lt;/p&gt;
&lt;h4&gt;Role of the Funder/Sponsor&lt;/h4&gt;
&lt;p&gt; The funders were not involved in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.&lt;/p&gt;
&lt;h4&gt;Acknowledgment &lt;/h4&gt;
&lt;p&gt;We thank the Digital Research Alliance of Canada for its computational resources.&lt;/p&gt;


The 10th Peer Review Congress

Domain-specific Pretrained Encoder Transformers for the Identification of Methodologically Rigorous Systematic Reviews: A Retrospective Modeling Study

low-resource setting challenges

artificial intelligence and large language models

quality of the literature

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Fangwen Zhou,<sup>1</sup> Muhammad Afzal,<sup>2</sup> Rick Parrish,<sup>1</sup> Ashirbani Saha,<sup>3</sup> Wael Abdelkader,<sup>1</sup> R. Brian Haynes,<sup>1</sup> Alfonso Iorio,<sup>1,4</sup> Cynthia Lokker<sup>1</sup></p>
<h4>Objective </h4>
<p>Systematic reviews are considered one of the strongest levels of evidence, providing essential information for clinical guidelines and bedside practice. However, the broad spectrum of review articles, including narrative reviews and other types, complicates the identification of high-quality systematic reviews.<sup>1</sup> This study aims to fine-tune and evaluate pretrained encoder transformers to identify high-quality systematic reviews from review articles using a large, reputable dataset.</p>
<h4>Design </h4>
<p>Review articles from McMaster’s Premium Literature Service (PLUS) were retrieved.<sup>2</sup> Articles were considered rigorous if they were systematic reviews that (1) stated the clinical topic; (2) described methods, including databases searched and inclusion criteria; (3) searched ≥1 major database; (4) reported the number of retrieved, reviewed, and included articles; and (5) did not exclude randomized clinical trials for treatment, primary prevention, quality improvement, or economics reviews; or included “inception cohort” for prognosis reviews.<sup>3</sup> Ground truth was established by research methodology experts using article full texts. Articles from 2003 to 2023 were randomly split 80:10:10 into training, validation, and testing sets. Articles in 2024 were used for external testing. A grid search of 7 pretrained models, 3 learning rates, 5 batch sizes, and 3 random seeds, with or without class weight adjustments, was conducted. Models were trained for ≤10 epochs, with an early stopping patience of 3. Titles and abstracts were used as inputs. Those longer than 512 tokens were truncated, and those shorter were padded. The model that achieved the lowest log loss on the validation set was further evaluated. A threshold of ≥0.50 was used for classification. Bootstrapping of 1000 iterations was used to estimate 95% confidence intervals.</p>
<h4>Results </h4>
<p>Among 40,091 reviews, 26,332 (65.7%) were considered rigorous and 31,593 were used for training. Of the 630 fine-tuned models, BioELECTRA with no class weight adjustments, learning rate 5E-5, batch size 64, and seed 2 had the lowest validation log loss of 0.1840. <b>Table 25-0887</b> details the characteristics of the datasets and the model’s performance, achieving &gt;94% area under the receiver operating characteristic curve and &gt;95% sensitivity. In review articles published in 2024, there was a notable drop in specificity and mild degradation in other metrics.</p>

![](https://assets.underline.io/markdown_image/1/image/6a45410551b49fc307836396dc72f212.png)

<h4>Conclusions </h4>
<p>Pretrained encoder transformer models, such as BioELECTRA, demonstrate robust performance in identifying rigorous systematic reviews based on PLUS’s criteria. These models can streamline the identification of high-quality evidence, reducing manual effort and enhancing the efficiency of evidence curation. Performance degradation in articles in 2024 may be due to changes in article structure and content over time. Additionally, generalizability to other datasets is unknown. Future efforts should establish standardized benchmarking datasets for systematic reviews, develop models for tools, such as AMSTAR 2, and account for temporal data drift to better support knowledge translation.</p>
<h4>References</h4>
<p>1. Shaheen N, Shaheen A, Ramadan A, et al. Appraising systematic reviews: a comprehensive guide to ensuring validity and reliability. <i>Front Res Metr Anal</i>. Published online December 21, 2023. doi:10.3389/frma.2023.1268045</p>
<p>2. Holland J, Haynes RB; McMaster PLUS Team Health Information Research Unit. McMaster Premium Literature Service (PLUS): an evidence-based medicine information service delivered on the Web. <i>AMIA Annu Symp Proc</i>. 2005;2005:340-344. <a href="https://www.ncbi.nlm.nih.gov/pubmed/16779058"><span class="Hyperlink CharOverride-6">https://www.ncbi.nlm.nih.gov/pubmed/16779058</span></a></p>
<p>3. Methodological criteria. McMaster University Health Information Research Unit. Accessed August 19, 2024. <a href="https://hiruweb.mcmaster.ca/hkr/what-we-do/methodologic-criteria"><span class="Hyperlink CharOverride-6">https://hiruweb.mcmaster.ca/hkr/what-we-do/methodologic-criteria</span></a></p>
<sup>1</sup>Health Information Research Unit, Department of Health Research Methods, Evidence, and Impact, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada, lokkerc@mcmaster.ca; <sup>2</sup>Department of Computing, Faculty of Computing, Engineering and the Built Environment, Birmingham City University, Birmingham, UK; <sup>3</sup>Department of Oncology, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada; <sup>4</sup>Department of Medicine, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>McMaster University, a nonprofit public academic institution, has contracts through the Health Information Research Unit under the supervision of Alfonso Iorio and R. Brian Haynes. These contracts involve professional and commercial publishers to provide newly published studies, which are critically appraised for research methodology and assessed for clinical relevance as part of the McMaster Premium Literature Service (McMaster PLUS). Cynthia Lokker and Rick Parrish receive partial compensation, and R. Brian Haynes is remunerated for supervisory responsibilities and royalties. Ashirbani Saha, Fangwen Zhou, Muhammad Afzal, and Wael Abdelkader are not affiliated with McMaster PLUS.</p>
<h4>Funding/Support</h4>
<p> Fangwen Zhou was funded through the Mitacs Business Strategy Internship grant (IT42947) with matching funds from EBSCO Canada.</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funders were not involved in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.</p>
<h4>Acknowledgment </h4>
<p>We thank the Digital Research Alliance of Canada for its computational resources.</p>


poster

<p>Beth Waymouth,<sup>1</sup> Heather Slater,<sup>1</sup> Angharad Goode,<sup>1</sup> Katie Allin,<sup>1</sup> Maria Kowalczuk<sup>1</sup></p>
<h4>Objective</h4>
<p> As publishers increasingly adopt artificial intelligence (AI)–driven tools, understanding editor engagement with AI-assisted editorial checks is crucial for maintaining publication standards. This study examines Handling Editors’ (HEs) interaction with an AI-powered check that flags manuscripts needing further verification during peer review and allows HEs and the editorial office (EOF) to collaborate on resolutions. After observing a decline in HE engagement, a survey was conducted to uncover the underlying reasons.</p>
<h4>Design</h4>
<p> We analyzed internal data tracking interaction with the AI-supported ethics check introduced to HEs in 2022. Existing editors were emailed about this new tool and given access and instructions, including a detailed Ethics Guidance document. Newly recruited HEs were introduced to the check during onboarding webinars. Data included the final status of the ethics check, collected quarterly between September 2022 and March 2024. Engagement was defined as the resolution of ethics checks by the HE rather than the EOF. To contextualize the engagement patterns, a survey was distributed to 4627 HEs across our journal portfolio who were active in 2024 and consented to be recontacted. Data were collected for 2 weeks in December 2024, with a reminder issued after the first week.</p>
<h4>Results</h4>
<p> Initial analysis showed that editor engagement dropped from 40% (2899 of 7394 total checks) to 20% (2170 of 10,692) between September 2022 and November 2023. Engagement rose to 25% (1227 of 4766 checks) between January and March 2024; however, this increase was not statistically significant. Of the surveyed editors, 725 (16%) responded to at least the first question. Of 662 respondents, 67% amended the ethics check on at least some manuscripts, with 24% interacting with the check for every manuscript. Among surveyed editors, 33% indicated that they were unaware of the check. In the “Other” category (n = 66), 38 HEs (58%) said no ethics issues required action or manuscripts didn’t need ethical approval; 4 cited unfamiliarity with the check, and 4 responded that they did not remember. Out of 478 responses from HEs not amending the check on some or all manuscripts, 67% cited an expectation that this task fell to the EOF. This was especially true when handling higher manuscript volumes (<b>Table 25-0897</b>).</p>

![](https://assets.underline.io/markdown_image/1/image/02ba50380f883f04074a683fa93e2342.png)

<h4>Conclusions</h4>
<p> These results underscore the risk of editor disengagement when AI-assisted tools lack sufficient onboarding support and adequate training. For AI integration to be effective, publishers must employ consistent monitoring of engagement and explore ways to ensure AI tools are visible, actionable, and reliably applied across editorial checkpoints.</p>
<sup>1</sup>Frontiers Media S.A., Switzerland, maria.kowalczuk@frontiersin.org.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Heather Slater, Angharad Goode, Katie Allin, and Maria Kowalczuk are employees of Frontiers Media S.A. Beth Waymouth was an employee of Frontiers Media S.A. at the time when the study was conducted and when the abstract was written and originally submitted. Since May 2025, Beth Waymouth has been an independent researcher.</p>
<h4>Acknowledgment</h4>
<p> We would like to thank all our Handling Editors who participated in the survey. We also wish to thank our colleagues Elena Vicario and Simone Ragavooloo for their valuable suggestions and insightful comments on this project and the abstract.</p>


Strategic Insights Into Editor Engagement With AI-Assisted Tools based on Survey and Data Analysis of AI-Assisted Ethics Checks

<p>Jeremy Y. Ng,<sup>1,2,3</sup> Malvika Krishnamurthy,<sup>2,3</sup> Gursimran Deol,<sup>2,3</sup> Wid Al-Zahraa Al-Khafaji,<sup>2,3</sup> Vetrivel Balaji,<sup>4</sup> Magdalene Abebe,<sup>2,3</sup> Jyot Adhvaryu,<sup>2,3</sup> Tejas Karrthik,<sup>2,3</sup> Pranavee Mohanakanthan,<sup>2,3</sup> Adharva Vellaparambil,<sup>2,3</sup> Lex M. Bouter,<sup>5,6</sup> R. Brian Haynes,<sup>7</sup> Alfonso Iorio,<sup>7,8</sup> Cynthia Lokker,<sup>7</sup> Hervé Maisonneuve,<sup>9,10</sup> Ana Marušić,<sup>11</sup> David Moher<sup>1,12</sup></p>
<h4>Objective</h4>
<p> This study aimed to examine the attitudes and perceptions of editors in chief (EICs) of biomedical journals regarding the integration of artificial intelligence chatbots (AICs) into the scholarly publishing process. Prior research has explored AI use in publishing broadly, but limited data exist on EIC perspectives. Although AICs offer opportunities to streamline editorial tasks, such as plagiarism detection and language editing, they also introduce ethical, technical, and operational challenges. Understanding EIC perspectives is critical to shaping guidelines, policies, and training that align with the evolving role of AICs in scholarly publishing.</p>
<h4>Design</h4>
<p> We conducted a cross-sectional survey of EICs of all biomedical journals, inclusive of medical, nursing, health sciences, dentistry, nursing, public health, and pharmacology and toxicology disciplines, published by Springer Nature (including BMC), Taylor &amp; Francis, Elsevier, Wiley, and Sage. Eligible journals were identified through a combination of automated web scraping and manual verification. EICs were invited to participate in an anonymous SurveyMonkey survey conducted over 5 weeks, which included 3 follow-up reminders, from July through August 2024. The survey covered familiarity with AICs, current use, perceived benefits and challenges, and anticipated future roles. Quantitative data were analyzed using descriptive statistics, while qualitative responses were coded and thematically analyzed to identify key themes. Our protocol was registered<sup>1</sup> and followed the CHERRIES reporting guideline.<sup>2</sup></p>
<h4>Results</h4>
<p> Of 3381 EICs contacted, 510 responded (15.1% response rate), with 505 eligible participants and a completion rate of 87.0%. Most were familiar with AICs (66.7% [325 of 487]) but had not used them in editorial workflows (83.7% [401 of 479]). Perceived benefits included enhanced language and grammar support (70.8% [308 of 435]) and plagiarism screening (67.3% [294 of 437]). However, respondents expressed concerns about initial setup and training (83.9% [360 of 429]), ethical risks (80.6% [345 of 428]), and technical reliability (75.2% [322 of 428]). While only 49.6% (240 of 484) of participants reported that their journal had formal AIC policies, 89.5% (419 of 468) of respondents supported training initiatives to promote ethical and effective use. Despite limited current adoption, 78.9% (370 of 469) believed AICs will play an important role in the future of scholarly publishing, and 77.2% (363 of 470) anticipated their significance in advancing scientific research. Themes identified through thematic analysis of open-ended questions included “no AI in authorship or peer review,” referring to EICs’ reporting of current journal and publisher policy on their use, and “ethical, integrity, and privacy concerns,” referring to EIC perceptions of challenges with the use of AICs in the scholarly publishing process. Our study manuscript has been preprinted.<sup>3</sup></p>
<h4>Conclusions</h4>
<p> Biomedical journal EICs recognized the potential of AICs to enhance editorial processes but highlighted critical barriers, including ethical dilemmas, resource limitations, and insufficient policies and training. Structured interventions, including targeted training programs and robust ethical guidelines, are essential for addressing these challenges and ensuring responsible and effective integration of AICs into publishing workflows.</p>
<h4>References</h4>
<p>1. Ng JY, Krishnamurthy M, Balaji V, et al. Attitudes and perceptions of biomedical journal editors-in-chief towards the use of artificial intelligence chatbots in the scholarly publishing process: a cross-sectional survey across multiple publishers. OSF Registries. August 10, 2024. Accessed July 2, 2025. <a href="https://osf.io/xt6f2"><span class="Hyperlink CharOverride-6">https://osf.io/xt6f2</span></a></p>
<p>2. Eysenbach G. Improving the quality of web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). <i>J Med Internet Res.</i> 2004;6(3):e34. doi:10.2196/jmir.6.3.e34</p>
<p>3. Ng JY, Krishnamurthy M, Deol G, et al. Attitudes and perceptions of biomedical journal editors-in-chief towards the use of artificial intelligence chatbots in the scholarly publishing process: a cross-sectional survey. <i>medRxiv</i>. Preprint posted online May 27, 2025. doi:10.1101/2025.05.26.25328101</p>
<sup>1</sup>Centre for Journalology, Ottawa Methods Centre, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada, jeremyyng.phd@gmail.com; <sup>2</sup>Institute of General Practice and Interprofessional Care, University Hospital Tübingen, Tübingen, Germany; <sup>3</sup>Robert Bosch Center for Integrative Medicine and Health, Bosch Health Campus, Stuttgart, Germany; <sup>4</sup>Department of Computing and Software, Faculty of Engineering, McMaster University, Hamilton, Ontario, Canada;<sup> 5</sup>Department of Epidemiology and Data Sciences, Amsterdam Universities Medical Center, Amsterdam, the Netherlands, <sup>6</sup>Department of Philosophy, Faculty of Humanities, Vrije Universiteit, Amsterdam, the Netherlands; <sup>7</sup>Department of Health Research Methods, Evidence, and Impact, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada; <sup>8</sup>Department of Medicine, McMaster University, Hamilton, Ontario, Canada; <sup>9</sup>Consultant, Lyon, France; <sup>10</sup>Scientific Committee, Institute of Research and Action on Fraud and Plagiarism in Academia (IRAFPA), Geneva, Switzerland; <sup>11</sup>Department of Research in Biomedicine and Health and Center for Evidence-based Medicine, University of Split School of Medicine, Split, Croatia; <sup>12</sup>School of Epidemiology, Public Health and Preventive Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Lex M. Bouter, Ana Marušić, and David Moher are members of the Peer Review Congress Advisory Board but were not involved in the review or decision for this abstract. No other disclosures were reported.</p>


Attitudes and Perceptions of Biomedical Journal Editors in Chief Towards the Use of Artificial Intelligence Chatbots in the Scholarly Publishing Process

<p>Jeremy Y. Ng,<sup>1,2,3</sup> Daivat Bhavsar,<sup>1,2</sup> Neha Dhanvanthry,<sup>1,2</sup> Lex Bouter,<sup>4,5</sup> Teresa M. Chan,<sup>6</sup> Holger Cramer,<sup>1,2</sup> Annette Flanagin,<sup>7</sup> Alfonso Iorio,<sup>8,9</sup> Cynthia Lokker,<sup>8</sup> Hervé Maisonneuve,<sup>10,11</sup> Ana Marušić,<sup>12</sup> David Moher<sup>3,13</sup></p>
<h4>Objective </h4>
<p>Peer review is a cornerstone of scientific publishing, ensuring the rigor and credibility of research. However, it is increasingly strained by underexplored challenges, such as variability in quality, reviewer fatigue, and biases. Artificial intelligence chatbots (AICs) are emerging tools with the potential to address these challenges by automating tasks, such as identifying methodological flaws and improving language clarity. This study aimed to explore peer reviewers’ attitudes and perceptions toward the use of AICs in medical journal peer review, focusing on their benefits, challenges, and ethical implications.</p>
<h4>Design </h4>
<p>A large-scale, international, cross-sectional survey targeting peer reviewers of medical journals was conducted. Eligible participants had completed at least 1 peer review report for a MEDLINE-indexed journal within the last 24 months at the point of invitation. The names and email addresses of a complete sample of 72,847 corresponding authors who published in MEDLINE-indexed journals from September 1, 2024, to October 15, 2024, were collected. The survey was administered and data were collected via the SurveyMonkey survey platform. The survey included questions about participant demographics, familiarity with AICs, experiences with their use, and perceptions of AIC’s roles, benefits, challenges, and ethical implications in peer review. The survey included multiple-choice, yes/no, Likert scale, and open-ended questions to ensure comprehensive data collection. The survey was first piloted with a group of 7 peer reviewers to optimize question clarity and relevance. The survey was launched in April 2025 and remained open for 7 weeks in total. This study’s protocol was registered<sup>1</sup> and the CHERRIES<sup>2</sup> and STROBE<sup>3</sup> reporting guidelines were used to inform the reporting of this survey study.</p>
<h4>Results </h4>
<p>The survey received 1194 respondents from 33,388 opened email invitations in the first 4 weeks, 1018 of whom completed the survey (response rate: 3.0%). The majority (578/1018 [56.8%]) were senior-career researchers (ie, &gt;10 years of career experience). Most reported never having used an AIC for peer review purposes (707/999 [70.8%]), and notably, many (460/1007 [45.7%]) did not anticipate using AICs in the future. However, most respondents indicated interest in receiving training on AIC use for peer review tasks (587/980 [59.9%]). The most-recognized benefit of AIC use in peer review processes (ie, ”agree” or “strongly agree”) was “reduc[ing] the workload” (601/958 [62.7%]). Conversely, the most-recognized challenge was “risk producing errors or inaccuracies” (738/934 [79.0%]).</p>
<h4>Conclusions </h4>
<p>This study provides critical insights into peer reviewers’ attitudes and perceptions toward the use of AICs in the peer review process. Findings may inform the development of evidence-based guidelines and policies to ensure the ethical, transparent, and effective use of AICs, contributing to improvements in efficiency, quality, and equity in scholarly publishing.</p>
<h4>References</h4>
<p>1. Ng JY, Bhavsar D, Dhanvanthry N, et al. Attitudes and perceptions toward the use of artificial intelligence chatbots in medical journal peer review: a large-scale, international cross-sectional survey. OSF. May 21, 2025. <a href="https://osf.io/fhc2m"><span class="Hyperlink CharOverride-6">https://osf.io/fhc2m</span></a></p>
<p>2. Eysenbach G. Improving the quality of Web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). <i>J Med Internet Res</i>. 2004;6(3):e34. doi:10.2196/jmir.6.3.e34</p>
<p>3. von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP; STROBE Initiative. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. <i>The Lancet</i>. 2007;370(9596):1453-1457. doi:10.1016/S0140-6736(07)61602-X</p>
<sup>1</sup>Institute of General Practice and Interprofessional Care, University Hospital Tübingen, Tübingen, Germany, jeremyyng.phd@gmail.com; <sup>2</sup>Robert Bosch Center for Integrative Medicine and Health, Bosch Health Campus, Stuttgart, Germany; <sup>3</sup>Centre for Journalology, Ottawa Methods Centre, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada; <sup>4</sup>Department of Epidemiology and Data Science, Amsterdam University Medical Centers, Amsterdam, the Netherlands; <sup>5</sup>Department of Philosophy, Vrije Universiteit Amsterdam, Amsterdam, the Netherlands; <sup>6</sup>School of Medicine, Toronto Metropolitan University, Toronto, Ontario, Canada; <sup>7</sup><i>JAMA</i> and the JAMA Network, Chicago, IL, US; <sup>8</sup>Department of Health Research Methods, Evidence, and Impact, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada; <sup>9</sup>Department of Medicine, McMaster University, Hamilton, Ontario, Canada; <sup>10</sup>Consultant, Lyon, France; <sup>11</sup>Scientific Committee, Institute of Research and Action on Fraud and Plagiarism in Academia (IRAFPA), Geneva, Switzerland; <sup>12</sup>Department of Research in Biomedicine and Health and Center for Evidence-based Medicine, University of Split School of Medicine, Split, Croatia; <sup>13</sup>School of Epidemiology, Public Health and Preventive Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Canada.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Lex Bouter, Annette Flanagin, Ana Marušić, and David Moher are members of the Peer Review Congress Advisory Board but were not involved in the review or decision for this abstract.</p>


Attitudes and Perceptions Toward the Use of Artificial Intelligence Chatbots in Medical Journal Peer Review: A Large-Scale, International Cross-Sectional Survey

<p>Alexander Goldberg,<sup>1</sup> Ihsan Ullah,<sup>2</sup> Thanh Gia Hieu Khuong,<sup>3</sup> Benedictus Kent Rachmat,<sup>3</sup> Zhen (Zach) Xu,<sup>4</sup> Isabelle Guyon,<sup>2,3,5</sup> Nihar B. Shah<sup>1</sup></p>
<h4>Objective </h4>
<p>This study assessed the utility of a large language model (LLM) in evaluating compliance with submission standards at the 2024 Neural Information Processing Systems (NeurIPS) conference—a top-tier publication venue in artificial intelligence (AI) with a 15% to 25% acceptance rate that reviews full papers, not just abstracts. NeurIPS requires authors to complete a 15-question checklist promoting reproducibility, transparency, and ethical standards, similar to CONSORT<sup>1</sup> and STROBE.<sup>2</sup> We introduced an optional Checklist Assistant using GPT-4 (OpenAI) to provide presubmission feedback to authors on checklist accuracy.</p>
<h4>Design </h4>
<p>We conducted a cross-sectional study in which the Checklist Assistant was available to authors 8 days before the submission deadline. Authors could optionally use the Checklist Assistant, which used a general purpose third-party LLM (gpt-4-turbo-preview) to evaluate the accuracy of their responses to a 15-question checklist. For each question, the LLM received the author’s response, justification, and the full paper (including appendices), and assessed the accuracy of the response using simple prompt engineering. We conducted surveys about authors’ perceptions of the tool at registration and after using the tool. We received 539 preusage survey responses (out of 17,491 total papers submitted to the conference), 234 submissions to the Checklist Assistant, and 65 distinct postusage survey responses. To evaluate robustness of the Checklist Assistant, we manipulated responses to the Assistant to test whether it would be possible for authors to manipulate the Assistant’s scores of checklist accuracy.</p>
<h4>Results</h4>
<p> In 65 postusage surveys (<b>Figure 25-0958</b>), 46 authors found the Assistant useful and 46 indicated they would revise their papers or checklist responses based on its feedback. Authors’ expectations for the Checklist Assistant were more positive (59 of 65 positive responses on usefulness) than their perceptions after usage (46 of 65), potentially reflecting an overly optimistic outlook on the usefulness of the tool. Analysis of resubmissions indicated that authors made substantive revisions to their submissions in response to specific feedback from the LLM. Of 40 instances when authors submitted to the Assistant multiple times, they changed their answers 39 of 40 times and often increased the length of their checklist justifications significantly. Inaccuracy (20 of 52 free-form survey responses) and excessive strictness (14 of 52) were the most frequent issues flagged by authors. For more than half of submissions, the LLM recommended changes to at least 12 of 15 checklist items. Authors most commonly reported plans to improve justifications by adding detail or citations (n = 14) and clarify experimental details or data descriptions (n = 6). On 14 of 15 questions, the Assistant could be manipulated to improve accuracy scores by changing checklist content without improving the paper.</p>

![](https://assets.underline.io/markdown_image/1/image/be46bf0146e81fb335482d0967f37894.png)

<h4>Conclusions</h4>
<p> In this study, an LLM Checklist Assistant was shown to aid a small group of authors to ensure scientific rigor. However, the tool should not be used as a fully automated review tool that replaces human review.</p>
<h4><a id="_idTextAnchor004"></a>References</h4>
<p>1. Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. <i>Lancet</i>. 2001:1191-1194.</p>
<p>2. Vandenbroucke JP, et al. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. <i>Int J Surg</i>. 2014:1500-1524.</p>
<sup>1</sup>School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, US, akgoldbe@andrew.cmu.edu; <sup>2</sup>ChaLearn, Berkeley, CA, US; <sup>3</sup>University of Paris-Saclay, Paris, France; <sup>4</sup>Department of Computer Science, The University of Chicago, Chicago, IL, US; <sup>5</sup>Google DeepMind, San Francisco, CA, US.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>Nihar B. Shah is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.</p>
<h4>Funding/Support </h4>
<p>This research project has benefitted from the Microsoft Accelerating Foundation Models Research (AFMR) grant program, an INRIA Google Research grant, the ANR Chair of Artificial Intelligence HUMANIA ANR-19-CHIA-0022, NSF 1942124, and 2200410.</p>
<h4>Acknowledgment </h4>
<p>In preparing this experiment, we received advice and help from many people. We are particularly grateful to the NeurIPS 2024 organizers, including General Chair Amir Globerson; Program Chairs Danielle Belgrave, Cheng Zhang, Angela Fan, Jakub Tomczak, and Ulrich Paquet; and workflow team member Babak Rahmani, for participating in brainstorming discussions and contributing to the design. We have also received input and encouragement from Andrew McCallum of OpenReview, Anurag Acharya from Google Scholar, and Tristan Neuman from the NeurIPS board. Several volunteers contributed ideas and helped with various aspects of the preparation, including Jeremiah Liu, Lisheng Sun, Paulo Henrique Couto, Michael Brenner, Neha Nayak Kennard, and Adrien Pavao. This research project has benefited from the Microsoft AFMR grant program. We are grateful to Marc Schoenauer for supporting this effort with an INRIA Google Research grant. We acknowledge the support of ChaLearn, the ANR Chair of Artificial Intelligence HUMANIA ANR-19-CHIA-0022, NSF 1942124, and 2200410. Importantly, we are thankful to all the participants of the Checklist Assistant for volunteering to try it out and providing their valuable feedback.</p>


Use of an LLM as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experiment

<p>Vicente Amado Olivo,<sup>1</sup> Wolfgang Kerzendorf,<sup>1</sup> Nutan Chen,<sup>2</sup> Joshua V. Shields,<sup>1</sup> Bangjing Lu,<sup>1</sup> Andreas Flörs<sup>3</sup></p>
<h4>Objective</h4>
<p> The process of identifying peer reviewers is becoming more difficult due to a surge in submissions and declining acceptance rates for review invitations.<sup>1</sup> Responses to a 2024 IOP Publishing online survey suggest the peer review system is unevenly distributed, with 10% of reviewers conducting 50% of all reviews, while early-career researchers and scholars from underrepresented regions remain underused despite being willing to review.<sup>2</sup> Reviewer identification tools have been introduced to assist editors, but the tools are proprietary and closed source, limiting their accessibility and transparency to the broader scientific community. To expand the global pool of peer reviewers, open-source, AI-powered methods are needed to uniquely identify researchers within the expanding scientific literature and match them with appropriate review opportunities. Current author name disambiguation methods often rely on extensive data features, such as institutional affiliations, email addresses, or publication venues, which are not consistently available across digital libraries. We describe the development of a name disambiguation tool with fewer data features.</p>
<h4>Design</h4>
<p> Given the lack of existing author disambiguation tools for the Smithsonian Astrophysical Observatory/National Aeronautics and Space Administration Astrophysics Data System, we developed the Neural Author Name Disambiguator (NAND), a similarity-based neural network, and trained it on pairs of publications authored by individuals sharing the same name labeled with Open Researcher and Contributor ID (ORCID) identifiers. The training dataset included 2,698,778 pairs balanced between classes (0 if they shared the same ORCID and 1 if they had different ORCIDs), 553,496 unique publications, and 125,486 ORCID profiles of authors from the 2020 ORCID open data snapshot (and annual public data release). NAND was trained to disambiguate publications with minimal features (ie, author name, title, and abstract). We validated NAND performance using standard classification metrics: accuracy (percentage of correctly classified publication pairs), precision (true positives / [true positives + false positives]), recall (true positives / [true positives + false negatives]), and F1 scores (true positives / [true positives + 0.5(false positives + false negatives)]).</p>
<h4>Results</h4>
<p> NAND achieved a mean (SD) 94.64% (0.04%) accuracy on test set pairs of authors within the same name block (eg, J. Smith or Y. Wang). Mean (SD) precision, recall, and F1 scores were 96.67% (0.05%), 95.21% (0.11%), and 95.94% (0.03%), respectively.</p>
<h4>Conclusions</h4>
<p> Combining the disambiguation model with semantic expertise matching techniques could establish a practical framework for identifying qualified and willing reviewers across global institutions.<sup>3</sup> The open framework may help to reduce the burden on overused reviewers by expanding the global pool of available reviewers. While results are promising, this analysis was limited to physics publications; further validation is needed to assess generalizability to other scientific domains.</p>
<h4>References</h4>
<p>1. Publons. 2018 Global State of Peer Review. Accessed January 29, 2025. <a href="https://publons.com/static/Publons-Global-State-Of-Peer-Review-2018.pdf"><span class="Hyperlink CharOverride-6">https://publons.com/static/Publons-Global-State-Of-Peer-Review-2018.pdf</span></a></p>
<p>2. Brigham L, Brent-Jones E, Coombs A, et al. State of peer review 2024. Accessed January 29, 2025. <a href="https://ioppublishing.org/state-of-peer-review-2024-results/"><span class="Hyperlink CharOverride-6">https://ioppublishing.org/state-of-peer-review-2024-results/</span></a></p>
<p>3. Kerzendorf WE, Patat F, Bordelon D, van de Ven G, Pritchard TA. Distributed peer review enhanced with natural language processing and machine learning. <i>Nat Astronomy</i>. 2020;4:711-717. doi:10.1038/s41550-020-1038-y</p>
<sup>1</sup>Michigan State University, East Lansing, MI, US, amadovic@msu.edu; <sup>2</sup>Volkswagen Group, Wolfsburg, Germany; <sup>3</sup>GSI Helmholtzzentrum für Schwerionenforschung, Darmstadt, Germany.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> This work is supported in part by the National Science Foundation Research Traineeship Program (DGE-2152014) to Vicente Amado Olivo. Additionally, we gratefully acknowledge the European Space Agency for funding support through a traineeship for Vicente Amado Olivo.</p>
<h4>Additional Information</h4>
<p> We acknowledge the support and guidance from Markus Kissler-Patig and Jan Reerink at the European Space Agency. The authors used the free versions of ChatGPT (OpenAI) and Claude (Anthropic) for brainstorming and editing for logical flow. We take responsibility for the integrity of the content generated.</p>


Accuracy and Precision of a Neural Network Author Name Disambiguator

<p>Peiling Wang,<sup>1</sup> Dietmar Wolfram,<sup>2</sup> Scott Shumate<sup>3</sup></p>
<h4>Objective </h4>
<p>Open peer review processes promote transparency and accountability. As more publishers adopt open peer review,<sup>1</sup> accessible peer reviews become valuable corpora to develop AI tools for enhancing peer review processes.<sup>2</sup> The main purpose of peer review as stated in the Guidelines for Reviewers<sup>3</sup> is, “First, reviews provide constructive advice and recommendations to the authors on how their paper can be improved.” How can we determine the constructiveness of peer review reports? This exploratory study analyzed 11,321 peer review reports of the first version of 4100 manuscripts submitted to, and published in, <i>PeerJ</i> to identify constructive comments. Specifically, this dataset includes first reviews from November 11, 2012, to November 24, 2019. This study examines 3 research questions: (1) To what extent do reviewers provide constructive comments? (2) To what extent do reviewers use hedging to soften criticisms or recommendations? (3) Are there differences in constructive comments between signed and anonymous reviews?</p>
<h4>Design </h4>
<p>The gpt-4o-2024-08-06 model from OpenAI API was used to analyze reviews at the sentence level to identify criticisms or supportive comments about different aspects of the research, including study purpose, hypotheses, methods, results, discussion, and conclusions. A detailed system prompt (<a href="https://zenodo.org/records/15464322">https://zenodo.org/records/15464322</a>) was used with a temperature setting of 0. The model was instructed to categorize each sentence into a structured JSON output specifying section, aspects, evaluation, tone, and mode. Use of hedging tone includes phrases such as “I do not think the investigation was ‘rigorous’” or “has the potential to be much better and more relevant.” Building on the GPT analysis, we derived the following variables to measure review feature aspects, including (1) Pc, the percentage of critical sentences; (2) Ps, the percentage of supportive sentences; and (3) Ph, the percentage of hedging sentences. The constructive index (CI), considering both criticisms and supportive comments, is the percentage of sentences containing research comments.</p>
<h4>Results </h4>
<p>Of the 11,321 peer review reports, 7071 (62.5%) were anonymous and 4250 (37.5%) were signed (<b>Table 25-1055</b>). Few reports were supportive (overall median of 9%). Higher levels (27% of anonymous reviews and 25% of signed reviews) contained criticisms of the research. A similar level of hedging was observed. Significant differences were observed between the 2 groups; signed reviews showed significantly fewer criticisms and higher levels of hedging.</p>

![](https://assets.underline.io/markdown_image/1/image/c3c1db9d57bb2102d48d254f10bb4a4e.png)

<h4>Conclusions </h4>
<p>We measured the extent of research-related comments in peer review reports that authors opted to publish. Anonymous reviewers tend to be more critical. The use of hedging in reviews could obscure clear suggestions on how to improve the manuscript. This research represents an initial effort towards a model for an AI-assisted peer review system. Further analysis should fine-tune the AI algorithms for analysis of reviewer comments. In addition, authors’ perspectives on what constitutes constructive feedback warrant further study. This research had limitations. Review histories were not available for rejected manuscripts or for articles where authors opted out of publishing reviews.</p>
<h4>References</h4>
<p>1. Wolfram D, Wang P, Hembree A, Park H. Open peer review: promoting transparency in open science. <i>Scientometrics. </i>2020;125:1033-1051. doi:10.1007/s11192-020-03488-4</p>
<p>2. Wolfram D, Wang P, Abuzahra F. An exploration of referees’ comments published in open peer review journals: the characteristics of review language and the association between review scrutiny and citations. <i>Res Eval. </i>2021;30(3):314-322<i>. </i>doi:10.1093/reseval/rvab005</p>
<p>3. Technical Community on Real-Time Systems (TCRTS) of the IEEE Computer Society. Guidelines for reviewers. Accessed on May 29, 2025. <a href="https://cmte.ieee.org/tcrts/guidelines-for-reviewers"><span class="Hyperlink CharOverride-6">https://cmte.ieee.org/tcrts/guidelines-for-reviewers</span></a></p>
<sup>1</sup>University of Tennessee, Knoxville, US, peilingw@utk.edu; <sup>2</sup>University of Wisconsin, Milwaukee, US; <sup>3</sup>Austin Peay State University, Clarksville, TN, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>


An AI-assisted Analysis of Published PeerJ Open Peer Reviews

<p>Chirag J. Patel,<sup>1</sup> Arjun K. Manrai,<sup>1</sup> Randall J. Ellis,<sup>1</sup> John P. A. Ioannidis<sup>2</sup></p>
<h4>Background</h4>
<p> Large language models (LLMs) are increasingly used to synthesize biomedical evidence, yet it is unknown whether their inferred exposure-phenotype relationships mirror observational findings.<sup>1,2</sup> </p>
<h4>Objective</h4>
<p> To quantify concordance of exposure-phenotype empirical associations using National Health and Nutrition Examination Survey (NHANES) data with predictions from an LLM (ChatGPT 4o [OpenAI]).</p>
<h4>Methods</h4>
<p> We computed 115,056 exposure-phenotype partial correlations among 560 chemical, dietary, and clinical exposures and 268 phenotypes in 80,000 US adults (1999-2018 NHANES).<sup>3</sup> We then sampled 1500 pairs: 500 significant after Bonferroni correction, 500 significant only after false-discovery rate (FDR) correction, and 500 not significant—to fit within the LLM prompt window. An OpenAI-o1-aug24 model (temperature = 0) acting as a prompted “epidemiologist” was asked to predict the sign, magnitude, and <i>P</i> value of each pair and rated the evidence generated as high, trending, or borderline. Verifiability of LLM-cited references was assessed by estimating κ values and Spearman correlation.</p>
<h4>Results</h4>
<p> The LLM labeled 23 pairs as high evidence, 244 as trending evidence, and 1233 as borderline evidence. Sign concordance with NHANES was 73% for high-confidence pairs (κ = 0.49), 65% for Bonferroni pairs (κ = 0.25), 43% for FDR-only pairs (κ = 0.09), and 12% for borderline pairs (κ approximately 0) (<b>Figure 25-1123</b>). Random sign assignment yielded a κ of 0.0001. Concordance for correlation magnitudes was modest (Spearman ρ = 0.21). Permutation tests confirmed no greater-than-chance performance on nonsense data. Mechanistic explanations dominated high-confidence pairs; 41% of cited references (615 of 1500) were unverifiable.</p>

![](https://assets.underline.io/markdown_image/1/image/ca7fd3c645a90192139dbf1d062ea8bc.png)

<h4>Conclusions</h4>
<p> LLMs captured directionality for well-established exposure-phenotype associations but faltered on borderline signals, highlighting both the promise and current limits of LLM-based evidence synthesis in exposomic epidemiology.</p>
<h4>References</h4>
<p>1. Ioannidis JPA, Loy EY, Poulton R, Chia KS. Researching genetic versus nongenetic determinants of disease: a comparison and proposed unification. <i>Sci Transl Med.</i> 2009;1(7):7ps8. doi:10.1126/scitranslmed.3000247</p>
<p>2. Patel CJ, Ioannidis JPA. Studying the elusive environment in large scale. <i>JAMA</i>. 2014;311(21):2173-2174. doi:10.1001/jama.2014.4129</p>
<p>3. Patel CJ, Rehkopf DH, Leppert JT, et al. Systematic evaluation of environmental and behavioural factors associated with all-cause mortality in the United States National Health and Nutrition Examination Survey. <i>Int J Epidemiol</i>. 2013;42(6):1795-1810. doi:10.1093/ije/dyt208</p>
<sup>1</sup>Department of Biomedical Informatics, Harvard Medical School, Boston, MA, US, chirag@hms.harvard.edu; <sup>2</sup>Department of Medicine, Epidemiology and Population Health, Stanford University School of Medicine, Stanford, CA, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> John P. A. Ioannidis is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.</p>
<h4>Funding/Support</h4>
<p> This study was funded by grants R01ES032470 and U24ES036819 from the National Institutes of Environmental Health Sciences and grant R01DK137993 from the National Institute of Diabetes and Digestive and Kidney Diseases.</p>
<h4>Role of Funder/Sponsor</h4>
<p> The sponsors had no role in this study.</p>


Comparing Observational Exposure-Phenotype Correlations with Large Language Model Predictions

<p>Nicola Di Girolamo,<sup>1,2</sup> Reint Meursinge Reynders,<sup>3,4</sup> Ugo Di Girolamo<sup>5</sup></p>
<h4>Objective </h4>
<p>Consolidated Standard of Reporting of Trials (CONSORT) checklists help authors to adhere to reporting standards and allow editors and peer reviewers to ensure that critical information is reported in a randomized controlled trial (RCT) at a glance.<sup>1</sup> We pragmatically assessed the ability of 3 artificial intelligence (AI) large language model (LLM) chatbots provided with minimal instructions to properly complete a CONSORT checklist from manuscripts reporting RCTs before peer review in 2025.</p>
<h4>Design </h4>
<p>This cross-sectional study was registered on the Open Science Framework.<sup>2</sup> PDFs of manuscripts reporting RCTs before peer review that were consecutively published in <i>The BMJ</i> were extracted in reverse-chronological order for a total of 50 manuscripts. <i>The BMJ</i> granted permission for this study. In February 2025, each PDF was uploaded exactly as submitted by the authors to 3 LLMs (ChatGPT-4o, Gemini Advanced 2.0 Flash, and Claude 3.5 Sonnet), which were instructed to complete the CONSORT checklist in a table format,<sup>3</sup> including the reporting of the 37 CONSORT items and extraction of the text where the item was reported. The number of iterations and total time (seconds) required to obtain a visually acceptable table were recorded. Sensitivity, specificity, and accuracy for each LLM were calculated with human manual extraction as the reference standard. Manual extraction was performed by 1 operator (N.D.) with expertise in RCT methods. Generalized linear mixed models were built to evaluate the effect of PDF characteristics (number of pages, file size, and number of words) on the disagreements between AI and human assessment.</p>
<h4>Results </h4>
<p>Of the 1850 CONSORT items, ChatGPT provided decisions for 1776 (96.0%); 1159 of these were in agreement with human extraction (overall accuracy, 65.3% [95% CI, 63.0%-67.5%]; sensitivity, 61.3% [95% CI, 55.0%-67.3%]; specificity, 65.9% [95% CI, 63.5%-68.3%]). Gemini provided decisions for 1629 items (88.0%), of which 1521 were in agreement with human extraction (accuracy, 93.4% [95% CI, 92.1%-94.5%]; sensitivity, 97.6% [95% CI, 94.8%-99.1%]; specificity, 92.6% [95% CI, 91.1%-94.0%]). Claude provided decisions for 1480 items (80.0%), of which 1375 were in agreement with human extraction (accuracy, 92.9% [95% CI, 91.5%- 94.2%]; sensitivity, 56.1% [95% CI, 49.3%-62.8%]; specificity, 99.4% [95% CI, 98.8%-99.7%]). Accuracy of LLMs was particularly low for certain CONSORT items (<b>Figure 25-1152</b>). There were no significant associations between PDF characteristics and AI and human disagreements. Trial PDFs were a mean of 49 pages and 17,268 words. LLMs were able to compile visually acceptable CONSORT checklists on the first iteration in 76.7% (115 of 150) attempts: ChatGPT for 40 RCTs, Gemini for 36 RCTs, and Claude for 39 RCTs. Generation of a CONSORT checklist required a mean of 75 seconds, 38 seconds, and 47 seconds for ChatGPT, Gemini, and Claude, respectively (range, 16-249 seconds).</p>

![](https://assets.underline.io/markdown_image/1/image/39e42ee902f1b80f6a363027b6df1d1e.png)

<h4>Conclusions </h4>
<p>In early 2025, the LLMs Gemini and Claude, but not ChatGPT, were able to generate accurate CONSORT checklists from PDFs of manuscripts reporting RCTs before peer review in over 90% of cases, given minimal instructions and often in less than 1 minute.</p>
<h4>References<a id="_idTextAnchor008"></a><a id="_idTextAnchor009"></a></h4>
<p>1. Moher D, Jones A, Lepage L; CONSORT Group (Consolidated Standards for Reporting of Trials). Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. <i>JAMA</i>. 2001;285(15):1992-1995. doi:10.1001/jama.285.15.1992</p>
<p>2. Di Girolamo N, Meursinge Reynders R, Di Girolamo U. Comparison of different AI large language models for extraction of CONSORT items from submitted randomized controlled trials before peer-review. Open Science Framework. Registered February 13, 2025. Accessed July 5, 2025. <a href="https://osf.io/5v6h2"><span class="Hyperlink CharOverride-6">https://osf.io/5v6h2</span></a></p>
<p>3. Schulz KF, Altman DG, Moher D; CONSORT Group. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. <i>BMJ</i>. 2010;340:c332. doi:10.1136/bmj.c332</p>
<sup>1</sup>Department of Clinical Sciences, College of Veterinary Medicine, Cornell University, Ithaca, NY, US, nd374@cornell.edu; <sup>2</sup>Journal of Small Animal Practice, British Small Animal Veterinary Association, Gloucestershire, UK; <sup>3</sup>Department of Oral and Maxillofacial Surgery, Amsterdam University Medical Center, University of Amsterdam, Amsterdam, the Netherlands; <sup>4</sup>Private Practice of Orthodontics, Milan, Italy; <sup>5</sup>Compass, New York, NY, US.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>Nicola Di Girolamo is editor in chief of 2 peer-reviewed journals, one published by Elsevier and one by Wiley. No other disclosures were reported.</p>


Pragmatic Assessment of Different AI Large Language Models for Extraction of CONSORT Items From Randomized Controlled Trials Before Peer Review

<p>Josh Major,<sup>1</sup> Kurt Mahnken,<sup>1</sup> Alec Young,<sup>1</sup> Cameron O’Brien,<sup>1</sup> Andrew V. Tran,<sup>1</sup> Patrick Crotty,<sup>1</sup> Alicia Ito Ford,<sup>1,2</sup> Matt Vassar<sup>1,2</sup></p>
<h4>Objective</h4>
<p> The integration of artificial intelligence (AI) in scientific research presents both opportunities and challenges, particularly concerning transparency, ethics, and reproducibility.<sup>1</sup> This study aimed to evaluate how leading orthopedic and sports medicine (OSM) journals address 2 distinct outcomes: (1) the extent to which AI is permitted in research and publication and (2) the degree to which OSM journals are endorsing AI-specific reporting guidelines (RGs). The primary outcome was to assess the presence of AI-related policies, including requirements for AI disclosure, acceptance or prohibition of AI-generated images and content, and AI’s role in manuscript writing and authorship. Secondary outcomes examined the endorsement of AI-specific RGs and the relationship between AI policies and Journal Impact Factor.</p>
<h4>Design</h4>
<p> We conducted a cross-sectional analysis of the top 100 OSM journals ranked by the 2023 SCImago Journal Rank (SJR) indicator. Eligibility criteria included actively publishing peer-reviewed clinical journals with publicly accessible Instructions for Authors in English language. On September 29, 2024, 2 investigators independently screened, extracted, and reconciled data in a masked, duplicate manner. Data were extracted from journals’ Instructions for Authors pages and affiliated publisher websites where indicated. If a journal had no mention of AI policies or AI-specific RGs, the editorial teams were contacted directly to inquire about these gaps. Biserial correlation analyses in R version 4.4.1 (The R Foundation) and RStudio (Posit) examined AI policy relationships with Journal Impact Factor and SJR.</p>
<h4>Results</h4>
<p> The initial search yielded 319 journals, from which the top 100 were analyzed. Of these, 94 mentioned AI in their Instructions for Authors, with all 94 requiring disclosure of AI use and prohibiting AI as an author (<b>Table 25-1175</b>). Additionally, AI-assisted manuscript writing was permitted in all cases, while 87% allowed AI-assisted content generation and 64% permitted AI-generated images. Despite this widespread recognition of AI policies, AI-specific RGs were endorsed in only 1% of journals, recommending the use of the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) RG. No significant associations were observed between Journal Impact Factor and the presence of AI-related editorial policies. Additionally, no differences were found among the AI policies in the 4 quartiles of the OSM journals in this study.</p>
<a href="/wp-content/uploads/2025_conference/tables/prc25-1175t1.png"> <img loading="lazy" src="/wp-content/uploads/2025_conference/tables/prc25-1175t1.png"> </a>
<h4>Conclusions</h4>
<p> While some AI use in research is widely acknowledged by OSM journals, the lack of consistency in policies regarding AI-generated images and content suggest an area for improvement to standardize how AI can be applied in these journals. Furthermore, the lack of endorsement for AI-specific RGs suggests a critical gap in ensuring the transparency and methodological rigor of AI-integrated research. To address these issues, we recommend that OSM journals establish clear AI policies and endorse AI-specific RGs. The International Committee of Medical Journal Editors, Committee on Publication Ethics, and World Association of Medical Editors have taken initial steps; however, implementation may be difficult.<sup>2</sup> Implementing these measures will bridge existing gaps, promote transparency, and improve research quality in scientific publishing.</p>
<h4>References</h4>
<p>1. Prasana P, Mandal PK, Hussain D, et al. Intersection of orthopaedics and artificial intelligence: a review. <i>SSR Inst Int J Life Sci</i>. 2024;10(3):5544-5552. doi:10.21276/ssr-iijls.2024.10.3.21</p>
<p>2. Simera I, Moher D, Hoey J, Schulz KF, Altman DG. The EQUATOR Network and reporting guidelines: helping to achieve high standards in reporting health research studies. <i>Maturitas</i>. 2009;63(1):4-6. doi:10.1016/j.maturitas.2009.03.011</p>
<sup>1</sup>Office of Medical Student Research, Oklahoma State University Center for Health Sciences, Tulsa, OK, US, youngalec9.r@gmail.com; <sup>2</sup>Department of Psychiatry and Behavioral Sciences, Oklahoma State University Center for Health Sciences, Tulsa, OK, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Alicia Ito Ford reports receipt of funding from the Center for Integrative Research on Childhood Adversity, the Oklahoma Shared Clinical and Translational Resources, and internal grants from Oklahoma State University and Oklahoma State University Center for Health Sciences outside of the present work. Matt Vassar reports receipt of funding from the National Institute on Drug Abuse, the National Institute on Alcohol Abuse and Alcoholism, the US Office of Research Integrity, Oklahoma Center for Advancement of Science and Technology, and internal grants from Oklahoma State University Center for Health Sciences outside of the present work. No other disclosures were reported.</p>
<h4>Additional Information</h4>
<p> Andrew V. Tran is a co–corresponding author (andrewtranresearch@gmail.com).</p>


Artificial Intelligence Editorial Policies and Reporting Standards in Orthopedic and Sports Medicine Journals

<p>Heather Goodell,<sup>1</sup> Christine Beaty,<sup>1</sup> Jonathan Schultz,<sup>1</sup> Shilpi Mehra,<sup>2</sup> Chirag Jay Patel<sup>3</sup></p>
<h4>Objective</h4>
<p> The increase in abstract submissions to major scientific conferences requires efficient and reliable methods to ensure compliance with research integrity standards, which are essential for maintaining the credibility of science. Traditional review methods may not fully capture key issues, requiring a structured approach to evaluate submissions before final acceptance. This study presents a review process that uses artificial intelligence (AI) with human validation to assess the integrity of abstracts submitted to the American Heart Association (AHA) Scientific Sessions.</p>
<h4>Design</h4>
<p> An integrity evaluation process was applied by combining AI-driven analysis with human review of abstracts submitted for the 2024 AHA Scientific Sessions. An AI tool (Paperpal Preflight for Editorial Desk) was used to identify research integrity issues, including AI-generated text, author expertise misalignment, and reference accuracy. Subject matter experts then reviewed abstracts flagged for validation. AI checks categorized abstracts into <i>passed</i>, <i>warning</i>, or <i>critical</i> levels, with those not marked as <i>passed</i> requiring further human review.</p>
<h4>Results</h4>
<p> Among the 8477 submitted abstracts that were analyzed, 42 were flagged with a warning for integrity concerns (AI-generated text, author expertise misalignment, and reference accuracy); human reviewers cleared 3 of these. Further analysis revealed that 167 authors submitted 15 or more abstracts, 63 authors submitted 20 or more, and 13 authors submitted more than 30 abstracts. A total of 2624 abstracts had at least 1 author with 10 or more submissions, and 1438 abstracts had at least 1 author with 15 or more submissions. The number of submissions was considered when assessing research integrity concerns. Analysis of author submission counts helped flag questionable cases for closer review and the identification of potentially suspect activities.</p>
<h4>Conclusions</h4>
<p> The hybrid AI-human review process for abstract submissions was demonstrated to be an efficient and accurate evaluation method that should be scalable. By addressing concerns related to author submission volume and content errors, this approach can strengthen conference proceedings and support broader efforts to enhance research quality and trust in scientific dissemination. Future efforts will focus on refining and customizing AI models and optimizing reviewer workflows to further enhance this process.</p>
<sup>1</sup>American Heart Association, Dallas, TX, US; <sup>2</sup>Cactus Communications Pvt Ltd, Mumbai, India; <sup>3</sup>Cactus Communications Inc, Princeton, NJ, US; chirag.patel@cactusglobal.com.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Heather Goodell, Christine Beaty, and Jonathan Schultz are employed by the American Heart Association. Shilpi Mehra and Chirag Jay Patel are employed by Cactus Communications, which owns Paperpal Preflight for Editorial Desk. No other disclosures were reported.</p>


Next from The 10th Peer Review Congress

Strategic Insights Into Editor Engagement With AI-Assisted Tools based on Survey and Data Analysis of AI-Assisted Ethics Checks

Similar lecture

Understanding How a Language Model Assesses the Quality of Randomized Controlled Trials: Applying Shapley Additive Explanations to Encoder Transformer Classification Models