United States

&lt;p&gt;Mengyuan Fu,&lt;sup&gt;1,2&lt;/sup&gt; Kexin Ling,&lt;sup&gt;3&lt;/sup&gt; Xinyi Zhou,&lt;sup&gt;3&lt;/sup&gt; Sneha Dave,&lt;sup&gt;4&lt;/sup&gt; Can Li,&lt;sup&gt;3&lt;/sup&gt; Luwen Shi,&lt;sup&gt;2,3&lt;/sup&gt; Xiaodong Guan,&lt;sup&gt;2,3&lt;/sup&gt; Joseph S. Ross&lt;sup&gt;1,5,6&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Objective &lt;/h4&gt;
&lt;p&gt;This study aims to examine whether the information sources cited within National Organization for Rare Disorders (NORD) rare disease reports were publicly available, as well as any associated costs to purchase access, given that patients, caregivers, and clinicians may rely on these sources for critical information to inform their clinical decision-making.&lt;/p&gt;
&lt;h4&gt;Design &lt;/h4&gt;
&lt;p&gt;We randomly selected 20% of all NORD rare disease reports available as of February 2024.&lt;sup&gt;1&lt;/sup&gt; Citations within these reports were classified into 5 source types, including publications in scholarly journals (research articles, case reports, reviews), textbooks, and other internet resources. Each citation was searched using Google and categorized as open access (freely available online), accessible at cost, or unavailable online. For citations with associated costs to purchase access, we recorded the lowest listed price before taxes. A 10% cross-sample validation was conducted between each pair of researchers to check consistency. All data were collected between April and May, 2024. We used descriptive statistics to calculate the proportion of citations categorized by availability, overall and by year and source type, as well as median costs, overall and by source type. We used χ² tests to evaluate differences in open access rates by year and source type. A 2-sided &lt;i&gt;P&lt;/i&gt; &amp;lt; .05 was considered statistically significant. Analyses were performed using Stata MP version 16.0.&lt;/p&gt;
&lt;h4&gt;Results&lt;/h4&gt;
&lt;p&gt; Among 1333 NORD rare disease reports available as of February 2024, 267 were randomly selected for study. These 267 reports included 4445 citations to 3506 unique information sources; the median (IQR) number of citations per report was 15 (11-20), including 9 (5-13) research articles, 0 (0-1) case reports, 0 (0-0) review articles, 1 (0-3) textbook, and 2 (1-4) other internet resources. Among 3506 unique information sources cited, 1750 (49.9%) were open access, 1381 (39.4%) were accessible at cost, and 375 (10.7%) were unavailable online (&lt;b&gt;Table 25-0964&lt;/b&gt;). Among citations accessible only at cost, the median (IQR) cost was $31.5 ($15.0-$40.0), whereas the total cost per report for all citations accessible only at cost was $167.7 ($82.0-$258.7). The overall proportion of citations published in scholarly journals available as open access increased from 30.0% (258/860) prior to 2001 to 79.8% (95/119) from 2021 onwards (&lt;i&gt;P&lt;/i&gt; &amp;lt; .001).&lt;/p&gt;

![](https://assets.underline.io/markdown_image/1/image/9f667019d212a0e80f968da918f39cdb.png)

&lt;h4&gt;Conclusions &lt;/h4&gt;
&lt;p&gt;While NORD should be applauded for making its rare disease reports publicly available, we found that half of the information sources cited by these reports were not open access, potentially limiting equitable access to critical information for rare disease communities.&lt;/p&gt;
&lt;h4&gt;Reference&lt;/h4&gt;
&lt;p&gt;1. National Organization for Rare Disorders. Rare disease database. Accessed March 1, 2024. &lt;a href=&quot;https://rarediseases.org/rare-diseases/&quot;&gt;&lt;span class=&quot;Hyperlink CharOverride-6&quot;&gt;https://rarediseases.org/rare-diseases/&lt;/span&gt;&lt;/a&gt;&lt;/p&gt;
&lt;sup&gt;1&lt;/sup&gt;Department of Internal Medicine, Yale School of Medicine, New Haven, CT, US, joseph.ross@yale.edu; &lt;sup&gt;2&lt;/sup&gt;International Research Center for Medicinal Administration, Peking University, Beijing, China; &lt;sup&gt;3&lt;/sup&gt;Department of Pharmacy Administration and Clinical Pharmacy, School of Pharmaceutical Sciences, Peking University, Beijing, China; &lt;sup&gt;4&lt;/sup&gt;Generation Patient, Indianapolis, IN, US; &lt;sup&gt;5&lt;/sup&gt;Center for Outcomes Research and Evaluation, Yale–New Haven Health System, New Haven, CT, US; &lt;sup&gt;6&lt;/sup&gt;Department of Health Policy and Management, Yale School of Public Health, New Haven, CT, US.&lt;/p&gt;
&lt;h4&gt;Conflict of Interest Disclosures &lt;/h4&gt;
&lt;p&gt;Mengyuan Fu currently receives research support from the National Natural Science Foundation of China (72304011). Sneha Dave is the founder and executive director of Generation Patient, an organization that receives grant funding from Arnold Ventures, Commonwealth Fund, Disability Inclusion Fund, Third Wave Fund, Responsible Technology Youth Power Fund, the Helmsley Charitable Trust, Patient-Centered Outcomes Research Institute, and Lucile Packard Foundation for Children’s Health. Joseph S. Ross currently receives research support through Yale University from Johnson and Johnson to develop methods of clinical trial data sharing, from the Food and Drug Administration for the Yale-Mayo Clinic Center for Excellence in Regulatory Science and Innovation (CERSI) program (U01FD005938), from the Agency for Healthcare Research and Quality (R01HS022882), and from Arnold Ventures; formerly received research support from the Medical Device Innovation Consortium as part of the National Evaluation System for Health Technology (NEST) and from the National Heart, Lung and Blood Institute of the National Institutes of Health (NIH) (R01HS025164, R01HL144644); was an expert witness at the request of Relator’s attorneys, the Greene Law Firm, in a qui tam suit alleging violations of the False Claims Act and Anti-Kickback Statute against Biogen Inc. that was settled in September 2022; and is a deputy editor for JAMA. No other disclosures were reported.&lt;/p&gt;


The 10th Peer Review Congress

[V] Public Access to Information Cited in Rare Disease Reports

online publishing

open and public access

dissemination of information

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Mengyuan Fu,<sup>1,2</sup> Kexin Ling,<sup>3</sup> Xinyi Zhou,<sup>3</sup> Sneha Dave,<sup>4</sup> Can Li,<sup>3</sup> Luwen Shi,<sup>2,3</sup> Xiaodong Guan,<sup>2,3</sup> Joseph S. Ross<sup>1,5,6</sup></p>
<h4>Objective </h4>
<p>This study aims to examine whether the information sources cited within National Organization for Rare Disorders (NORD) rare disease reports were publicly available, as well as any associated costs to purchase access, given that patients, caregivers, and clinicians may rely on these sources for critical information to inform their clinical decision-making.</p>
<h4>Design </h4>
<p>We randomly selected 20% of all NORD rare disease reports available as of February 2024.<sup>1</sup> Citations within these reports were classified into 5 source types, including publications in scholarly journals (research articles, case reports, reviews), textbooks, and other internet resources. Each citation was searched using Google and categorized as open access (freely available online), accessible at cost, or unavailable online. For citations with associated costs to purchase access, we recorded the lowest listed price before taxes. A 10% cross-sample validation was conducted between each pair of researchers to check consistency. All data were collected between April and May, 2024. We used descriptive statistics to calculate the proportion of citations categorized by availability, overall and by year and source type, as well as median costs, overall and by source type. We used χ² tests to evaluate differences in open access rates by year and source type. A 2-sided <i>P</i> &lt; .05 was considered statistically significant. Analyses were performed using Stata MP version 16.0.</p>
<h4>Results</h4>
<p> Among 1333 NORD rare disease reports available as of February 2024, 267 were randomly selected for study. These 267 reports included 4445 citations to 3506 unique information sources; the median (IQR) number of citations per report was 15 (11-20), including 9 (5-13) research articles, 0 (0-1) case reports, 0 (0-0) review articles, 1 (0-3) textbook, and 2 (1-4) other internet resources. Among 3506 unique information sources cited, 1750 (49.9%) were open access, 1381 (39.4%) were accessible at cost, and 375 (10.7%) were unavailable online (<b>Table 25-0964</b>). Among citations accessible only at cost, the median (IQR) cost was $31.5 ($15.0-$40.0), whereas the total cost per report for all citations accessible only at cost was $167.7 ($82.0-$258.7). The overall proportion of citations published in scholarly journals available as open access increased from 30.0% (258/860) prior to 2001 to 79.8% (95/119) from 2021 onwards (<i>P</i> &lt; .001).</p>

![](https://assets.underline.io/markdown_image/1/image/9f667019d212a0e80f968da918f39cdb.png)

<h4>Conclusions </h4>
<p>While NORD should be applauded for making its rare disease reports publicly available, we found that half of the information sources cited by these reports were not open access, potentially limiting equitable access to critical information for rare disease communities.</p>
<h4>Reference</h4>
<p>1. National Organization for Rare Disorders. Rare disease database. Accessed March 1, 2024. <a href="https://rarediseases.org/rare-diseases/"><span class="Hyperlink CharOverride-6">https://rarediseases.org/rare-diseases/</span></a></p>
<sup>1</sup>Department of Internal Medicine, Yale School of Medicine, New Haven, CT, US, joseph.ross@yale.edu; <sup>2</sup>International Research Center for Medicinal Administration, Peking University, Beijing, China; <sup>3</sup>Department of Pharmacy Administration and Clinical Pharmacy, School of Pharmaceutical Sciences, Peking University, Beijing, China; <sup>4</sup>Generation Patient, Indianapolis, IN, US; <sup>5</sup>Center for Outcomes Research and Evaluation, Yale–New Haven Health System, New Haven, CT, US; <sup>6</sup>Department of Health Policy and Management, Yale School of Public Health, New Haven, CT, US.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>Mengyuan Fu currently receives research support from the National Natural Science Foundation of China (72304011). Sneha Dave is the founder and executive director of Generation Patient, an organization that receives grant funding from Arnold Ventures, Commonwealth Fund, Disability Inclusion Fund, Third Wave Fund, Responsible Technology Youth Power Fund, the Helmsley Charitable Trust, Patient-Centered Outcomes Research Institute, and Lucile Packard Foundation for Children’s Health. Joseph S. Ross currently receives research support through Yale University from Johnson and Johnson to develop methods of clinical trial data sharing, from the Food and Drug Administration for the Yale-Mayo Clinic Center for Excellence in Regulatory Science and Innovation (CERSI) program (U01FD005938), from the Agency for Healthcare Research and Quality (R01HS022882), and from Arnold Ventures; formerly received research support from the Medical Device Innovation Consortium as part of the National Evaluation System for Health Technology (NEST) and from the National Heart, Lung and Blood Institute of the National Institutes of Health (NIH) (R01HS025164, R01HL144644); was an expert witness at the request of Relator’s attorneys, the Greene Law Firm, in a qui tam suit alleging violations of the False Claims Act and Anti-Kickback Statute against Biogen Inc. that was settled in September 2022; and is a deputy editor for JAMA. No other disclosures were reported.</p>


poster

<p>Lesley Uttley,<sup>1</sup> Louise Falzon<sup>1</sup></p>
<h4>Objective </h4>
<p>This study aimed to identify current challenges in peer review relating to research culture using a targeted thematic analysis of research culture in health and biomedical research from a wider scoping review.<sup>1</sup></p>
<h4>Design </h4>
<p>To understand challenges of peer review influenced by research culture, the literature searches of a wider scoping review<sup>1</sup> were subject to a focused qualitative thematic analysis relating specifically to peer review, following PRISMA-ScR (scoping review) guidelines<sup>2</sup> and adhering to Nowell et al’s<sup>3</sup> trustworthiness criteria for thematic analysis. PubMed and Web of Science were searched for empirical studies published between January 2012 and April 2024 in health and biomedical research culture. Eligible articles for this study were those that reported relevance to peer review (regarding manuscript publication or funding applications) in the title or abstract. Editorials, discussion pieces without data, and articles relating to peer review for conferences were excluded. The main outcome was the qualitative themes from the included articles that were associated with peer review. Relevant papers were subjected to inductive content analyses by 2 authors (L.U. and L.F.), which involved noting key findings to identify recurring themes relating to the research culture topic or finding discussed. The themes were then deductively categorized according to a framework for research integrity across 4 domains: inclusivity, transparency, rigor, and objectivity.<sup>1</sup></p>
<h4>Results </h4>
<p>Fifty-three papers were included from a total of 1680 citations generated by literature searches. Relevant themes (n = 20) regarding the challenges of peer review, ordered by prevalence, were reviewed for thematic content across the 4 domains (<span class="CharOverride-4"><b>Figure 25-0850</b></span>). Inclusivity included gender biases (n = 8), status bias (n = 2), and diversity of peer review panels (n = 5); transparency included clarity of journal expectations (n = 11), open peer review advocacy (n = 4), and blinding of author characteristics (n = 6); rigor included predatory journals (n = 3), prevention of spin (n = 4), lack of statistical examination (n = 2), potential benefits of artificial intelligence (n = 3), need for training in peer review (n = 3), and potential harms from artificial intelligence (n = 1); and objectivity included willingness/refusal/lack of incentive (n = 4), burden (n = 5), influence of conflicts of interest of reviewers (n = 3), unprofessional comments (n = 1), self-citation bias (n = 2), novelty or positivism bias (n = 2), confirmation bias against innovative research (n = 1), and homophily (preference for shared characteristics of authors) or cronyism (n = 2).</p>

![](https://assets.underline.io/markdown_image/1/image/f39da803064b7591577d5d22f43931f3.png)

<h4>Conclusions </h4>
<p>A variety of research culture influences underpin peer review challenges in the traditional academic publishing model. Themes pertain to 4 cornerstones of research integrity<sup>1</sup> relating to inclusive workforces, transparency of processes, use of best practice methods, and the objectivity (and incentives) of peer review. Emerging threats from technological innovations in conjunction with incentives fuelled by research culture are likely shaping the research landscape. Sustained and joined-up conversations between authors, publishers, and reviewers about potential peer review innovations are required to safeguard genuine research and aid the dissemination of findings that promise authentic advancements in scientific research.</p>
<h4>References</h4>
<p>1. Uttley L, Falzon L, Byrne JA, et al. Research culture influences in health and biomedical research: rapid scoping review and content analysis. <i>J Clin Epidemiol</i>. 2024;178:111616. doi:10.1016/j.jclinepi.2024.111616</p>
<p>2. Tricco AC, Lillie E, Zarin W, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. <i>Ann Intern Med</i>. 2018;169(7):467-473. doi:10.7326/M18-0850</p>
<p>3. Nowell LS, Norris JM, White DE, Moules NJ. Thematic analysis: striving to meet the trustworthiness criteria. <i>Int J Qual Methods</i>. 201728;16(1). doi:10.1177/1609406917733847</p>
<sup>1</sup>School of Medicine and Population Health, University of Sheffield, UK, l.uttley@sheffield.ac.uk.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>None reported.</p>
<h4>Funding/Support</h4>
<p> Lesley Uttley is supported by a grant from the UKRI Medical Research Council (MR/Z504063/1).</p>


[V] Research Culture Influences in Peer Review: A Targeted Thematic Analysis of Current Challenges in Peer Review

<p>Anne G. Dudley,<sup>1</sup> George Koch,<sup>2</sup> Kyle Rose,<sup>3</sup> Roei Golan,<sup>4</sup> Jennifer Regala,<sup>5</sup> Casey Seideman,<sup>6</sup> Amanda North,<sup>7</sup> Kevin Koo,<sup>8</sup> Kevan Sternberg,<sup>9</sup> Gina Badalato,<sup>10</sup> Benjamin Dropkin,<sup>11</sup> Nicholas Chakiryan,<sup>12</sup> Robert Siemens,<sup>13</sup> Peter Clark,<sup>14</sup> Andrew Harris<sup>11</sup> </p>
<h4>Objective</h4>
<p> Peer review is a critical aspect of academic publishing, yet the process takes significant time and energy for the reviewer and is a voluntary activity. Current surveys report high levels of urologist burnout, and recent events, including the COVID-19 pandemic, have led to a shift toward personal priorities outside of work potentially limiting reviewer pools. Within urology, editors report difficulty finding appropriate numbers of peer reviewers for submitted manuscripts. We sought to assess motivations to participate in the peer review process within a pool of recent <i>Journal of Urology</i> reviewers. </p>
<h4>Design</h4>
<p> <i>The Journal of Urology</i> partnered with members of the American Urologic Association publications team to develop and administer a web-based survey to a diverse group of reviewers from September 1 to December 31, 2023. All authors and reviewers over the preceding 3 years were invited to participate. The survey addressed various aspects including career stage, their experience as reviewers, and peer review process challenges, incentives, motivators, and feedback needs. </p>
<h4>Results</h4>
<p> Respondents (n = 275) completed an average of 9 reviews in the past 12 months and reported 16 years of experience as reviewers. Most reviewers were experienced urologists less than 11 years from training (64% [176]) with only 7% (18) currently in training (resident/fellow). Time emerged as a key variable with 86% (236) of respondents declining additional reviews due to time constraints. A total of 67% (184) of respondents reported reviewing time was worthwhile, yet only 35% (96) felt appropriately recognized for time and effort, and 55% (151) reported incentives would increase time spent on a peer review. Motivations to review included “to give back” (80% [220]), “to learn” (71% [195]), “to get involved” (61% [168]), and “to grow my career” (39% [107]). Most respondents (91% [250]) read other reviewers’ reviews to learn. When asked to select specific incentives to review more papers, American Urologic Association products such as waived meeting fees and membership were highly valued (64% [176]; 62% [170]), followed by recognition by local department leadership (43% [118]) and money (40% [109]). Only 14% (38) of respondents desired gear or swag, and only 23% (63) desired to be named in the journal alongside the manuscript.</p>
<h4>Conclusions</h4>
<p> Peer review motivations are diverse and suggest that urologists participate for professional development and an ongoing desire to learn and participate in the field as a whole. Study limitations include nonresponder bias, limited survey period, and lack of granular data on personal and professional motivators. Time remains an important constraint, but incentives may increase allocated time for academic pursuits. Professional meeting/membership fee waivers may be motivators to increase participation. Local efforts to recognize reviewers within departments may work synergistically to increase available reviewers and fulfill career development goals.</p>
<sup>1</sup>Connecticut Children’s, Hartford, CT, US, annedudleymd@gmail.com; <sup>2</sup>The Ohio State University Wexner Medical Center, Columbus, OH, US; <sup>3</sup>Ochsner Medical Center, New Orleans, LA, US; <sup>4</sup>Florida State University School of Medicine, Gainesville, FL, US; <sup>5</sup>Wolters Kluwer Health, Baltimore, MD, US; <sup>6</sup>Doernbecher Children’s Hospital at OHSU, Portland, OR, US; <sup>7</sup>The Children’s Hospital at Montefiore, Bronx, NY, US; <sup>8</sup>Mayo Clinic College of Medicine and Science, Rochester, MN, US; <sup>9</sup>Northwestern Medical Center, Chicago, IL, US; <sup>10</sup>Columbia University, New York, NY, US; <sup>11</sup>University of Kentucky, Lexington, KY, US; <sup>12</sup>H Lee Moffitt Cancer Center, Tampa, FL, US; <sup>13</sup>Queen’s University, Kingston, ON, Canada; <sup>14</sup>Levine Cancer Institute, Charlotte, NC, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Acknowledgment</h4>
<p> We thank Martha Keyes and the Journal of Urology publications staff for their assistance with this initiative. </p>


[V] Motivations to Participate in the Peer Review Process at the Journal of Urology

<p>Bahar Mehmani,<sup>1</sup> Silvia Dobre,<sup>1</sup> Ramadurai Petchiapppan<sup>1</sup> </p>
<h4>Objective</h4>
<p> To encourage evidence-based peer review studies, Elsevier launched the Peer Review Workbench (PRW) in 2022. Here we report on the studies that have since worked in PRW, as we believe peer review studies at large scale are needed for the progress of research on research and understanding how to improve the quality of the peer review process. </p>
<h4>Design</h4>
<p> PRW is a curated and anonymized dataset of all Elsevier journal manuscript and peer review metadata on more than 5 million authors and reviewers. In our white paper,<sup>1</sup> we describe all the attributions and results of some calculations such as variation of interrater agreement, number of peer reviewers per manuscript, peer review rounds, gender balance, and country representation in editorial, authorship, and reviewers’ composition across different disciplines and impact factor quartiles. Interested researchers are provided with a proposal protocol<sup>2</sup> to submit their research plan. All protocols go through rounds of peer review by the academic advisors of the PRW, who are listed in our page.<sup>2</sup> The accepted applications receive an onboarding training by our data scientist (R.P.) who further prepares the relevant data segment suitable for their primary and secondary research purposes. </p>
<h4>Results</h4>
<p> Since September 2022, PRW has received 12 proposals, of which 6 have been accepted. Of the 6 accepted proposals, all have been onboarded and are actively working with the data. Thirty-one researchers from 17 universities in 9 countries are involved in these research projects. Based on the self-reported data, 39% (n = 12) of these researchers are women and 61% (n = 19) are men; the options “Other” and “Prefer not to disclose” were both at 0%. We promote transparency in science and therefore publish all accepted proposals on a dedicated Peer Review Workbench Research Paper Series page on SSRN.<sup>2</sup> <b>Box 25-0963</b> shows the list of research proposals currently working with the data. So far, these proposals published on SSRN have attracted over 170 downloads and almost 800 views. </p>

![](https://assets.underline.io/markdown_image/1/image/808b0e6a6b910d2e6fd1130accd36e53.png)

<h4>Conclusions</h4>
<p> The breadth of the ongoing studies encourages us to continue adding Elsevier journal and manuscript metadata annually to PRW and organize our seasonal seminars starting from autumn 2025 to report on the progress and findings of each study. This may also help improve the visibility of PRW. </p>
<h4>References</h4>
<p>1. Petchiappan R, James K, Plume A, et al. Analysing Elsevier Journal Metadata within peer review Workbench. SSRN. Revised June 9, 2025. Accessed July 16, 2025. <a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4211833"><span class="Hyperlink CharOverride-6">https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4211833</span></a></p>
<p>2. SSRN. PRW: Peer Review Workbench Research Paper Series. Accessed July 16, 2025. <a href="https://www.ssrn.com/index.cfm/en/peer-review-workbench/"><span class="Hyperlink CharOverride-6">https://www.ssrn.com/index.cfm/en/peer-review-workbench/</span></a></p>
<sup>1</sup>Elsevier, Amsterdam, the Netherlands, bahar.mehmani@elsevier.com. </p>
<h4>Conflict of Interest Disclosures</h4>
<p> All authors are employees of Elsevier, the owner of the Peer Review Workbench, the publisher of the journals whose metadata is hosted on PRW, and the owner of the submission system used for collecting the data. Elsevier is also the owner of the Scopus database that was used to enrich the journals’ metadata with their different impact factor quartiles and subject areas. Bahar Mehmani is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. </p>


[V] Review of Proposals Submitted to Elsevier's Peer Review Workbench

<p>Sean Hays,<sup>1</sup> Tyler Carneal,<sup>1</sup> Christopher Kirman<sup>1</sup></p>
<h4>Objective</h4>
<p> A pilot study was conducted in which a community of SciPinion science experts (&gt;32,000 registered users) and the SciPinion web platform performed rapid peer review of a study.</p>
<h4>Design </h4>
<p>Review design elements were included to avoid sources of bias in key components of the peer review (reviewer participation, selection, engagement, and reporting). A review announcement, released in January 2025, included the general topic and expertise needs (eg, developmental neurotoxicity) but did not identify the specific manuscript. Within 24 hours of the announcement, 152 scientists applied and 5 reviewers were selected automatically by the web platform based on expertise metrics (years of experience, publications, counts of key words relevant to the manuscript). The review panel consisted of qualified scientists (mean [SD] years of experience, 37 [14] years; range, 21-52 years; mean [SD] publication count, 416 [124]; range, 216-550) across 4 countries. Reviewers were automatically verified via a community-driven reputation system based on past interactions with the web platform. To improve robustness of the process, the review was structured as a 3-round modified Delphi format. A $250 honorarium was offered to reviewers to complete their work under an accelerated schedule. Reviewers were masked to each other. In round 1, reviewers worked independently to review the manuscript and answer structured questions that included confidence ratings (0-10 scale, with higher number indicating higher confidence) for components of the manuscript with explanations and confidence in relying on the reported study for making policy decisions by health agencies. In round 2, reviewers worked deliberatively in reviewing and commenting on each other’s answers (13 comments submitted). In round 3, the reviewers returned to their round 1 responses, revised as needed after deliberation, and submitted final responses.</p>
<h4>Results </h4>
<p>All 3 rounds of the review were completed within 1 week of panel selection, and honoraria were initiated within 24 hours of completion. A comparison of the panel’s independent input from round 1 and their deliberative input in round 3 yielded a decrease in the confidence score of the manuscript (mean [SD] confidence score of 4.2 [2.5] after round 1 was 3.8 [2.2] after round 3) and an improvement in consensus across reviewers (based on Tastle and Wierman<sup>1</sup>), from 0.71 after round 1 to 0.77 after round 3) (<b>Figure 25-1065</b>). Confidence in using the study to support decision-making by health agencies was considered low by the review panel (mean [SD] score, 3.0 [1.2]).</p>

![](https://assets.underline.io/markdown_image/1/image/1045176dc2fe1ee30a141edb4aac3dc3.png)

<h4>Conclusions</h4>
<p> This pilot study found that rapid (ie, within 1 week) peer reviews can be performed with a process that includes multiple rounds of engagement and collaborative sharing of reviewer ratings and comments in a scalable manner via automation of multiple steps in the process, including ranking of candidate reviewers, contracting (agreement defining scope of work and compensation), scheduling of rounds, messaging, and payments. </p>
<h4>Reference</h4>
<p>1. Tastle WJ, Wierman MJ. Consensus and dissention: a measure of ordinal dispersion. <i>Int J Approx Reason</i>. 2007;45(3):531-545. doi:10.1016/j.ijar.2006.06.024</p>
<sup>1</sup>SciPinion, Bozeman, MT, ckirman@scipinion.com.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support </h4>
<p>This work was self-funded by SciPinion.</p>


[V] Use of a 3-Round Modified Delphi Format to Support Rapid Peer Review

<p>Julián C. Velásquez Paz,<sup>1</sup> Andrés Zorrilla Vaca,<sup>2</sup> Markus Klimek,<sup>3</sup> Jose A. Calvache<sup>1,3</sup> </p>
<h4>Objective</h4>
<p> In recent years, systematic reviews (SRs) and meta-analyses (MAs)—traditionally regarded as reliable sources of causal evidence due to their methodological rigor—have increasingly been compromised by the inclusion of predatory literature, which is marked by misleading information and questionable editorial practices.<sup>1</sup> Despite ongoing efforts,<sup>2</sup> predatory literature continues to pose a major challenge in evidence synthesis. We aimed to assess the degree of inclusion of randomized controlled trials (RCTs) published in potentially predatory journals into SRs with MAs published in leading anesthesiology journals.</p>
<h4>Methods</h4>
<p> We conducted a cross-sectional study in which we included SRs with MAs published between 2021 and 2022 in the top 10 high-ranked anesthesia journals based on the Web of Science’s Journal Citation Report for 2021. Exclusion criteria included SRs without MAs or network MAs as well as those including observational studies and nonhuman studies. To determine whether an RCT was published in a potentially predatory journal, 2 authors independently followed a stepwise process: (1) an initial assessment of the publisher’s reputation, confirming if it was widely recognized and reputable in the academic publishing community; (2) verification of the journal’s indexing in the Committee on Publication Ethics or Directory of Open Access Journals; and (3) application of the Loyola Marymount University Journal Evaluation Tool (JET).<sup>3</sup> Discrepancies between the 2 authors were resolved by consensus. The JET evaluates journals based on key criteria such as transparency of editorial processes, rigor of peer review practices, adherence to ethical standards, presence in reputable indexing databases, archiving practices, and the quality of the journal’s website; each aspect is scored using defined indicators based on their credibility and risk of predatory practices (<b>Figure 25-1165</b>). We summarized our results using descriptive statistics.</p>

![](https://assets.underline.io/markdown_image/1/image/4fcb2f14a0f0a0a3896c444182448a51.png)

<h4>Results </h4>
<p>A total of 79 SRs with MAs including 1110 RCTs were analyzed, with a median of 10 RCTs per SR. Most studies originated from the US, China, and India (432 [38.9%]). After applying the stepwise approach, 29 (2.6%) of the analyzed RCTs were identified as published in journals classified by our process as untrustworthy; in addition, RCTs published in potentially predatory journals were included in 14 (17.7%) of the 79 studied SRs. Finally, after reviewing Beall’s list, <a href="https://www.predatoryjournals.org/"><span class="Hyperlink CharOverride-6">www.predatoryjournals.org</span></a>, and journals excluded from the Web of Science or Scopus since 2017, we found that only 5 (17.2%) of the RCTs published in journals classified as untrustworthy appeared on any of these lists.</p>
<h4>Conclusions</h4>
<p> This study highlights that RCTs published in potentially predatory journals are being cited in SRs with MAs published in leading anesthesiology journals. Our findings indicate that while lists of potentially predatory journals can be helpful, they are probably not sufficient to fully identify actual predatory journals. Further analysis is needed to determine the impact of RCTs published by potentially predatory journals on the main findings of these MAs.</p>
<h4>References</h4>
<p>1. Grudniewicz A, Moher D, Cobey KD, et al. Predatory journals: no definition, no defence. <i>Nature</i>. 2019;576(7786):210-212. doi:10.1038/d41586-019-03759-y</p>
<p>2. Cukier S, Helal L, Rice DB, et al. Checklists to detect potential predatory biomedical journals: a systematic review. <i>BMC Med</i>. 2020;18(1):104. doi:10.1186/s12916-020-01566-1</p>
<p>3. Rele S, Kennedy M, Blas N. Journal Evaluation Tool. Loyola Marymount University. 2017. Accessed July 10, 2024. <a href="https://digitalcommons.lmu.edu/librarian_pubs/40"><span class="Hyperlink CharOverride-6">https://digitalcommons.lmu.edu/librarian_pubs/40</span></a></p>
<sup>1</sup>Department of Anaesthesiology, Universidad del Cauca, Popayan, Colombia, jacalvache@unicauca.edu.co; <sup>2</sup>Department of Anesthesiology, Perioperative and Pain Medicine, Brigham and Women’s Hospital, Boston, MA, US; <sup>3</sup>Department of Anesthesiology, Erasmus University Medical Center Rotterdam, the Netherlands.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>


[V] Inclusion of Randomized Controlled Trials Published by Potentially Predatory Journals in Anesthesiology Systematic Reviews: A Cross-Sectional Study

<p>Zexing Song,<sup>1,2</sup> Cecilie Jespersen,<sup>3,4</sup> Asbjørn Hróbjartsson,<sup>3,4</sup> S. Joseph Kim,<sup>1,5,6</sup> Rob Fowler,<sup>1,5</sup> Peter C. Austin,<sup>1,6</sup> An-Wen Chan<sup>1,2,5</sup></p>
<h4>Objective</h4>
<p> Outcome switching between study design and reporting is a potential source of bias in observational studies, but there is a paucity of evidence as to its frequency. We aimed to estimate the prevalence of outcome switching in observational studies of interventions (defined as controlled cohort studies investigating the causal effects of interventions on health-related outcomes). Secondary aims included assessing the completeness of prespecification of primary outcomes and factors associated with outcome switching.</p>
<h4>Design</h4>
<p> This meta-epidemiological study involved longitudinal analyses of observational studies of interventions prospectively registered on ClinicalTrials.gov within 1 month of their study start date between 2014 and 2016 that had results published in a peer-reviewed journal. We screened registry records from January through December 2024 to create the study sample and completed outcome data extraction and analysis from January through April 2025. Complete outcome prespecification required explicit definition in the registry of the measurement variable, analysis metric, method of aggregation (the statistic to summarize the outcome within each group), and time point of the outcome. We evaluated outcome switching by identifying discrepancies in the primary outcomes between the registry and published articles, including omission (prespecified primary outcomes not reported), downgrading (prespecified primary outcomes reported as nonprimary), upgrading (prespecified nonprimary outcomes reported as primary), and introduction of new primary outcomes not listed in the registry. We considered outcome switching to favor statistically significant results if a new statistically significant primary outcome was introduced or upgraded or a nonsignificant one was downgraded. We performed multivariable logistic regression to estimate the association between study characteristics and outcome switching.</p>
<h4>Results</h4>
<p> We screened 9965 registry records labelled as observational studies and included 127 eligible studies with results published between January 2015 and October 2024. Only 23 studies (18%) completely prespecified their primary outcome in the registry, and the method of aggregation was the least commonly defined element (33 [26%]). Outcome switching was found in 60 studies (47%), and only 1 of these studies (2%) provided a rationale for the changes. The most common discrepancy was omission (32 [25%]) followed by downgrading (30 [24%]), introduction of new primary outcomes (23 studies with 29 new primary outcomes [18%]), and upgrading (2 [2%]). New primary outcomes differed most commonly between the registry and published articles in the measurement variable (21 of 29 [72%]) and time point (15 of 29 [52%]). Among 54 studies that had discrepancies not limited to omitted primary outcomes, statistically significant results were favored in 80% (43 of 54). No study characteristics were significantly associated with outcome switching (<span class="CharOverride-4"><b>Table 25-0874</b></span>).</p>

![](https://assets.underline.io/markdown_image/1/image/d12bf58a0e21db28d130f25b903c1bba.png)

<h4>Conclusions</h4>
<p> Unreported outcome switching and inadequate outcome prespecification were common in observational studies of interventions. These findings underscore the need for improved registration practices and greater transparency to better understand the risk of bias in observational research.</p>
<sup>1</sup>Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada, zexing.song@mail.utoronto.ca; <sup>2</sup>Division of Dermatology, Women’s College Research Institute, Women’s College Hospital, Toronto, Ontario, Canada; <sup>3</sup>Cochrane Denmark &amp; Centre for Evidence-Based Medicine Odense (CEBMO), University of Southern Denmark, Odense, Denmark; <sup>4</sup>Open Patient data Explorative Network (OPEN), Odense University Hospital, Odense, Denmark; <sup>5</sup>Department of Medicine, University of Toronto, Toronto, Ontario, Canada; <sup>6</sup>ICES, Toronto, Ontario, Canada.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> An-Wen Chan is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. No other disclosures were reported.</p>


[V] Outcome Switching in Observational Studies of Interventions: Comparison of Registration Records and Published Articles

<p>Warren Stern,<sup>1</sup> Ioanna Iliopulos,<sup>2</sup> Christopher Boyd<sup>2</sup></p>
<h4>Objective </h4>
<p>The lack of agreement on minimum standards for disclosure of irradiation parameters results in the need to repeat costly studies, uncertainty in observed effects, and difficulty validating research outcomes. In turn, the Brookhaven National Laboratory (BNL) conducted a meta-study to gain greater quantitative and qualitative insights into this issue. After the results were published in 2022,<sup>1,2</sup> 2 annual Compatibility in Irradiation Research Protocols Expert Roundtable (CIRPER) meetings were conducted by BNL with the support of the National Nuclear Security Administration’s Office of Radiological Security. We describe the process of advocacy research that was implemented to address this issue—that is, conducting a technical meta-analysis, identifying gaps and actions based on its results, sufficiently addressing those gaps, and then publishing the outcome along with the recommendation in a peer-reviewed article to encourage dissemination and adoption by the wider community.</p>
<h4>Design </h4>
<p>The study that BNL published in 2022 identified over 450 published articles and demonstrated that although source specifications (eg, model, energy, and spectra) are well reported, experimental methodologies and irradiation protocols are not. Based on these findings, participants for CIRPER were carefully selected to represent a diverse segment of the radiation research community, such as from the fields of medical physics and radiobiology, working in academic, health care, and industry settings alongside federal partners, national laboratory personnel, and device manufacturer representatives. The meeting provided a venue to bring together diverse stakeholders that typically do not collectively interact and consisted of detailed workshops designed to parse out necessary methodological parameters for published and federally funded research.</p>
<h4>Results </h4>
<p>To increase transparency and reproducibility, experts at the meeting agreed on a standard set of disclosure requirements researchers working with x-ray irradiators should include when publishing their work (<b>Table 25-0890</b>). The meeting also resulted in further discussions on how session participants can support CIRPER’s efforts to harmonize the reporting of irradiation protocols and methods. </p>

![](https://assets.underline.io/markdown_image/1/image/888e8e7417cbddc785d4dac9fd76c0bf.png)

<h4>Conclusions </h4>
<p>As the research community moves to x-ray–based irradiation technologies from cesium 137–based ones, reporting parameters must adapt and update accordingly. The BNL meta-analyses uncovered data that quantifies this barrier. A strategy for advocacy research was designed and implemented accordingly to appropriately address this problem by (1) further identifying gaps in the reporting process, (2) gathering relevant stakeholders to design an informed approach, and (3) gaining commitment from journal editorial boards, funding agencies, and researchers to adopt CIRPER recommendations. This effort has already had a positive impact on the quality of peer-reviewed and funded research and serves as a model for improving other aspects of peer review and publication.<sup>3</sup></p>
<h4>References</h4>
<p>1. Stern W, Zia S, Boyd C, Iliopulos I, Peiris P, Fedurin M. <i>Alternative Technologies Meta Study Report</i>. US Department of Energy; 2022. doi:10.2172/1895064</p>
<p>2. Stern W, Zia S, Boyd C, Iliopulos I. <i>Alternative Technologies Adopters Outreach Survey Report</i>. US Department of Energy; 2022. doi:10.2172/1895065</p>
<p>3. Stern W, Alaei P, Berbeco R, et al. Achieving consistent reporting of radiation dosimetry by adoption of compatibility in irradiation research protocols expert roundtable (CIRPER) recommendations. <i>Radiat Res</i>. 2024;201(3):267-269. doi:10.1667/RADE-23-00234.1</p>
<sup>1</sup>Brookhaven National Laboratory, Upton, NY, US, wstern@bnl.gov; <sup>2</sup>MARC Strategics, Ann Arbor, MI, US.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>None reported.</p>
<h4>Funding/Support </h4>
<p>This work was funded and sponsored by an agency of the US government. </p>
<h4>Role of the Funder/Sponsor </h4>
<p>The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.</p>
<h4>Disclaimer </h4>
<p>Neither the US government nor any agency thereof, nor any of their employees, nor any of their contractors, subcontractors, or their employees, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or any third party’s use or the results of such use of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. The views and opinions of the authors expressed herein do not necessarily state or reflect those of the US government or any agency thereof. </p>


[V] Developing a Harmonized Approach for Reporting Irradiation Protocols and Methods for Research Using X-ray Irradiators

<p>Laura Caquelin,<sup>1</sup> Rachel Heyard,<sup>2</sup> Stephanie Zellers,<sup>3</sup> Hanno Würbel,<sup>4</sup> Gustav Nilsonne<sup>1</sup></p>
<h4>Objective</h4>
<p> Computational reproducibility is vital to research quality, enabling others to replicate analyses and achieve the same results. Despite its importance, it is rarely assessed systematically. This study aims to evaluate whether computational reproducibility review during the publication process improves reproducibility compared with standard peer review. We hypothesize that this intervention will improve reproducibility, encourage code sharing, and reduce errors.</p>
<h4>Design</h4>
<p> This randomized controlled trial was designed to enroll manuscripts submitted to partnering journals that meet inclusion criteria, including open data availability and inferential analysis. Manuscripts will be randomized 1:1 to peer review with computational reproducibility review (intervention group) or standard peer review without intervention (control group). The computational reproducibility review involves reproducing the essential statistical results using shared data (with or without the code). Feedback is provided to authors in the intervention group during the peer review process. The reproducibility of the manuscripts randomized to the control group will be assessed after publication only. The primary outcome is the proportion of manuscripts for which we successfully reproduced the essential statistical results after publication. Secondary outcomes include rates of code sharing, overt errors identified, time required for the review, publication timelines, and categorized reproducibility issues.</p>
<h4>Results</h4>
<p> This study began in January 2025, with a preregistered protocol openly available on the Open Science Framework.<sup>1</sup> Recruitment is ongoing, and as of April 2025, 28 manuscripts submitted to the partnering journal GigaSicence have been screened, with 17 deemed eligible. The project is scheduled to run through September 2026, with the goal of including 200 manuscripts. This presentation will outline key insights gained during the study’s implementation and preliminary results.</p>
<h4>Conclusions</h4>
<p> This study fills an important gap in assessing research quality by testing a practical way to improve computational reproducibility. If successful, the results could support wider use of reproducibility reviews in academic journals, leading to more reliable and trustworthy science.</p>
<h4>Reference</h4>
<p>1. Caquelin L, Heyard R, Zellers S, et al. A study protocol for an intervention study using computational reproducibility testing to improve reproducibility. Open Science Framework. Updated January 29, 2025. doi:10.17605/OSF.IO/3Y8FP</p>
<sup>1</sup>Karolinska Institutet, Stockholm, Sweden, laura.caquelin@ki.se; <sup>2</sup>Center for Reproducible Science at the Epidemiology, Biostatistics and Prevention Institute, University of Zurich, Zurich, Switzerland; <sup>3</sup>Institute for Molecular Medicine Finland, University of Helsinki, Helsinki, Finland; <sup>4</sup>Division of Animal Welfare, University of Bern, Bern, Switzerland.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> This study is part of the iRISE (Improving Reproducibility in Science) project, which has received funding from the European Union under the Horizon Europe program (grant agreement 101094853).</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funder had no role in the data collection, analysis, or conclusions of the abstract.</p>


[V] Testing Computational Reproducibility Review in Editorial Workflows of Academic Journals: A Randomized Controlled Trial from the European iRISE project

<p>Benjamin Djulbegovic,<sup>1</sup> Iztok Hozo,<sup>2</sup> Renata Iskander,<sup>5</sup> Austin J. Parish,<sup>3,4</sup> Jonathan Kimmelman,<sup>5</sup> John P. A. Ioannidis<sup>4</sup></p>
<h4>Objective</h4>
<p> Randomized clinical trials (RCTs) are commonly considered important for detecting small treatment benefits. However, previous studies have shown they are equally likely to detect large (“dramatic”) treatment effects.<sup>1-3</sup> Forecasting the likelihood of future large treatment effects is critical for informing policies on resource allocation in conducting human RCTs, including cancer trials.</p>
<h4>Design </h4>
<p>We conducted a systematic review to inform generalized Pareto distribution (GPD) under extreme value theory to predict future maximum treatment effects based on data from the past 65 years. We included consecutive cancer RCTs (“cohorts”) identified by funders or trial registries designed to minimize publication bias and analyzed all trials regardless of publication status. Five such cohorts have been described in the literature to date. </p>
<h4>Results </h4>
<p>Between 1955 and 2018, a total of 716 RCTs testing 984 experimental vs standard treatments in 349,947 patients were conducted and published by 2022, averaging approximately 20 RCTs per year. The shape parameter of the GPD had positive values, indicating no upper limit on maximum treatment effects. We found that the treatment with the largest effect in the past had an odds ratio (OR) of 45 (95% CI, 2-1008). If effect patterns remain the same and a similar pace of performing RCTs continues, the largest predicted future effect over the next 50 years would be an OR of 23 (95% CI, 4-43) (<span class="CharOverride-4"><b>Figure 25-0942</b></span>). We estimated a 20% probability of detecting new treatment effects with an OR greater than 50 in the next 50 years. We also found that conducting more RCTs (40 or 60 per year) would double or triple the probability of detecting breakthrough treatments with dramatic effects.</p>

![](https://assets.underline.io/markdown_image/1/image/e757b092f8aa49cda20b7b719367bd63.png)

<h4>Conclusions</h4>
<p> Our analysis indicates there may be no upper bound on the maximum discoverable treatment effects in cancer RCTs, but effect estimates will likely remain within the range of those observed between 1955 and 2022. Conducting more RCTs would accelerate the detection of treatments with large effects.</p>
<h4>References</h4>
<p>1. Djulbegovic B, Kumar A, Glasziou P, Miladinovic B, Chalmers I. Medical research: Trial unpredictability yields predictable therapy gains. <i>Nature</i>. 2013;500(7463):395-396. doi:10.1038/500395a</p>
<p>2. Hozo I, Djulbegovic B, Parish AJ, Ioannidis JPA. Identification of threshold for large (dramatic) effects that would obviate randomized trials is not possible. <i>J Clin Epidemiol</i>. 2022;145:101-111. doi:10.1016/j.jclinepi.2022.01.016</p>
<p>3. Djulbegovic B, Kumar A, Soares HP, et al. Treatment success in cancer: new cancer treatment successes identified in phase 3 randomized controlled trials conducted by the National Cancer Institute-sponsored cooperative oncology groups, 1955 to 2006. <i>Arch Intern Med</i>. 2008;168(6):632-642. doi:10.1001/archinte.168.6.632</p>
<sup>1</sup>Medical University of South Carolina, Division of Medical Hematology and Oncology, Department of Medicine, Charleston, SC, US, djulbegov@musc.edu; <sup>2</sup>Department of Mathematics, Indiana University Northwest, Gary, IN, US; <sup>3</sup>Department of Emergency Medicine, Lincoln Medical Center, Bronx, NY; <sup>4</sup>Stanford Prevention Research Center, Department of Medicine, Stanford University School of Medicine; Department of Epidemiology and Population Health, Stanford University School of Medicine; Department of Biomedical Data Science, Stanford University School of Medicine; Department of Statistics, Stanford University School of Humanities and Sciences, Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, US; <sup>5</sup> Department of Equity, Ethics and Policy School of Population and Global Health, McGill University, Montreal, QC, Canada.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>John P. A. Ioannidis is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. No other disclosures reported. </p>
<h4>Additional Information </h4>
<p>Datasets for this work were obtained with support of grants from the US National Institute of Health: R01CA140408, R01NS044417, R01NS052956, and R01CA133594 (Benjamin Djulbegovic). </p>


[V] Estimation of an Upper Limit on the Maximum Effect That Can Be Detected in Randomized Trials of Cancer Therapeutics

<p>Ambar Khan,<sup>1</sup> Ania Bobrowska,<sup>2</sup> Chloe Coelho,<sup>2</sup> Hannah Frost,<sup>2</sup> Swati Kumar,<sup>2</sup> Hannah Russell,<sup>3</sup> Anna Noel-Storr,<sup>4</sup> Molly Murton<sup>2</sup></p>
<h4>Objective</h4>
<p> This study aimed to evaluate a traditional (validated search filter) vs machine learning (classifier) approach for identifying randomized clinical trials (RCTs), in the context of a real-life systematic literature review (SLR) project. </p>
<h4>Design</h4>
<p> A real-life SLR was conducted following standard practices as recommended by the Cochrane Handbook,<sup>1</sup> covering all identification stages from electronic database searches to full-text review, to identify RCTs on fibrodysplasia ossificans progressiva (FOP; a rare disease). Two approaches were compared for the searches conducted in January 2025: (1) combining population search terms with the Scottish Intercollegiate Guidelines Network (SIGN) RCT search filter (“filter” approach) and (2) using population search terms only and running the results through the Cochrane RCT classifier at a 99% recall threshold (“classifier” approach). All subsequent stages were conducted in an identical manner by the same review team, with both abstracts and full texts dual-screened by independent reviewers against prespecified eligibility criteria. The time taken for each approach was measured and reviewer feedback was gathered via a modified NASA Task Load Index. Performance metrics for each approach were calculated, including the accuracy, sensitivity, specificity, and precision. McNemar test was used to assess statistical significance of the differences in the performance metrices, and a <i>P</i> value &lt;.05 was considered statistically significant. </p>
<h4>Results</h4>
<p> Prior to application of the classifier or filter, 9588 records were found for the FOP population. After application of the classifier or filter, 2327 and 1582 abstracts were identified as RCTs, respectively. At the end of the full-text review stage for each approach, the same 10 papers reporting on FOP RCTs were identified, demonstrating equal efficacy in terms of finding relevant papers in the context of a real-life SLR. However, when looking more generally at the classification of RCT vs non-RCT publications within the 9588 records (irrespective of ultimate inclusion in the SLR), sensitivity was significantly higher for the classifier vs the filter (McNemar test <i>P</i> &lt; .001; <b>Table 25-0997</b>). The trade-off for higher sensitivity of the classifier was that it had a significantly lower specificity and therefore required 31% more time than the filter approach (25.79 vs 17.86 hours). This also resulted in a less positive user experience in effort and frustration domains. </p>

![](https://assets.underline.io/markdown_image/1/image/bf3eeb47d6679394a2ede2d090402f99.png)

<h4>Conclusions</h4>
<p> For SLRs, where the identification of all relevant evidence is crucial, the performance of the classifier was superior to a traditional search filter. However, this was at the cost of increased time and reduced specificity. Further testing of the application of machine learning classifiers to augment literature review processes in real-life projects (including other, larger disease areas) is required. The risk of bias in this research, stemming from the researchers being unblinded to the approach, should also be noted.</p>
<h4>Reference</h4>
<p>1. Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al. <i>Cochrane Handbook for Systematic Reviews of Interventions Version 6.3</i>. Cochrane; 2022.</p>
<sup>1</sup>Costello Medical, London, UK; <sup>2</sup>Costello Medical, Cambridge, UK, molly.murton@costellomedical.com; <sup>3</sup>Costello Medical, Manchester, UK; <sup>4</sup>Cochrane, London, UK.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> Ambar Khan, Ania Bobrowska, Chloe Coelho, Hannah Frost, Swati Kumar, Hannah Russell, and Molly Murton reported being employees of Costello Medical. No other disclosures were reported.</p>
<h4>Funding/Support</h4>
<p> This study was funded by Costello Medical. </p>
<h4>Role of the Funder/Sponsor</h4>
<p> Costello Medical supported all aspects of the study, including the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and the decision to submit the abstract for presentation.</p>


Next from The 10th Peer Review Congress

[V] Research Culture Influences in Peer Review: A Targeted Thematic Analysis of Current Challenges in Peer Review

Similar lecture

Assessing the Impact of Clinical Trial Data Sharing: A Cross-Sectional Study of Clinical Trials Sharing Data on the YODA Project Platform