United States

&lt;p&gt;Yuanxi Jia,&lt;sup&gt;1&lt;/sup&gt; Yiwen Jiang,&lt;sup&gt;2&lt;/sup&gt; Karen A. Robinson,&lt;sup&gt;3&lt;/sup&gt; Jinling Tang&lt;sup&gt;4&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Objective&lt;/h4&gt;
&lt;p&gt; Researchers conducting randomized clinical trials (RCTs) might focus more on primary outcomes than secondary outcomes. Therefore, when the design and conduct of an RCT were manipulated for positive results, the effect size of the primary outcome may be overestimated more than that of the secondary outcomes. Among RCTs with similar design, the discrepant effect estimates of the same outcome between those designating it as a primary outcome (PO-RCTs) and others designating it as a secondary outcome (SO-RCTs) may indicate the impact of bias. This study aimed to compare the effect estimates of the same outcomes between PO-RCTs and SO-RCTs. PO-RCTs were hypothesized to produce higher effect estimates than SO-RCTs.&lt;/p&gt;
&lt;h4&gt;Design&lt;/h4&gt;
&lt;p&gt; This meta-research study using a matched cohort design included 153 meta-analyses assessing the efficacy/effectiveness of health interventions published in the Cochrane Database of Systematic Reviews between 2021 and 2023. A total of 1073 RCTs with parallel design, individual randomization, superiority tests, recruitment in or after 2006, and negative control (placebo, sham, no intervention, waiting list, minimal intervention, attention control, or standard care) were included. Within each meta-analysis, RCTs designating the meta-analyzed outcome as primary outcome were classified as PO-RCTs (553 RCTs), while those designating the meta-analyzed outcome as secondary outcome were classified as SO-RCTs (520 RCTs). RCTs that did not define a primary outcome were excluded. The PO-RCTs were compared with the SO-RCTs using 2-stage random-effect meta-analyses: the effect estimates from RCTs were transformed into odds ratios (ORs); within each meta-analysis, the ORs from PO-RCTs and SO-RCTs were compared as a ratio of ORs (ROR); and the RORs across meta-analyses were combined. The primary analyses were conducted among all RCTs and prospectively registered RCTs. The risk of bias for blinding was assessed using the Cochrane Risk of Bias Tool.&lt;/p&gt;
&lt;h4&gt;Results &lt;/h4&gt;
&lt;p&gt;Among 1073 RCTs from 153 meta-analyses, PO-RCTs produced ORs 1.27 (95% CI, 1.14-1.42; &lt;i&gt;I&lt;/i&gt;&lt;sup&gt;2&lt;/sup&gt; = 43.6%) times higher than SO-RCTs. Among 372 prospectively registered RCTs from 122 meta-analyses, PO-RCTs produced ORs 1.21 (95% CI, 1.03-1.41; &lt;i&gt;I&lt;/i&gt;&lt;sup&gt;2&lt;/sup&gt; = 22.3%) times higher than SO-RCTs. When restricted to RCTs with high/unclear risk in performance bias, detection bias, or both types of bias, PO-RCTs produced ORs 1.48 (95% CI, 1.26-1.75), 1.44 (95% CI, 1.21-1.71), and 1.60 (95% CI, 1.30-1.96) times higher than SO-RCTs, respectively. When restricted to RCTs with low risk in performance bias, detection bias, or both types of bias, PO-RCTs produced ORs 1.08 (95% CI, 0.90-1.28), 1.04 (95% CI, 0.87-1.24), and 1.06 (95% CI, 0.90-1.26) times higher than SO-RCTs, respectively.&lt;/p&gt;
&lt;h4&gt;Conclusions&lt;/h4&gt;
&lt;p&gt; The effect estimate for an outcome could be 27% larger when designated as a primary outcome than a secondary outcome in RCTs, which may be partially attributed to bias. Researchers conducting systematic reviews may need to note whether an outcome was designated as primary or secondary in RCTs, perform sensitivity analyses, and interpret results considering the impact of potential bias.&lt;/p&gt;
&lt;sup&gt;1&lt;/sup&gt;Yong Loo Lin School of Medicine, National University of Singapore, Singapore, yx.jia@nus.edu.sg; &lt;sup&gt;2&lt;/sup&gt;Shenzhen Institute of Advanced Technology, Shenzhen, China; &lt;sup&gt;3&lt;/sup&gt;School of Medicine, Johns Hopkins University, Baltimore, US; &lt;sup&gt;4&lt;/sup&gt;Shenzhen University of Advanced Technology, Shenzhen, China. &lt;h4&gt;Conflict of Interest Disclosures&lt;/h4&gt; None reported. &lt;h4&gt;Funding/Support&lt;/h4&gt; This study was supported by the Shenzhen Science and Technology Program (grant KQTD20190929172835662) from the Shenzhen Municipal Government, Guangdong Province, China, and the Outstanding Youth Innovation Fund from the Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences (grant E2G019). &lt;h4&gt;Role of the Funder/Sponsor&lt;/h4&gt; The funders had no roles in the design, conduct, or reporting of this study.



The 10th Peer Review Congress

Effect Estimates for the Same Outcomes Designated as Primary vs Secondary in Randomized Clinical Trials: A Meta-Research Study

quality of trials

publication

bias

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Yuanxi Jia,<sup>1</sup> Yiwen Jiang,<sup>2</sup> Karen A. Robinson,<sup>3</sup> Jinling Tang<sup>4</sup></p>
<h4>Objective</h4>
<p> Researchers conducting randomized clinical trials (RCTs) might focus more on primary outcomes than secondary outcomes. Therefore, when the design and conduct of an RCT were manipulated for positive results, the effect size of the primary outcome may be overestimated more than that of the secondary outcomes. Among RCTs with similar design, the discrepant effect estimates of the same outcome between those designating it as a primary outcome (PO-RCTs) and others designating it as a secondary outcome (SO-RCTs) may indicate the impact of bias. This study aimed to compare the effect estimates of the same outcomes between PO-RCTs and SO-RCTs. PO-RCTs were hypothesized to produce higher effect estimates than SO-RCTs.</p>
<h4>Design</h4>
<p> This meta-research study using a matched cohort design included 153 meta-analyses assessing the efficacy/effectiveness of health interventions published in the Cochrane Database of Systematic Reviews between 2021 and 2023. A total of 1073 RCTs with parallel design, individual randomization, superiority tests, recruitment in or after 2006, and negative control (placebo, sham, no intervention, waiting list, minimal intervention, attention control, or standard care) were included. Within each meta-analysis, RCTs designating the meta-analyzed outcome as primary outcome were classified as PO-RCTs (553 RCTs), while those designating the meta-analyzed outcome as secondary outcome were classified as SO-RCTs (520 RCTs). RCTs that did not define a primary outcome were excluded. The PO-RCTs were compared with the SO-RCTs using 2-stage random-effect meta-analyses: the effect estimates from RCTs were transformed into odds ratios (ORs); within each meta-analysis, the ORs from PO-RCTs and SO-RCTs were compared as a ratio of ORs (ROR); and the RORs across meta-analyses were combined. The primary analyses were conducted among all RCTs and prospectively registered RCTs. The risk of bias for blinding was assessed using the Cochrane Risk of Bias Tool.</p>
<h4>Results </h4>
<p>Among 1073 RCTs from 153 meta-analyses, PO-RCTs produced ORs 1.27 (95% CI, 1.14-1.42; <i>I</i><sup>2</sup> = 43.6%) times higher than SO-RCTs. Among 372 prospectively registered RCTs from 122 meta-analyses, PO-RCTs produced ORs 1.21 (95% CI, 1.03-1.41; <i>I</i><sup>2</sup> = 22.3%) times higher than SO-RCTs. When restricted to RCTs with high/unclear risk in performance bias, detection bias, or both types of bias, PO-RCTs produced ORs 1.48 (95% CI, 1.26-1.75), 1.44 (95% CI, 1.21-1.71), and 1.60 (95% CI, 1.30-1.96) times higher than SO-RCTs, respectively. When restricted to RCTs with low risk in performance bias, detection bias, or both types of bias, PO-RCTs produced ORs 1.08 (95% CI, 0.90-1.28), 1.04 (95% CI, 0.87-1.24), and 1.06 (95% CI, 0.90-1.26) times higher than SO-RCTs, respectively.</p>
<h4>Conclusions</h4>
<p> The effect estimate for an outcome could be 27% larger when designated as a primary outcome than a secondary outcome in RCTs, which may be partially attributed to bias. Researchers conducting systematic reviews may need to note whether an outcome was designated as primary or secondary in RCTs, perform sensitivity analyses, and interpret results considering the impact of potential bias.</p>
<sup>1</sup>Yong Loo Lin School of Medicine, National University of Singapore, Singapore, yx.jia@nus.edu.sg; <sup>2</sup>Shenzhen Institute of Advanced Technology, Shenzhen, China; <sup>3</sup>School of Medicine, Johns Hopkins University, Baltimore, US; <sup>4</sup>Shenzhen University of Advanced Technology, Shenzhen, China. <h4>Conflict of Interest Disclosures</h4> None reported. <h4>Funding/Support</h4> This study was supported by the Shenzhen Science and Technology Program (grant KQTD20190929172835662) from the Shenzhen Municipal Government, Guangdong Province, China, and the Outstanding Youth Innovation Fund from the Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences (grant E2G019). <h4>Role of the Funder/Sponsor</h4> The funders had no roles in the design, conduct, or reporting of this study.



technical paper

<p>Yang Zhang,<sup>1,2</sup> Mengyuan Duan,<sup>1,2</sup> Guoguang Zhao,<sup>3</sup> Penghu Wei,<sup>2,3</sup> Xiuyuan Hao<sup>4</sup></p>
<h4>Objective </h4>
<p>Given the increasing role of artificial intelligence (AI) in academic writing, it is important to understand how these tools are integrated into research practices. This study aimed to explore the use of AI in academic writing by Chinese medical scholars and to identify potential factors influencing their adoption and perceptions of AI. </p>
<h4>Design </h4>
<p>A total of 378 peer reviewers of <i>Chinese Medical Journal</i> were invited to participate in a survey using an electronic questionnaire via email between January 2, 2025, and January 12, 2025. The questionnaire included questions about AI tool use in research paper writing, reasons for nonuse, stages of use, experience of declaration, and perception of effects of AI use. The survey also explored what role AI should have and where researchers believe AI should be acknowledged in publications. The Fisher exact test was used to identify significant differences between subgroups based on age and use.</p>
<h4>Results </h4>
<p>Of the 378 reviewers invited, 159 (42.1%) provided valid responses. Among the respondents, 93 (58.5%) reported using AI tools in their research paper writing, and 66 (41.5%) did not (<b>Table 25-0878</b>). There was a significant difference in declared use of AI across age groups, with younger researchers more likely to use AI tools (76.1% of those ≤30 years, 59.1% of those 31-45 years, 27.8% of those 46-60 years, and 14.3% of those &gt;60 years; <i>P</i> &lt; .001). The primary reasons for not using AI tools included copyright or ethical issues (53.0%), concerns about inaccuracy (47.0%), and unfamiliarity with AI tools (40.9%). Notably, there were still 27.3% non-AI users considering using AI when technology matures. The most common stages at which AI tools were used was after writing for language translation and polishing (90.3%), followed by idea generation before writing (16.1%) and structural organization during writing (15.1%). However, only 29.0% of participants declared AI use in past submissions. A total of 60.2% found the tools to be very helpful, while none considered them to have negative impact. The preferred roles and acknowledgement of AI tools in papers included a separate AI writing declaration section (61.6%), not acknowledging AI at all (17.0%), reporting in the methods section (12.6%), reporting in the conflict of interest section (5.0%), and including AI as a co-author (3.8%), which did not significantly differ between AI users and nonusers; nor did these views differ significantly by age.</p>

![](https://assets.underline.io/markdown_image/1/image/5a3f77baa1c566657fd81faf9e7dac2a.png)

<h4>Conclusions </h4>
<p>In this study, nearly 60% of Chinese medical scholars reported use of AI tools in writing and preparing papers, particularly for language translation and polishing. However, many reported concerns about the accuracy, ethics, and maturity of AI technology. Current AI guidelines in academic publishing lack a unified standard, particularly in usage scope and enforcement. Some journals lack mandatory disclosure policies for AI use, standardized templates for reporting, or requirements to submit archival copies of original data or content before AI use, which might account for incomplete reporting or failure to report use of AI. Further research is needed to assess the long-term impact of AI on academic writing.</p>
<p><sup>1</sup>National Center for Neurological Disorders, Xuanwu Hospital Capital Medical University, Beijing, China; <sup>2</sup>Editorial Office, Brain Network Disorders, Xuanwu Hospital Capital Medical University, Beijing, China; <sup>3</sup>Department of Neurosurgery, Xuanwu Hospital Capital Medical University, Beijing, China; <sup>4</sup>Editorial Office, Chinese Medical Journal, Chinese Medical Association Publishing House, Beijing, China, haoxiuyuan@cma.org.cn. 
<h4>Conflict of Interest Disclosures</h4> None reported. 
<h4>AdditionaI Information</h4> ChatGPT (version GPT-4o mini; OpenAI) was used on January 20, 2025, for language translation, grammar checking, and manuscript polishing to ensure the clarity and coherence of this abstract.


Artificial Intelligence Use and Acknowledgment in Medical Research Writing Among Chinese Scholars

<p>Ian Bulovic,<sup>1</sup> Susmitha Wunnava,<sup>1,2</sup> Wonjin Yoon,<sup>1,3</sup> Adam G. Dunn,<sup>1,4</sup> Florence T. Bourgeois,<sup>1,2,3</sup> Timothy Miller<sup>1,2,3</sup></p>
<h4>Objective </h4>
<p>Selective inclusion and reporting of clinical trial outcomes may result in biases in the medical evidence on benefits and harms of treatments.<sup>1</sup> ClinicalTrials.gov provides the opportunity to identify changes between prespecified outcomes and those eventually reported in publications, but linkage and comparison of these outcomes requires extensive manual curation, precluding studies of more than a few hundred trials.<sup>2,3</sup> ClinicalTrials.gov archives all versions of registration records as changes are made by investigators, providing the basis to monitor trial outcome changes over time if the information could be efficiently processed. Our objective was to use a large language model (LLM) to estimate changes to primary outcomes at scale.</p>
<h4>Design </h4>
<p>OpenAI’s Gpt-4o-2024-08-06 model was prompted (without training) to identify trials with meaningful changes in specified primary outcomes between trial registrations at study start and in the final record (prompts and scripts used for processing data available at <a href="https://github.com/Machine-Learning-for-Medical-Language/outcome-switching">https://github.com/Machine-Learning-for-Medical-Language/outcome-switching</a>. Meaningful changes were defined as addition, removal, or substantial modification to primary outcomes. Substantial modifications included changes to outcome measures, measurement procedures, or measurement time frame. The model performance was evaluated using a set of 100 manually labeled and adjudicated records, showing F1 scores of 0.88 for outcome addition, 0.81 for outcome removal, and 0.67 for outcome modification (F1 scores range from 0 to 1 and balance sensitivity and positive predictive value). The cohort consisted of interventional trials studying US Food and Drug Administration (FDA)–regulated drugs that were prospectively registered and started January 1, 2008, to December 31, 2022 (allowing ≥2 years’ follow-up). Variables of interest (start year, therapeutic area, industry funding, randomization, and enrollment size) were selected a priori and extracted from registration records using structured data available for download from ClinicalTrials.gov. Univariate analyses and multivariate logistic regression were performed to assess for associations between outcome changes and variables of interest.</p>
<h4>Results </h4>
<p>Among 27,227 trials studying FDA-regulated drugs, 11,078 (40.7%) had changes to primary outcomes after trial start. This included 3850 trials (14.1%) with addition of an outcome, 3123 (11.5%) with removal of an outcome, and 9105 (33.4%) with substantial outcome modifications. Trials with industry funding were significantly more likely to have an outcome change (47.7% vs 32.9%; <i>P</i> &lt; .001) (<b>Table 25-1101</b>). This difference was observed for each of the outcome change types, including addition (17.4% vs 10.4%; <i>P</i> &lt; .001), removal (12.9% vs 9.9%; <i>P</i> &lt; .001), and modification (39.7% vs 26.4%; <i>P</i> &lt; .001). Multivariate analysis demonstrated reduction in outcome changes over time (<i>P</i> &lt; .001) and a positive association with industry funding (<i>P</i> &lt; .001).</p>

![](https://assets.underline.io/markdown_image/1/image/baee5d16656423f10f95e3df2f048727.png)

<h4>Conclusions </h4>
<p>LLMs can be used to monitor for changes in clinical trial outcomes efficiently and at scale using information available in ClinicalTrials.gov. Outcome changes in trials of FDA-regulated drugs are common and associated with industry funding but have decreased over time. Future work should explore use of LLMs to identify outcome changes between trial registrations and published articles.</p>
<h4>References</h4>
<p>1. Turner EH, Cipriani A, Furukawa TA, Salanti G, de Vries YA. Selective publication of antidepressant trials and its influence on apparent efficacy: updated comparisons and meta-analyses of newer versus older trials. <i>PLoS Med</i>. 2022;19(1):e1003886. doi:10.1371/journal.pmed.10038862</p>
<p>2. Wang A, Menon R, Li T, et al. Has the degree of outcome reporting bias in surgical randomized trials changed? a meta-regression analysis. <i>ANZ J Surg</i>. 2023;93:76-82. doi:10.1111/ans.182733</p>
<p>3. Hinkel J, Heneghan C, Bankhead C. Selective outcome reporting in cancer studies: a scoping review. <i>medRxiv</i>. 2024:07.02.24309826. doi:10.1101/2024.07.02.24309826</p>
<h4>Affiliations</h4> <sup>1</sup>Computational Health Informatics Program, Boston Children’s Hospital, Boston, MA, US, florence_bourgeois@hms.harvard.edu; <sup>2</sup>Harvard-MIT Center for Regulatory Science, Harvard Medical School, Boston, MA, US; <sup>3</sup>Department of Pediatrics, Harvard Medical School, Boston, MA, US; <sup>4</sup>Biomedical Informatics and Digital Health, Faculty of Medicine and Health, University of Sydney, Sydney, Australia. <h4>Conflict of Interest Disclosures</h4> None reported. <h4>Funding/Support</h4> National Library of Medicine, National Institutes of Health (R01LM012976, R01LM012973). <h4>Role of the Funder/Sponsor</h4> The funder played no role in the design or reporting of the study. <h4>Additional Information</h4> Florence T. Bourgeois and Timothy Miller are co–senior authors.


Detection and Monitoring of Outcome Reporting Changes Using a Large Language Model: Application to FDA-Regulated Drug Trials

<p>Yulin Yu,<sup>1</sup> Yong-Yeol Ahn,<sup>2</sup> Daniel Romero<sup>1,2,3,4</sup></p>
<p><h4>Objective</h4>Datasets are critical in artificial intelligence (AI) research, serving as the foundation for training, evaluating, and developing models. As interest in accelerating AI-driven discoveries grows, it becomes important to examine how strategically repurposing datasets, using them for research topics different from their original context, relates to scientific outcomes. Previous research suggests that combining existing knowledge in novel ways, or recombinant novelty, is associated with greater scientific influence.<sup>1,2</sup> While recent studies have shown that combining datasets in unexpected ways correlates with impact, they have not examined how datasets interact with the topics they are used to study.<sup>3</sup> To address this, we draw on transformational creativity theory, which modifies the accepted conceptual space by altering or removing existing dimensions, such as drug repurposing. Similarly, AI datasets can be repurposed for new research topics. This study addressed 2 questions: (1) are published articles that repurpose datasets associated with higher scientific disruption or recognition? and (2) do repurposed uses of datasets influence how future articles apply the data, and is this repurpose adoption associated with greater scientific outcomes? We conceptualized scientific impact along 2 dimensions: (1) the extent to which an article is recognized and immediately used by research communities, which can be measured by citations, and (2) the degree to which the research article disrupts existing research paradigms. </p>
<p><h4>Design</h4>We analyzed 13,637 published AI articles collected from Papers With Code and linked to the OpenAlex database from 2015 to 2021. We quantified data repurposing by comparing the semantic content of each article to previous articles using the same datasets. For each article, the number of other articles using the same dataset varied (though we set a minimum threshold of at least 1 prior reuse). Articles were embedded using sentence-level language models. We assessed the relationship between repurposing and scientific disruption, defined as the extent to which an article with repurposed data is cited instead of its references, using ordinary least squares regression. We used negative binomial regression to analyze citation counts. Control variables included dataset novelty, article novelty, author information (team size and experience), publication year, citation/reference counts, and disciplinary fields. These variables help account for potential confounders; for example, more recent papers or datasets may naturally have more prior reuse by other studies. We measured repurpose adoption by comparing each article’s similarity to prior vs future works using the same datasets. A positive score suggests that future articles adopted the new topical use.</p>
<h4>Results</h4>
<p> Greater data repurposing was significantly associated with higher disruption scores, corresponding to a 10.2%-SD increase (<i>P</i> &lt; .001) (<b>Figure 25-1139</b>, A). However, articles with repurposed data were slightly negatively associated with citation counts compared with articles without repurposed data, corresponding to a 1.7%-SD decrease, but this was not statistically significant (<i>P</i> &gt; .05) (<b>Figure 25-1139</b>, B). Articles with 1-SD higher repurpose adoption were modestly more disruptive (+2%; <i>P</i> &lt; .001) and showed a stronger positive association with future citations (<i>P</i> &lt; .001). This study is limited by reliance on the OpenAlex database, which may have incomplete metadata.</p>

![](https://assets.underline.io/markdown_image/1/image/2bc5a75630407e6300c494e42d20f8b3.png)

<h4>Conclusions </h4>
<p>In this study, repurposing datasets was associated with greater disruption, though it may have initially reduced recognition. However, when novel data were adopted for reuse in future research, they were associated with both disruption and increased citations. These findings have implications for data sharing and research training aimed at promoting innovative reuse of data.</p>
<h4>References</h4>
<p>1. Uzzi B, Mukherjee S, Stringer M, Jones B. Atypical combinations and scientific impact. <i>Science</i>. 2013;342(6157):468-472. doi:10.1126/science.1240474</p>
<p>2. Leahey E, Lee J, Funk RJ. What types of novelty are most disruptive? <i>Am Sociol Rev</i>. 2023;88(3):562-597. doi:10.1177/00031224231168074</p>
<p>3. Yu Y, Romero DM. Does the use of unusual combinations of datasets contribute to greater scientific impact? <i>Proc Natl Acad Sci U S A</i>. 2024;121(15):e2318482121. doi:10.1073/pnas.2402802121 </p>
<sup>1</sup>School of Information, University of Michigan, Ann Arbor, US, yulinyu@umich.edu; <sup>2</sup>Luddy School of Informatics, Computing, and Engineering, Indiana University, Bloomington, US; <sup>3</sup>Center for the Study of Complex Systems, University of Michigan, Ann Arbor, US; <sup>4</sup>Computer Science and Engineering Division, University of Michigan, Ann Arbor, US. <h4>Conflict of Interest Disclosures</h4> None reported.



Data Repurpose in AI Studies and Scientific Outcomes

<p>Nicola Di Girolamo,<sup>1,2</sup> Reint Meursinge Reynders,<sup>3,4</sup> Ugo Di Girolamo<sup>5</sup></p>
<h4>Objective </h4>
<p>Statements like “to our knowledge, this is the first time that” are ambiguous,<sup>1</sup> but are often used when describing a potential novel aspect of an article, often in lieu of a literature search.<sup>2</sup> Although seemingly innocuous, similar paraphrases lack reproducibility and accountability. This study assessed the prevalence of the statement “to our knowledge” (TOK) in the biomedical literature and in leading medical journals in the past 75 years as well as before and after implementing an intervention to reduce such prevalence in 2 journals.</p>
<h4>Design</h4>
<p> Full texts of articles available in PubMed Central (PMC) were searched using PMC’s phrase index for paraphrases of the statement TOK. Prevalence was measured in articles available in PMC published by 6 leading medical journals (<i>The Lancet</i>,<i> NEJM</i>,<i> The BMJ</i>,<i> JAMA</i>,<i> Ann Int Med</i>,<i> PLOS Medicine</i>) and in all articles in PMC in 2 time periods: January 1950 to December 2019 and January 2020 to December 2024 by 1 author (N.D.G.). To evaluate the search strategy, all articles published in 2024 in <i>The Lancet</i> with full text available in PMC (N = 43) were manually screened for TOK and compared with results of the engine search. To evaluate how often TOK was reported, all <i>The Lancet</i> research articles published in 2024 (N = 166) were manually screened. In 2 journals, <i>Journal of Small Animal Practice</i> (<i>JSAP</i>) and <i>Journal of Exotic Pet Medicine</i> (<i>JEPM</i>), an intervention to advise authors against use of these phrases and in favor of a scoping search of the literature was implemented in 2020, and TOK prevalence was measured in 2018 and 2024.</p>
<h4>Results </h4>
<p>Among 4488 articles available in PMC published in 6 leading medical journals in 2020 to 2024, the prevalence of TOK was 21.7% (974), highest in <i>PLOS Medicine</i> (48.4%) and lowest in <i>NEJM </i>(1.3%) (<b>Table 25-1191</b>). Prevalence among these journals was 5.7% (95% CI, 4.5%-6.9%) higher than all journals included in PMC. Compared with articles published in 1950 to 2019, prevalence of TOK in the last 5 years increased 8.2% (95% CI, 6.9%-9.6%) in leading medical journals and 3.1% (95% CI, 3.1%-3.2%) in all PMC journals. The search strategy had a sensitivity of 66.7% (95% CI, 34.9%-90.1%), a specificity of 100.0% (95% CI, 88.8%-100.0%), and an overall accuracy of 90.7% (95% CI, 77.9%-97.4%). In the manually screened articles, TOK was mentioned 5 times in 1 article (0.6%), 3 times in 6 articles (3.6%), 2 times in 17 articles (10.2%), and 1 time in 57 articles (34.3%). In <i>JSAP</i> and <i>JEPM</i>, TOK prevalence in articles decreased from 37.4% to 7.6% after the intervention.</p>

![](https://assets.underline.io/markdown_image/1/image/be8f320194fc090146374050eee46028.png)

<h4>Conclusions </h4>
<p>Paraphrases of TOK have been highly prevalent in the biomedical literature for the past 75 years and continue to be prevalent, including in leading medical journals. A simple intervention can decrease the use of this statement. This study is limited to only the portion of articles available in PMC, does not include all articles published by the 6 leading medical journals, and does not differentiate between version types (eg, accepted manuscript, published article) or article types (eg, research, review, or opinion).</p>
<h4>References</h4>
<p>1. Vaisrub S. To the best of our knowledge, which is limited at best. <i>JAMA</i>. 1979;241(3):278.</p>
<p>2. Di Girolamo N, Meursinge Reynders R. On “authors’ knowledge” and contrast-enhanced ultrasonography in rabbits. <i>Vet Radiol Ultrasound</i>. 2019;60(4):371.</p>
<h4>Affiliations</h4> <sup>1</sup>Department of Clinical Sciences, College of Veterinary Medicine, Cornell University, Ithaca, NY, US, nd374@cornell.edu; <sup>2</sup>Journal of Small Animal Practice, British Small Animal Veterinary Association, Gloucestershire, UK; <sup>3</sup>Department of Oral and Maxillofacial Surgery, Amsterdam University Medical Center, University of Amsterdam, Amsterdam, the Netherlands; <sup>4</sup>private practice of orthodontics, Milan, Italy; <sup>5</sup>Compass, New York, NY, US. <h4>Conflict of Interest Disclosures</h4> Nicola Di Girolamo was an editor in chief of 2 peer-reviewed journals at the time of submission, 1 published by Elsevier and 1 by Wiley. The other authors declare no competing interests nor conflicts of interest.



Prevalence of the Statement "to Our Knowledge" and Similar Paraphrases in Current and Past Biomedical Literature

<p>Alexander Goldberg,<sup>1</sup> Ivan Stelmakh,<sup>2</sup> Kyunghyun Cho,<sup>3</sup> Alice Oh,<sup>4</sup> Alekh Agarwal,<sup>5</sup> Danielle Belgrave,<sup>6</sup> Nihar B. Shah<sup>1</sup></p>
<h4>Objective </h4>
<p>What biases and errors arise in evaluating the quality of peer reviews? We studied this question, driven by 2 primary motivations: (1) designing incentive systems for high-quality reviewing based on review quality assessments and (2) evaluating experiments within peer review processes.</p>
<h4>Design </h4>
<p>We conducted a large-scale study at the 2022 Conference on Neural Information Processing Systems (NeurIPS), a top-tier venue in machine learning, inviting reviewers, conference chairs, and authors to evaluate reviews. This study was approved by Carnegie Mellon University Institutional Review Board. Evaluators rated reviews (scores and text) on overall quality (7-point Likert scale) and individual criteria (constructiveness, coverage, understanding, and substantiation; 5-point Likert scale). To assess uselessly elongated review bias in randomized controlled trials, we created elongated versions of 10 paper reviews by adding noninformative content (duplicated instructions and copied paper abstracts), increasing length from 200-300 to 600-850 words. We randomly assigned 458 reviewers into 2 equally sized groups: the control group evaluated a short review, and the treatment group evaluated an artificially lengthened review. We used a Mann-Whitney <i>U</i> test with a test statistic that measures the proportion of paper review pairs in which the treatment (long review) was rated higher than the control (short review). To assess author outcome bias, we collected observational data from 3429 authors and 4311 paper reviewers and/or conference organizers evaluating 9870 reviews. We compared author ratings on “accept” vs “reject” reviews of their own papers, controlling for review quality by matching 418 review pairs with similar nonauthor ratings. A Mann-Whitney <i>U</i> test was conducted on these pairs. To assess interevaluator (dis)agreement, miscalibration, and subjectivity, we estimated the magnitude of various sources of error using the observational data described above.</p>
<h4>Results </h4>
<p>Evaluators exhibited a significant elongated review bias (<i>P</i> &lt; .001). Elongated reviews received a mean score of 4.29 vs. 3.73 for original reviews. Each individual criterion also showed length bias (<i>P</i> &lt; .05). Authors were positively biased toward reviews recommending acceptance (<span class="CharOverride-4"><b>Figure 24-0809</b></span>). Authors gave “reject” reviews a mean score of 3.24 compared with 4.65 on “accept” reviews. We also found a statistically significant bias in criteria scores (<i>P</i> &lt; .001 for all criteria). The magnitude of error due to inconsistency, miscalibration, and subjectivity were similar for reviews of reviews and for reviews of papers at NeurIPS.</p>

![](https://assets.underline.io/markdown_image/1/image/688e99baaf2057220b6c8658dfb72e64.png)

<h4>Conclusions </h4>
<p>Our results suggest that the various problems that exist in reviews of papers, most notably bias toward factors like length and review recommendation, also arise in reviewing of reviews. Since our experiment in 2022, our results are increasingly relevant to the evaluation of large language models in peer review settings,<sup>1,2</sup> where the length bias we established through our randomized controlled trial can bias results on the perceived efficacy of large language models.<sup>3</sup></p>
<h4>References</h4>
<p>1. Thakkar N, Yuksekgonul M, Silberg J, et al. Can LLM feedback enhance review quality? a randomized study of 20K reviews at ICLR&#160;2025. <i>arXiv</i>. Preprint posted online April 13, 2025. doi:10.48550/arXiv.2504.09737</p>
<p>2. D’Arcy M, Hope T, Birnbaum L, Downey D. MARG: multi-agent review generation for scientific papers. <i>arXiv</i>. Preprint posted online January 8, 2024. doi:10.48550/arXiv.2401.04259</p>
<p>3. Steyvers M, Tejeda H, Kumar A, et al. What large language models know and what people think they know. <i>Nature Machine Intelligence</i>. 2025;7:221-231. doi:10.1038/s42256-024-00976-7</p>
<sup>1</sup>School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, US, akgoldbe@andrew.cmu.edu; <sup>2</sup>New Economic School, Moscow, Russia; <sup>3</sup>Center for Data Science, New York University, New York, NY, US; <sup>4</sup>School of Computing, KAIST, Daejeon, South Korea; <sup>5</sup>Google Research, New York, NY; <sup>6</sup>GSK, London, UK. <h4>Conflict of Interest Disclosures</h4> Kyunghyun Cho is affiliated with Genentech. Nihar B. Shah is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract. <h4>Acknowledgment</h4> We are greatly indebted to the participants of this experiment for providing evaluations of reviews, thereby helping understand the promises and challenges of evaluating review quality, and consequently also shedding light on the design of incentives and experiments in peer review.






Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Assessments

<p>Charvi Rastogi,<sup>1</sup> Xiangchen Song,<sup>2</sup> Zhijing Jin,<sup>3,4</sup> Ivan Stelmakh,<sup>5</sup> Hal Daumé III,<sup>6</sup> Kun Zhang,<sup>2</sup> Nihar B. Shah<sup>2</sup></p>
<h4>Objective </h4>
<p>Many peer-review processes in computer science involve reviewers submitting independent reviews followed by a discussion between reviewers of each article on a typed forum (online discussion board). A key policy question is whether reviewers should remain anonymous to each other. This study investigated 7 research questions (RQs): RQ 1. Do reviewers discuss more when anonymous to each other or not? RQ 2. Are decisions closer to senior or junior reviewers’ opinions across conditions? RQ 3. Are reviewers more polite when not anonymous? RQ 4. Do self-reported reviewer experiences differ? RQ 5. Do reviewers prefer one condition? RQ 6. What factors do reviewers consider important in this policy decision? RQ 7. Have reviewers experienced dishonest behavior when their identity is revealed to other reviewers?</p>
<h4>Design </h4>
<p>A randomized controlled trial was conducted in the Conference on Uncertainty in Artificial Intelligence (UAI) in 2022, where full articles (not abstracts) were reviewed. Reviewers and articles were randomly assigned to either a condition where reviewer identities were hidden from each other or one where they were visible. Reviewers were then matched to articles within each condition using a semiautomated procedure.<sup>1</sup> An anonymous survey of reviewers was also administered. The following measurements were made: RQ 1. Average posts per reviewer-article pairs were compared; test statistic: difference across conditions. RQ 2. Test statistic: difference in the fraction of articles where the reviewer closest to the final decision was senior. RQ 3. Politeness was scored 1 to 5 using a locally deployed large language model<sup>2</sup> with few-shot prompting; scores were averaged across iterations and paraphrased prompts; test statistic: normalized Mann-Whitney U test. RQ 4. Reviewers rated 5 aspects of their experience on a 5-point Likert scale; differences across conditions were tested using a normalized Mann-Whitney U test. RQ 5. Reviewers rated overall preference on a 5-point scale mapped from −2 to 2; test statistic: Cohen <i>d</i>. RQ 6. Reviewers rated the importance of 6 factors in deciding on anonymity policy, each from 1 (least important) to 6. RQ 7. Reviewers reported experience of any dishonest behavior due to reviewer identities being visible to other reviewers, with checkboxes “Yes, in UAI 2022,” “Yes, in another venue,” “Not sure,” and “No.” The test statistics also served as a measure of the effect sizes. <i>P</i> values were computed via permutation testing.</p>
<h4>Results </h4>
<p>Overall, 322 papers were reviewed under the anonymous condition (116 accepted) and 310 papers under nonanonymous (114 accepted), with exactly 289 reviewers in both conditions. There were 611 discussion posts made by reviewers in the anonymous condition and 514 in the nonanonymous condition. The results for the 7 research questions are provided in <b>Table 24-0812</b>.</p>

![](https://assets.underline.io/markdown_image/1/image/bd5b7dcf2758e3af6ff9169164e2cab0.png)

<h4>Conclusions </h4>
<p>Small but significant differences favoring anonymous discussions were found. Subsequent computer science conferences have drawn on these findings for their policy choices, with a greater inclination toward anonymity in reviewer discussions.</p>
<h4>References</h4>
<p>1. Shah N. An overview of challenges, experiments, and computational solutions in peer review (extended version). July 7, 2025. Accessed July 16, 2025. <a href="https://www.cs.cmu.edu/~nihars/preprints/SurveyPeerReview.pdf"><span class="CharOverride-10">https://www.cs.cmu.edu/~nihars/preprints/SurveyPeerReview.pdf</span></a></p>
<p>2. Chiang W, Li Z, Lin Z, et al. Vicuna: an open-source chatbot impressing GPT-4 with 90%* ChatGPT quality. LMSYSORG. March 2023. <a href="https://lmsys.org/blog/2023-03-30-vicuna/"><span class="CharOverride-10">https://lmsys.org/blog/2023-03-30-vicuna/</span></a></p>
<sup>1</sup>Google DeepMind, New York, NY, US; <sup>2</sup>Carnegie Mellon University, Pittsburgh, PA, US, nihars@cs.cmu.edu; <sup>3</sup>ETH Zurich, Zurich, Switzerland; <sup>4</sup>Max Planck Institute, Tübingen, Germany; <sup>5</sup>New Economic School, Moscow, Russia; <sup>6</sup>University of Maryland, Baltimore, MD, US. <h4>Conflict of Interest Disclosures</h4> As per author affiliations above. In addition, Zhijing Jin is going to join the University of Toronto, and Kun Zhang has a partial appointment at Mohamed bin Zayed University of Artificial Intelligence. <h4>Funding/Support</h4> ONR N000142212181, NSF 1942124, 2200410, 2229881, NIH R01HL159805. <h4>Role of Funder/Sponsor</h4> The funders played no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation. <h4>Acknowledgment</h4> This work was conducted when Charvi Rastogi and Ivan St

Anonymizing Reviewers to Each Other in Peer Review Discussions: A Randomized Controlled Trial

<p>John Carpenter,<sup>1</sup> Andrea Corvillón<sup>1</sup></p>
<h4>Objective </h4>
<p>In 2021,<sup>1</sup> the Atacama Large Millimeter/submillimeter Array (ALMA) transitioned from single-anonymous panel reviews to dual-anonymous distributed peer review to manage the growing volume of proposal submissions and mitigate potential biases.<sup>2</sup> We conducted a retrospective cohort study to examine associations between this procedural change and proposal rankings across principal investigator (PI) demographic characteristics, using 7 years of data under the previous format (2012-2018) and 4 years under the new process (2021-2024).</p>
<h4>Design </h4>
<p>We analyzed proposal rankings from 12 ALMA cycles. From 2011 to 2018 (cycles 0-6), proposals were reviewed in topical panels under single-anonymous peer review. In 2019 (cycle 7), investigator lists were randomized while panels were retained. No review process was held in 2020 due to the COVID-19 pandemic. In 2021 (cycle 8), ALMA implemented dual-anonymous, distributed peer review for most proposals. We examined rankings by 3 PI demographic characteristics: (1) experience (number of cycles in which the PI submitted proposals), (2) regional affiliation (Chile, East Asia, Europe, North America, or other), and (3) sex. We grouped proposals by review era: single-anonymous panel review (cycles 1-6; 2012-2018; 9091 proposals) and dual-anonymous distributed peer review (cycles 8-11; 2021-2024; 6490 proposals). Cycle 0 and cycle 7 were excluded from the analysis: cycle 0 because all PIs were, by definition, first-time users of ALMA, and cycle 7 because it was a transitional year for implementing dual anonymity.</p>
<h4>Results </h4>
<p>Proposal rankings were normalized from 0 (best) to 1 (worst) for comparability across cycles. Proposal counts by demographic subgroup and review era are reported in <b>Table 25-1053</b>. We compared median normalized rankings between review eras using 10,000 bootstrap samples to generate 95% CIs and 2-sided <i>P</i> values. We observed the following associations: (1) for experience, PIs submitting in all cycles had better rankings during single-anonymous panel review than under dual-anonymous distributed review (<i>P</i> = .006), and first-time PIs ranked lowest in both systems with no significant change in rankings (<i>P</i> = .19); (2) for regional affiliation, East Asian PIs showed improved rankings after the transition (<i>P</i> &lt; .001), rankings for European PIs declined (<i>P</i> = .009) but remained above average, and rankings for PIs from North America, Chile, and other regions did not show significant changes (<i>P</i> &gt; .60); and (3) for sex, no statistically significant differences in rankings were observed between male- and female-led proposals in either review system (<i>P</i> = .12).</p>

![](https://assets.underline.io/markdown_image/1/image/ef41fffb9d8404953c23a5d7853259dc.png)

<h4>Conclusions </h4>
<p>Dual-anonymous, distributed peer review was associated with reduced disparities by PI experience and region, consistent with reduced prestige and geographic bias. While increased score variability may have contributed, the nonuniform changes (ie, some groups improved, others remained stable) are inconsistent with a purely noise-driven explanation. These findings suggest that systematic shifts in reviewer behavior, not merely increased randomness, underlie the observed trends. Disentangling the effects of dual anonymity, distributed review, and other concurrent changes (eg, increased use of artificial intelligence) remains an important direction for future research.</p>
<h4>References</h4>
<p>1. Donovan Meyer J, Corvillón A, Carpenter JM, et al. Analysis of the ALMA Cycle 8 distributed peer review process. <i>Bull Am Astron Soc</i>. 2022;54(1):43. doi:10.3847/25c2cfeb.4ece85d4</p>
<p>2. Carpenter JM, Corvillón A, Donovan Meyer J, et al. Update on the systematics in the ALMA proposal review process after Cycle 8. <i>Publ Astron Soc Pac</i>. 2022;134:045001. doi:10.1088/1538-3873/ac5b89</p>
<sup>1</sup>Joint ALMA Observatory, Santiago, Chile, john.carpenter@alma.cl. <h4>Conflict of Interest Disclosures</h4> John Carpenter and Andrea Corvillón are employed by the Joint ALMA Observatory, which is jointly managed by Associated Universities Inc/National Radio Astronomy Observatory, the European Organisation for Astronomical Research in the Southern Hemisphere, and the National Astronomical Observatory of Japan on behalf of the ALMA partnership. <h4>Funding/Support</h4> This work was supported by the Joint ALMA Observatory. <h4>Role of the Funder/Sponsor</h4> The funder had a role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and decision to submit the abstract for presentation. <h4>Acknowledgment</h4> ALMA is a partnership of the European Organization for Astronomical Research in the Southern Hemisphere (representing its member states), National Science Foundation (US), and National Institutes of Natural Sciences (Japan), together with the National Research Council of Canada (Canada), the National Science and Technology Council and Academia Sinica Institute of Astronomy and Astrophysics (Taiwan), and the Korea Astronomy and Space Science Institute (Republic of Korea), in cooperation with the Republic of Chile.



Dual Anonymous and Distributed Peer Review for Proposal Review Rankings at the ALMA Observatory

<p>Christos P. Kotanidis,<sup>1,2,3</sup> Sarah Gorey,<sup>1,4</sup> Harleen Marwah,<sup>1</sup> Abarna Pearl,<sup>1</sup> Darren Taichman,<sup>1</sup> Mary Beth Hamel<sup>1,5</sup></p>
<h4>Objective</h4>
<p> Editorial review forms the cornerstone of scientific publishing, ensuring that published research adheres to the highest quality standards.<sup>1</sup> We evaluated how editorial processes shape manuscripts by examining changes in abstracts between submission and publication.</p>
<h4>Design</h4>
<p> We assessed research articles submitted to the <i>New England Journal of Medicine</i> (<i>NEJM</i>) in 2022 and published in <i>NEJM</i> or rejected and subsequently published in 437 other journals. Using author names, we searched PubMed for articles within 2 years of submission. Four physicians reviewed search results to match published manuscripts with submitted manuscripts. Abstract similarity was quantified using term frequency–inverse document frequency,<sup>2,3</sup> assigning a cosine similarity score ranging from 0 (dissimilar) to 1 (identical). To assess if similarity changes reflected substantive differences, masked physician reviewers conducted qualitative comparisons of submitted and published versions of abstracts from clinical trials. Abstracts were assessed for substantive changes in the following 4 domains: trial design, primary outcome, adverse events, and conclusion (TPAC). A score of +1 was assigned for a substantive domain change favoring the published version and −1 for domain change favoring the submitted version, with the domain scores summed, resulting in a total TPAC score range of −4 (submitted better) to +4 (published better).</p>
<h4>Results</h4>
<p> We matched 1360 PubMed records from 2756 research articles initially submitted to <i>NEJM</i>. Of these, 201 were published in <i>NEJM</i> and 1159 were rejected by <i>NEJM</i> and published in the 437 other journals. The median (IQR) similarity score between submitted and published abstracts was 0.71 (0.58-0.83). Abstracts of articles published in the 5 general medicine journals with a journal impact factor (JIF) above 50 per the 2023 Clarivate report had significantly lower similarity scores compared with those published in other journals (<span class="CharOverride-4"><b>Figure 25-0940</b></span>, A). For qualitative comparisons, we identified abstracts from randomized controlled trials. On average, published abstracts improved in 0.9 domains, with conclusion being the most frequently modified domain (44.2%). Similarity score correlated negatively with TPAC (Spearman ρ coefficient, −0.39; <span class="CharOverride-4"><b>Figure 25-0940</b></span>, B), indicating that lower similarity reflected more substantive revisions. Among abstracts published in the 5 high-JIF general medicine journals, 72.1% improved in at least 1 domain, compared with 48.3% of abstracts published in the other 433 journals. This difference was consistent across all TPAC domains, with conclusion changes being the most different (48% vs 27%).</p>


![](https://assets.underline.io/markdown_image/1/image/bcf1d262f8a68ab13bfbe8e357b30334.png)

<h4>Conclusions</h4>
<p> Our comparisons of submitted and published abstracts suggest that peer review and editorial processes result in substantial revision of research reports. This report focused on submissions to 1 general medical journal and within a 2-year publication window. These findings reinforce the importance of editorial oversight in scientific publishing, highlighting its role in refining study design communication, reporting of outcomes, and overall clarity in research dissemination.</p>
<h4>References</h4>
<p>1. Roll SC. The value and process of high-quality peer review in scientific professional journals. <i>J Diagn Med Sonography</i>. 2019;35(5):359-362. doi:10.1177/8756479319853800</p>
<p>2. Aizawa A. An information-theoretic perspective of tf–idf measures. <i>Inf Process &amp; Manage</i>. 2003;39(1):45-65. doi:10.1016/S0306-4573(02)00021-3</p>
<p>3. Ramos J. Using tf-idf to determine word relevance in document queries. In: <i>Proceedings of the First Instructional Conference on Machine Learning</i>. 2003.</p>
<sup>1</sup>New England Journal of Medicine, Boston, MA, US, ckotanidis@ nejm.org; <sup>2</sup>Heart and Vascular Center, Division of Cardiovascular Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, US; <sup>3</sup>Division of Cardiovascular Medicine, Radcliffe Department of Medicine, University of Oxford, Oxford, UK; <sup>4</sup>Stroke Clinical Trials Network Ireland (SCTNI), University College Dublin Clinical Research Centre, Mater Hospital, Dublin, Ireland; <sup>5</sup>Division of General Medicine and Primary Care, Department of Medicine, Harvard Medical School, Beth Israel Deaconess Medical Center, Boston, MA, US. <h4>Conflict of Interest Disclosures</h4> Christos P. Kotanidis, Sarah Gorey, Harleen Marwah, and Abarna Pearl are Editorial Fellows, Darren Taichman is a Deputy Editor, and Mary Beth Hamel is the Executive Editor at the New England Journal of Medicine.



Changes to Research Article Abstracts Between Submission and Publication

<p>Nicholas LaBerge,<sup>1</sup> Sam Zhang,<sup>1,2,3</sup> Quinten McElhiney,<sup>1</sup> Daniel B. Larremore,<sup>1,2</sup> Aaron Clauset<sup>1,2</sup></p>
<h4>Objective</h4>
<p> To make publicly available a deidentified dataset of manuscript submissions and associated editorial metadata at <i>Science</i> and <i>Science Advances</i> (2 elite multidisciplinary journals) and quantify the manuscript and author characteristics associated with outcomes over the 2 stages of evaluation at these highly selective journals: (1) editorial review, when journal editors screen and select a smaller set of submissions for detailed consideration, and (2) peer review, when editors recruit outside experts to evaluate manuscripts. Peer review at elite scientific journals is a high-stakes process whose outcomes have a broad influence in science and society. However, the need to maintain peer review’s confidentiality has limited the range of data available for scientific study. This lack of peer review data makes it difficult to assess how well elite journals achieve the scientific community’s ideals or to improve and test theories to improve their evaluation processes.</p>
<h4>Design</h4>
<p> We introduced and described a manuscript-level dataset of 110,303 deidentified evaluations of manuscripts submitted to <i>Science</i> and <i>Science Advances</i> over a 5-year period (2015-2019), each with a standard set of author and manuscript characteristics, and we conducted a series of logistic regression analyses to quantify the correlates of success in the initial editorial review stage (desk rejections) and the subsequent peer review stage at both journals.</p>
<h4>Results</h4></span> Each manuscript record includes author, editor, and manuscript characteristics, including topic, team size, institutional prestige, gender, geographic region, evaluation scores, and editor decisions; personally identifiable and reidentifiable information, including the text of the reviews, is excluded. Our statistical analyses revealed strong associations with institutional prestige, team size, manuscript topic, and country, which are primarily attributable (via a mediation analysis) to the influence of the editor, even as the tenor of advice from outside experts correlates strongly with the final editorial decision (<b>Figure 25-0972</b>). Corresponding authors who are men appear to have a small but significant advantage with editors and reviewers at <i>Science</i>, while authors based in China have a significant disadvantage.</p>

![](https://assets.underline.io/markdown_image/1/image/e1bd441e68d25c2bc4c3481b9d8f95e7.png)

<p class="abstract_abstract-text ParaOverride-5"><span class="CharOverride-4"><h4>Conclusions</h4></span> This deidentified dataset will support further investigation by peer review researchers into the 2-stage evaluation process of these 2 elite journals, and our results will help quantify the complex interactions among editors, reviewers, manuscript characteristics, and author characteristics at elite journals.</p>
<sup>1</sup>University of Colorado Boulder, Boulder, CO, US, aaron.clauset@ colorado.edu; <sup>2</sup>Santa Fe Institute, Santa Fe, NM, US; <sup>3</sup>University of Vermont, Burlington, VT, US. <h4>Conflict of Interest Disclosures</h4> Aaron Clauset is a Deputy Editor at Science Advances; all other authors declare no conflicts of interest. <h4>Funding/Support</h4> This work was supported by Air Force Office of Scientific Research Award FA9550-19-1-0329 (to Nicholas LaBerge, Sam Zhang, Daniel B. Larremore, and Aaron Clauset), National Science Foundation (NSF) SBE Award 2219609 (to Nicholas LaBerge, Sam Zhang, Daniel B. Larremore, and Aaron Clauset), NSF Graduate Research Fellowship Award DGE 2040434 (to Sam Zhang), and NSF Alan T. Waterman Award SMA-2226343 (to Daniel B. Larremore). <h4>Role of Funder/Sponsor</h4> The funders had no role in this research.



Manuscript Characteristics Associated With Editorial Review and Peer Review Outcomes at Science and Science Advances

<p>Nicola Adamson,<sup>1</sup> Andy Collings<sup>1</sup></p>
<h4>Objective </h4>
<p>In October 2022, <i>eLife</i> announced a Publish, Review, Curate model<sup>1</sup> in which all peer-reviewed submissions are published as reviewed preprints, accompanied by an <i>eLife</i> Assessment<sup>2</sup> and public reviews. Authors decide which revisions to undertake and when to declare a version of record (VOR), without an accept/reject decision after peer review. We investigated the extent to which authors revise and improve their work.</p>
<h4>Design </h4>
<p>An <i>eLife</i> Assessment is prepared for each submission sent for review and subsequent revisions, which summarizes the strength of evidence and significance of findings using terms from a common vocabulary. We extracted terms used in <i>eLife</i> Assessments for all 2918 reviewed preprints and their subsequent versions to the end of 2024. To ensure publications had completed the entire process and we could evaluate the full extent of changes, we selected the 1504 where a VOR had been declared. From these, 213 were excluded because either the VOR or original reviewed preprint had included multiple terms to describe the evidence. Using a retrospective analysis, we studied the remaining 1291 reviewed preprints and their respective VOR, calculating the distributions of and changes to terms between first and final versions and the number of rounds of revision.</p>
<h4>Results </h4>
<p>Panel A in <b>Figure 25-0989</b> shows the terms used to describe the strength of evidence in VORs and the first version of the reviewed preprints. The evidence term improved between versions in 39.4% of cases (n = 509) and remained the same in 57.7% of VORs (n = 745). Of VORs in which the evidence was described as incomplete in the first reviewed preprint, 76.5% (n = 199) improved to solid or better, and in papers originally described as solid, 49.3% improved to convincing or better (n = 217). In papers in which the evidence was originally described as inadequate (n = 23), this improved in 78.3% of instances (n = 18) and approximately one-half ended as solid or better (47.8%, n = 11). The majority of VORs (82.6%, n = 1066) were declared after 1 round of revision and 13.5% (n = 174) after 2 or more rounds. Of the 4.0% of VORs declared without revisions (n = 51), only 13.7% had an evidence term of inadequate (n = 0) or incomplete (n = 7). The significance of the findings (<b>Figure 25-0989</b>, B) was most frequently described using valuable (VOR: 31.1%, n = 401) or important (VOR: 49.0%, n = 633); overall, significance terms remained the same in 78.2% of cases (n = 1010).</p>

![](https://assets.underline.io/markdown_image/1/image/b3f8c42937978d78fd6b86c9caa9c9d2.png)

<h4>Conclusions </h4>
<p>Authors generally complete at least 1 round of revision, even though they have the option to proceed without, and multiple rounds of revision are uncommon. Where the evidence was originally described as incomplete, the majority of authors revised to improve this before the VOR. Terms used to describe the significance of the findings change less often than evidence terms, perhaps due to significance being more closely linked to the original research question under investigation.</p>
<h4>References</h4>
<p>1. Eisen MB, Akhmanova A, Behrens TE, et al. Scientific publishing: peer review without gatekeeping. <i>eLife</i>. October 20, 2022;11:e83889. doi:10.7554/eLife.83889</p>
<p>2. <i>eLife</i>’s New Model: What is an <i>eLife</i> assessment? <i>eLife.</i> October 20, 2022. Accessed January 30, 2025. <a href="https://elifesciences.org/inside-elife/db24dd46/elife-s-new-model-what-is-an-elife-assessment">https://elifesciences.org/inside-elife/db24dd46/elife-s-new-model-what-is-an-elife-assessment</a></p>
<sup>1</sup>eLife Sciences Publications Ltd, 95 Regent Street, Cambridge, CB2 1AW, UK, n.adamson@elifesciences.org. <h4>Conflict of Interest Disclosures</h4> We have no competing interests to declare. Both authors are employed by eLife Sciences Publications Ltd; we received no external funding for this work. <h4>Acknowledgment</h4> We thank Fiona Hutton, Fred Atherden, Emily Packer, and George Currie for their comments and suggestions on the submission, and Fred Atherden for extracting data for analysis.



Downloads

Next from The 10th Peer Review Congress

Artificial Intelligence Use and Acknowledgment in Medical Research Writing Among Chinese Scholars

Similar lecture

Registered Clinical Trial Trends in East Asia and the United States, 2014 to 2025

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from The 10th Peer Review Congress

Artificial Intelligence Use and Acknowledgment in Medical Research Writing Among Chinese Scholars

Similar lecture

Registered Clinical Trial Trends in East Asia and the United States, 2014 to 2025

Downloads