United States

&lt;p&gt;Qinyue Liu,&lt;sup&gt;1&lt;/sup&gt; Yagmur &lt;span lang=&quot;ar-SA&quot;&gt;Öztürk&lt;/span&gt;,&lt;sup&gt;1&lt;/sup&gt; Cyril Labbé&lt;sup&gt;1&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Objective &lt;/h4&gt;
&lt;p&gt;Abstracts of scientific publications are widely used in scientometrics and text mining because they can be extracted in large quantities from bibliometric databases, and they convey important information about the publications. However, we found misregistered abstracts in Crossref published by &lt;i&gt;Science&lt;/i&gt; and the &lt;i&gt;Proceedings of the National Academy of Sciences&lt;/i&gt; (&lt;i&gt;PNAS&lt;/i&gt;). We aimed at estimating the proportion of misregistered abstracts in our dataset and providing the textual similarity between the correct abstracts and the misregistered texts. A previous study also found misregistered metadata in Crossref.&lt;sup&gt;1&lt;/sup&gt;&lt;/p&gt;
&lt;h4&gt;Design &lt;/h4&gt;
&lt;p&gt;Having recently worked on citation analysis, we developed a script to automatically collect a dataset of citations extracted from articles published between 2016 and 2024 in &lt;i&gt;The Lancet, Cell&lt;/i&gt;, and &lt;i&gt;Joule&lt;/i&gt;. Each citation context is paired with the cited abstract mainly extracted using Crossref API. This dataset includes 19,822 citation context-abstract pairs. We noticed that certain abstracts on Crossref are misregistered. For &lt;i&gt;Science&lt;/i&gt;, a section that introduces and cites the current article, and for &lt;i&gt;PNAS&lt;/i&gt;, a section called Significance, are misregistered as abstracts on Crossref. We developed a script to automatically find these cases from our dataset by searching for distinctive features (string “et al” for &lt;i&gt;Science&lt;/i&gt; and “Significance” for &lt;i&gt;PNAS&lt;/i&gt;). For the cases in &lt;i&gt;Science&lt;/i&gt;, we also manually collected the correct abstracts, and used the Sentence-BERT (SBERT)&lt;sup&gt;2&lt;/sup&gt; model to calculate the textual similarity between the misregistered abstracts and the correct abstracts.&lt;/p&gt;
&lt;h4&gt;Results &lt;/h4&gt;
&lt;p&gt;Of the 821 abstracts of &lt;i&gt;Science&lt;/i&gt; publications, 402 (48%) contained the distinctive features.&lt;i&gt; &lt;/i&gt;We manually verified the results, and 400 of 402 abstracts were misregistered. The 2 correctly registered abstracts were for report types of publication where there was no other text to be misregistered. Of the 669 &lt;i&gt;PNAS&lt;/i&gt; abstracts, 243 (36%) contained Significance at the beginning. Moreover, most of the misregistered abstracts in our dataset, as estimated, had high textual similarity scores (cosine similarity) with the correct abstracts. The scores range from 0.58 to 0.99, with an average of 0.83. This phenomenon occurs because the misregistered abstracts summarize the article and mention key points, resembling the original abstracts.&lt;/p&gt;
&lt;h4&gt;Conclusions &lt;/h4&gt;
&lt;p&gt;Metadata misregistration in large repositories like Crossref can compromise data quality. Although our method detected misregistered abstracts in a small portion of the dataset using distinctive features, its scalability requires validation with larger datasets.&lt;/p&gt;
&lt;h4&gt;References&lt;/h4&gt;
&lt;p&gt;1. Cioffi A, Coppini S, Massari A, et al. Identifying and correcting invalid citations due to DOI errors in Crossref data. &lt;i&gt;Scientometrics&lt;/i&gt;. 2022;127:3593-3612. doi:10.1007/s11192-022-04367-w&lt;/p&gt;
&lt;p&gt;2. Reimers N, Gurevych I. Sentence-BERT: Sentence embeddings using Siamese BERT-Networks. In: &lt;i&gt;Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)&lt;/i&gt;. Association for Computational Linguistics; 2019:3982-3992. doi:10.18653/v1/D19-1410&lt;/p&gt;
&lt;sup&gt;1&lt;/sup&gt;Université Grenoble Alpes, French National Centre for Scientific Research, Grenoble INP, Laboratoire d’Informatique de Grenoble, Grenoble, France, qinyue.liu@univ-grenoble-alpes.fr.&lt;/p&gt;
&lt;h4&gt;Conflict of Interest Disclosures&lt;/h4&gt;
&lt;p&gt; None reported.&lt;/p&gt;
&lt;h4&gt;&lt;a id=&quot;_idTextAnchor041&quot;&gt;&lt;/a&gt;Funding/Support&lt;/h4&gt;
&lt;p&gt; &lt;a id=&quot;_idTextAnchor042&quot;&gt;&lt;/a&gt;The NanoBubbles project has received Synergy grant funding from the European Research Council, within the European Union’s Horizon 2020 program, grant 951393.&lt;/p&gt;
&lt;h4&gt;Role of Funder/Sponsor &lt;/h4&gt;
&lt;p&gt;The European Research Council had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.&lt;/p&gt;


The 10th Peer Review Congress

[V] Science Journal Abstracts Misregistered in the Crossref Database

repositories

and scientometrics

informatics

dissemination of information

bibliometrics

The Tenth Peer Review Congress featured 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

This year’s meeting was hybrid, and all plenary sessions were livestreamed with opportunities to view presentations and engage with participants in person and virtually during and after the meeting. All recordings are now available on Underline for viewing.

<div style="position:relative;padding-top:max(60%,324px);width:100%;height:0;"><iframe style="position:absolute;border:none;width:100%;height:100%;left:0;top:0;" src="https://online.fliphtml5.com/ebtyf/mget/"  seamless="seamless" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" ></iframe></div>


#### Why? 
Our aim is to encourage research into the quality and credibility of peer review and scientific publication, to establish the evidence base on which scientists can improve the conduct, reporting, and dissemination of scientific research.

#### Who? 
Participants include editors and publishers of scientific peer-reviewed journals, researchers, funders, bibliometric and informatics experts, information innovators, librarians, journalists, policymakers, ethicists, scientific information disseminators, and anyone interested in the progress of the scientific information enterprise and the quality of scientific evidence.

The Congress embraces a wide range of disciplines, including biomedicine, health and life sciences, applied sciences, basic sciences, physical and chemical sciences, mathematics, computer sciences, engineering, economics, and social sciences as well as new and emerging disciplines.

#### What? 
The Congress provides a forum for the presentation and discussion of new research into the quality and credibility of editorial and funding peer review, scientific and scholarly publication, research methods and reporting practices, identification and management of biases, transparency, reproducibility, access, publication and dissemination metrics, identification and management of threats to the integrity of science, and advances in information exchange.

**Follow us**

[![](https://assets.underline.io/markdown_image/1/image/662a22766b9788ed3766cb23cd156e32.png)](https://x.com/peerrevcongress)  [![](https://assets.underline.io/markdown_image/1/image/bb105afb1773a01fa169aa97ca1af49a.png)](https://www.youtube.com/channel/UCsG_0nKi3_Zfyv0yRPdxjWQ)  [![](https://assets.underline.io/markdown_image/1/image/034e1e70a194a9a4178fdf470ef796db.png)](https://bsky.app/profile/peerreviewcongress.bsky.social)



You need to log in with the email address you registered with. 



Please log in to explore this event.

Registration is required to attend this event, please [here](https://peerreviewcongress.org/registration-information/) to secure your ticket. Please check your spam/promotional inbox too.



Please register!

The Tenth Peer Review Congress features 3 days for presentations of new research into peer review and all aspects of scientific publication, bias, quality of reporting, and information access and dissemination.

<p>Qinyue Liu,<sup>1</sup> Yagmur <span lang="ar-SA">Öztürk</span>,<sup>1</sup> Cyril Labbé<sup>1</sup></p>
<h4>Objective </h4>
<p>Abstracts of scientific publications are widely used in scientometrics and text mining because they can be extracted in large quantities from bibliometric databases, and they convey important information about the publications. However, we found misregistered abstracts in Crossref published by <i>Science</i> and the <i>Proceedings of the National Academy of Sciences</i> (<i>PNAS</i>). We aimed at estimating the proportion of misregistered abstracts in our dataset and providing the textual similarity between the correct abstracts and the misregistered texts. A previous study also found misregistered metadata in Crossref.<sup>1</sup></p>
<h4>Design </h4>
<p>Having recently worked on citation analysis, we developed a script to automatically collect a dataset of citations extracted from articles published between 2016 and 2024 in <i>The Lancet, Cell</i>, and <i>Joule</i>. Each citation context is paired with the cited abstract mainly extracted using Crossref API. This dataset includes 19,822 citation context-abstract pairs. We noticed that certain abstracts on Crossref are misregistered. For <i>Science</i>, a section that introduces and cites the current article, and for <i>PNAS</i>, a section called Significance, are misregistered as abstracts on Crossref. We developed a script to automatically find these cases from our dataset by searching for distinctive features (string “et al” for <i>Science</i> and “Significance” for <i>PNAS</i>). For the cases in <i>Science</i>, we also manually collected the correct abstracts, and used the Sentence-BERT (SBERT)<sup>2</sup> model to calculate the textual similarity between the misregistered abstracts and the correct abstracts.</p>
<h4>Results </h4>
<p>Of the 821 abstracts of <i>Science</i> publications, 402 (48%) contained the distinctive features.<i> </i>We manually verified the results, and 400 of 402 abstracts were misregistered. The 2 correctly registered abstracts were for report types of publication where there was no other text to be misregistered. Of the 669 <i>PNAS</i> abstracts, 243 (36%) contained Significance at the beginning. Moreover, most of the misregistered abstracts in our dataset, as estimated, had high textual similarity scores (cosine similarity) with the correct abstracts. The scores range from 0.58 to 0.99, with an average of 0.83. This phenomenon occurs because the misregistered abstracts summarize the article and mention key points, resembling the original abstracts.</p>
<h4>Conclusions </h4>
<p>Metadata misregistration in large repositories like Crossref can compromise data quality. Although our method detected misregistered abstracts in a small portion of the dataset using distinctive features, its scalability requires validation with larger datasets.</p>
<h4>References</h4>
<p>1. Cioffi A, Coppini S, Massari A, et al. Identifying and correcting invalid citations due to DOI errors in Crossref data. <i>Scientometrics</i>. 2022;127:3593-3612. doi:10.1007/s11192-022-04367-w</p>
<p>2. Reimers N, Gurevych I. Sentence-BERT: Sentence embeddings using Siamese BERT-Networks. In: <i>Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)</i>. Association for Computational Linguistics; 2019:3982-3992. doi:10.18653/v1/D19-1410</p>
<sup>1</sup>Université Grenoble Alpes, French National Centre for Scientific Research, Grenoble INP, Laboratoire d’Informatique de Grenoble, Grenoble, France, qinyue.liu@univ-grenoble-alpes.fr.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4><a id="_idTextAnchor041"></a>Funding/Support</h4>
<p> <a id="_idTextAnchor042"></a>The NanoBubbles project has received Synergy grant funding from the European Research Council, within the European Union’s Horizon 2020 program, grant 951393.</p>
<h4>Role of Funder/Sponsor </h4>
<p>The European Research Council had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.</p>


poster

<p>Younwoo Ki,<sup>1,2</sup> Chungsoo Kim,<sup>3,4</sup> Yuan Lu,<sup>3</sup> Joshua D. Wallach,<sup>5</sup> Behnood Bikdeli,<sup>4,6,7</sup> Milton Packer,<sup>8,9</sup> Harlan M. Krumholz,<sup>4</sup> Seng Chan You<sup>1,2</sup></p>
<h4>Objective </h4>
<p>Recent trends in cardiovascular journals suggest a growing emphasis on publishing guidelines, reviews, and statements.<sup>1,2</sup> Citations, often used to assess the scholarly impact of journals, also influence researcher evaluation and library purchasing decisions. To understand these changes and their implications, we examined the evolving contribution of original research to citation metrics in leading cardiovascular journals.</p>
<h4>Design </h4>
<p>Our cross-sectional bibliometric study used Scopus to identify all articles published in the <i>Journal of the American College of Cardiology</i> (JACC), <i>Circulation</i> (CIRC), and <i>European Heart Journal</i> (EHJ) from 2008 to 2018, with associated citations. Using section headings on respective journal websites and article metadata (title, abstract, and pages), we manually classified articles into (1) original research articles (ORAs, including nonsystematic meta-analyses); (2) reviews as nonoriginal research articles (NORA-reviews, narrative or systematic); (3) guidelines, statements, and statistics as NORAs (NORA-GSS), and (4) uncitable items (brief reports, case reports, commentaries, and editorials). Uncitable items were excluded from analysis. Outcomes were (1) ORAs’ and NORAs’ mean 2-year citation counts, (2) proportion of 2-year citations by each category among total 2-year citations to all citable items, and (3) publication count and proportions of each category among total citable items.</p>
<h4>Results</h4>
<p> Among 22,740 articles, 9117 (39.9%) were ORAs, 2286 (10.0%) were NORA-reviews, 909 (4.0%) were NORA-GSS, and 10,566 (46.2%) were uncitable items. From 2008 to 2018, Journal Impact Factors (JIFs) in each journal increased, but their 2-year mean (SD) citations of ORA plateaued (JACC, 24.9 [24.3] to 26.4 [28.9]; CIRC, 24.00 [26.3] to 23.4 [23.6]; and EHJ, 15.8 [16.9] to 25.2 [20.8]) (<b>Figure 25-0977</b>, A). The annual publication of ORAs among total citable items declined across all 3 journals, although to varying extents (counts with proportions: JACC, 333 [80.5] to 245 [65.8]; CIRC, 415 [79.2] to 267 [78.1]; and EHJ, 251 [87.9] to 165 [69.6]), reflecting the decreased proportion of ORA 2-year citations among total citations (proportion of 2-year citations: JACC, 71.9% [9249/12,858] to 55.1% [7561/13,728]; CIRC, 68.1% [10,657/15,646] to 55.8% [6708/12,027]; and EHJ, 70.1% [4023/5739] to 36.8% [5073/13,771]) (<b>Figure 25-0977</b>, B). Although the annual publications of NORA-GSS were small in portion (counts with proportions: JACC, 33 [7.5%] to 22 [5.7%]; CIRC, 40 [7.0%] to 47 [13.0%]; and EHJ, 9 [3.1%] to 29 [11.9%]), the proportion of their 2-year citations among the total substantially increased (proportion of 2-year citations: JACC, 10.6% [1362/12,858] to 20.0% [2749/13,728]; CIRC, 19.3% [3019/15,646] to 37.7% [4535/12,027]; and EHJ, 16.6% [955/5739] to 52.4% [7211/13,771]) (<b>Figure 25-0977</b>, B).</p>

![](https://assets.underline.io/markdown_image/1/image/047d3162fe848a6fbfe0c11af9284812.png)

<h4>Conclusions </h4>
<p>Rising JIFs in major cardiology journals were associated primarily with an expanding share of nonoriginal content. Overreliance on JIF in academic evaluation may unconsciously steer journals toward prioritizing highly cited nonoriginal content, systematically undermining the value of primary research.</p>
<h4>References</h4>
<p>1. Genuis SJ. The proliferation of clinical practice guidelines: professional development or medicine-by-numbers? <i>J Am Board Fam Pract</i>. 2005;18(5):419-425. doi:10.3122/jabfm.18.5.419 </p>
<p>2. Shuaib W, Khan MS, Shahid H, Valdes EA, Alweis R. Bibliometric analysis of the top 100 cited cardiovascular articles. <i>Am J Cardiol</i>. 2015;115(7):972-981. doi:10.1016/j.amjcard.2015.01.029</p>
<sup>1</sup>Institute for Innovation in Digital Healthcare, Yonsei University, Seoul, Korea; <sup>2</sup>Department of Biomedical Systems Informatics, Yonsei University College of Medicine, Seoul, Korea, chandryou@yuhs.ac; <sup>3</sup>Section of Cardiovascular Medicine, Department of Internal Medicine, Yale University School of Medicine, New Haven, CT, US; <sup>4</sup>Center for Outcomes Research and Evaluation, Yale New Haven Hospital, New Haven, CT, US; <sup>5</sup>Department of Epidemiology, Rollins School of Public Health, Emory University, Atlanta, GA, US; <sup>6</sup>Division of Cardiovascular Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, US; <sup>7</sup>Thrombosis Research Group, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, US; <sup>8</sup>Baylor Heart and Vascular Institute, Baylor University Medical Center, Dallas, TX, US; <sup>9</sup>Imperial College London, London, UK.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>Several authors hold editorial positions at the Journal of the American College of Cardiology (JACC): Yuan Lu and Behnood Bikdeli as executive associate editors, Joshua D. Wallach and Seng Chan You as associate editors, Milton Packer as senior consulting editor, and Harlan M. Krumholz as editor in chief. Joshua D. Wallach reported receiving grants from Johnson &amp; Johnson (through the Yale Open Data Access Project), Arnold Ventures, the National Institutes of Health, and the US Food and Drug Administration, as well as consulting fees from Hagen Berman Sobol Shapiro LLP and Dugan Law Firm APLC outside the submitted work. Seng Chan You reported being a chief technology officer of PHI Digital Healthcare and receiving grants from Daiichi Sankyo outside the submitted work.</p>
<h4>Funding/Support</h4>
<p> This study was supported by a Severance Hospital Research Fund for Clinical Excellence (SHRC) (C-2024-0011).</p>


[V] Trends in Citation Impacts of Original Research in Major Cardiovascular Journals, 2008-2018

<p>S. Scott Graham,<sup>1</sup> Joshua B. Barbour,<sup>2</sup> Zoltan P. Majdik,<sup>3</sup> Madeline Bruegger,<sup>1</sup> Carlee A. Baker,<sup>1</sup> Justin F. Rousseau<sup>4</sup></p>
<h4>Objective</h4>
<p> Research on industry funding and author conflicts of interest (COIs) suggests that centralization of funding may lead to detrimental effects on biomedical research.<sup>1,2</sup> This study was conducted to assess relationships between COI networks and drug adverse event rates reported to the US Food and Drug Administration.</p>

<h4>Design</h4> We used a GPT 4.0 pipeline (a series of computational steps applied to a dataset for transformation and/or analysis) to identify and classify reported author COIs and funders associated with 302 of the most commonly used drugs in the US. Included articles included up to 150 of the most relevant PubMed results when searching for each drug’s proprietary and generic name. For each declared COI, the funder was identified and funder types were classified by a fine-tuned GPT 4.0 model as industry, federal, university, patient advocacy organization, professional society, university, or hospital/clinic. COI relationships for each drug product were assembled into network models mapping relationships among all reported funders and all articles for each product. Each network was evaluated for network robustness by measuring critical failure threshold (CFT) (ie, the minimum number of nodes that would need to be removed to result in a 50% collapse of the network).<sup>3</sup> COI network robustness is a useful measure of the centralization of the funding network as a low number of funder networks would have lower failure thresholds. A network with many independent funders would require more nodes to be removed before network failure. Controlling for COI rates, drug features (over-the-counter availability, generic availability, US Drug Enforcement Administration schedule, usage rates), and publication features (mean article year of publication, total authorships in the network), we evaluated the relationship between network robustness and reported adverse events rates (total, serious, and fatal). A secondary model was fit to determine whether individual funder type frequencies or heterogeneity were associated with failure threshold. The GPT classifier was compared with a human annotated sample and evaluated for precent accuracy.</p>
<h4>Results</h4>
<p> We identified 37,993 articles associated with the 302 drugs. The GPT 4.0 classifier achieved 98.9% accuracy compared with the human annotated sample. In the adverse events models, CFT returned adjusted incident rate ratios (IRRs) of 0.94 (95% CI, 0.91-0.97; <i>P</i> &lt; .01) for total events, 0.94 (95% CI, 0.91-0.97, <i>P</i> &lt; .01) for serious events, and 0.94 (95% CI, 0.91-0.98; <i>P</i> &lt; .01) for fatal events. The number of funders per network was not a significant factor in any adverse events model. The network features model (<b>Table 25-0837</b>) returned significant relationships between the CFT and the number of funders in the top 10% by out-degree, a network science measure of the number of outgoing connections from a given node (IRR, 1.03 [95% CI, 1.02-1.04]; <i>P</i> &lt; .001) and funder-type heterogeneity (IRR, 1.19 [95% CI, 1.12-1.26]; <i>P</i> &lt; .001).</p>

![](https://assets.underline.io/markdown_image/1/image/c1c38c93f45a0d4c889dd7b12468fb91.jpg)

<h4>Conclusions</h4>
<p> In this study, COI network robustness was inversely associated with adverse events, including serious and fatal events, reported in drug studies. Secondary analyses indicate that this was associated with funding network centralization and homogeneity. The products evaluated in this study were associated with lower adverse event rates when the collected research had a larger number of funders, regardless of funder type. This suggests that research funding policy should encourage funding diversity.</p>
<h4>References</h4>
<p class="abstract_reference-text ParaOverride-4">1. Graham SS, Harrison KR, Edward JC, Majdik ZP, Barbour JB, Rousseau JF. Beyond bias: aggregate approaches to conflicts of interest research and policy in biomedical research. <i>World Med Health Policy</i>. 2024;16(3):489-505. doi:10.1002/wmh3.608.</p>
<p class="abstract_reference-text ParaOverride-4">2. Sismondo S. Ghost management: how much of the medical literature is shaped behind the scenes by the pharmaceutical industry? <i>PLoS Med</i>. 2007;4(9):e286. doi:10.1371/journal.pmed.0040286</p>
<p class="abstract_reference-text ParaOverride-4">3. Albert R, Jeong H, Barabási AL. Error and attack tolerance of complex networks. <i>Nature</i>. 2000;406(6794):378-382. doi:10.1038/35019019</p>
<sup>1</sup>Department of Rhetoric &amp; Writing, The University of Texas at Austin, Austin, TX, US, ssg@utexas.edu; <sup>2</sup>Department of Communication, University of Illinois Urbana-Champaign, Urbana, IL, US; <sup>3</sup>Department of Communication, North Dakota State University, Fargo, ND, US; <sup>4</sup>Department of Neurology &amp; Peter O’Donnell Jr. Brain Institute, The University of Texas Southwestern Medical Center, Dallas, TX, US.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>S. Scott Graham received grant funding from the National Institute of General Medical Sciences (NIGMS). Joshua B. Barbour received grant funding from the NIGMS, NSF, IC2, and Blue Cross Blue Shield; royalties form Cengage Publishing; and consulting fees from The Aslan Group and BrainCheck. Justin F. Rousseau received grant funding from the NIGMS, National Library of Medicine, National Institute on Aging, National Institute of Allergy and Infectious Diseases, National Institute of Mental Health, National Center for Advancing Translational Sciences, Texas Child Mental Health Care Consortium, Michal &amp; Susan Dell Foundation, Texas Alzheimer’s Research and Care Consortium, Austin Public Health, and the Health Care Cost Institute. No other disclosures were reported.</p>
<h4>Funding/Support</h4>
<p> Funding for this research was provided by the NIGMS of the National Institutes of Health under award number R01GM141476.</p>
<p class="abstract_conflict ParaOverride-7" lang="fr-FR"><h4>Role of the Funder/Sponsor</h4> The sponsor had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation.</p>


[V] Conflict of Interest Network Robustness and Funder Homogeneity Associated With Reported Adverse Events and Deaths in Published Drug Studies

Alessandro Bianconi,<sup>1</sup> Matteo Fiore,<sup>1</sup> Maria Elena Flacco,<sup>2</sup> Lamberto Manzoli<sup>1</sup></p>
<h4>Objective </h4>Despite global initiatives to improve transparency regarding conflicts of interest (COIs) in scientific publications, the disclosure of competing interests remains inadequate among authors of clinical practice guidelines (CPGs), as well as for other types of publications.<sup>1,2</sup> Open access databases like Open Payments offer a potential resource for assessing undisclosed COIs in scientific literature.<sup>3</sup> This study evaluates the prevalence of undisclosed industry-related financial COIs among authors of US-issued CPGs on obesity management from 2016 to present.</p>
<h4>Design </h4>On June 3, 2024, we identified CPGs on obesity management issued by US agencies through 3 repositories (UpToDate, Medscape, and Guidelines International Network). COI statements for each guideline were extracted. To identify undisclosed industry-related financial COIs, we cross-referenced authors who reported no competing interests with the Open Payments database,<sup>3</sup> considering only any form of payment and fund received prior to the guideline publication date. Undisclosed COIs were classified based on cumulative amounts as greater than or equal to $10,000 and less than $10,000. The presence of COIs was summarized as absolute and relative frequencies. Given the purely descriptive nature of this study, no hypotheses were tested.</p>
<h4>Results </h4>
<p>We retrieved 5 CPGs (American Academy of Pediatrics 2023, American Diabetes Association 2023, American Gastroenterological Association 2022, Veteran Affairs/Department of Defense 2020, Association of American Clinical Endocrinologists/American College of Endocrinology 2016) (<b>Table 25-1155</b>). Among 145 authors, 80 (55.2%) received payments from industry agents related to obesity prevention or treatment. Of these, 33 authors (41.3% [22.8% overall]) had undisclosed COIs. Among undisclosed COIs, 11 (33.3%) were greater than or equal to $10,000 and 22 (66.7%) were less than $10,000. Every guideline had at least 1 chair or first author with industry-related COIs. Seven of 9 (77.8%) of these prominent authors did not disclose any COI, though all undisclosed COIs were less than $10,000.</p>

![](https://assets.underline.io/markdown_image/1/image/87bbfb86490687a5c34e03c17da6b571.png)

<h4>Conclusions </h4>
<p>We found a relevant amount of undisclosed COIs in obesity management CPGs, and this amount may be underestimated because only financial interests could be evaluated. The presence of undisclosed COIs among CPG authors may reflect the complexities of COI disclosure rather than a straightforward lack of transparency. Variations in COI policies across scientific societies and publishers, which indicate the amount and type of payments that represent a COI that should be disclosed, may result in inconsistent reporting standards. This inconsistency presents an opportunity to harmonize COI management policies across publishing entities, promoting more comprehensive disclosure practices. Implementing regular monitoring of COI statements by peer reviewers and editors can further enhance transparency and trustworthiness. Tools like Open Payments can aid this process, though they have limitations, including potential inaccuracies and exclusion of nonfinancial COIs. Addressing these issues can strengthen the reliability of CPGs and other scientific publications. Further research should investigate the prevalence of undisclosed and overall COIs in other fields, assessing their potential impact on guideline recommendations and, thus, on real-world clinical practice.</p>
<h4>References</h4>
<p>1. Khan R, Scaffidi MA, Rumman A, Grindal AW, Plener IS, Grover SC. Prevalence of financial conflicts of interest among authors of clinical guidelines related to high-revenue medications. <i>JAMA Intern Med</i>. 2018;178(12):1712-1715. doi:10.1001/jamainternmed.2018.5106</p>
<p>2. Lenzer J, Hoffman JR, Furberg CD, Ioannidis JPA. Ensuring the integrity of clinical practice guidelines: a tool for protecting patients. <i>BMJ</i>. 2013;347:f5535. doi:10.1136/bmj.f5535</p>
<p>3. Open Payments. Accessed February 14, 2025. <a href="https://openpaymentsdata.cms.gov"><span class="Hyperlink CharOverride-6">https://openpaymentsdata.cms.gov</span></a></p>
<sup>1</sup>Department of Medical and Surgical Sciences, University of Bologna, Bologna, Italy, alessandro.bianconi4@studio.unibo.it; <sup>2</sup>Department of Environmental and Prevention Sciences, University of Ferrara, Ferrara, Italy.</p>
<h4>Conflict of Interest Disclosures </h4>
<p>None reported.</p>


[V] Disclosed and Undisclosed Conflicts of Interest in US Guidelines for the Management of Obesity

<p>Oludele M. Solaja<sup>1</sup></p>
<h4>Objective</h4>
<p> Scientific peer review often excludes Indigenous knowledge (IK), favoring Western scientific norms. In Nigeria, valuable local insights on health and environmental practices are marginalized, limiting their impact and perpetuating epistemic injustice.<sup>1</sup> Integrating IK into peer review can enhance research relevance, cultural acceptance, and credibility, providing region-specific solutions crucial for sustainable development in Nigerian communities. This research examines how IK can be incorporated into scientific peer review to remove epistemological racism and improve the representation and practicality of IK in Nigerian environment and health research.<sup>2</sup></p>
<h4>Design</h4>
<p> The study follows a 3-phase approach: (1) literature search, (2) stakeholder consultation, and (3) development and pilot testing of guidelines. In phase 1, a comprehensive literature review of 142 peer-reviewed articles (identified via PubMed, Scopus, and African Journals Online) was conducted to identify gaps in IK recognition within environmental and health research in Nigeria. Phase 2 included 396 surveys, 30 semistructured interviews, and 3 focus group discussions involving Nigerian researchers, peer reviewers, and IK holders. This engagement was conducted to highlight cultural barriers and biases in existing peer review practices. Phase 3 involved developing standardized guidelines based on stakeholder feedback to incorporate IK into peer review, followed by pilot testing with a subset of 12 articles from 3 Nigerian journals (<i>Nigerian Journal of Environmental Sciences and Technology</i>, <i>FUTA Journal of Research in Sciences</i>, <i>Ibadan Journal of the Social Sciences</i>) from April to June 2025 to assess feasibility and effectiveness. In this phase, structured reviewer and editorial feedback was collected and is currently being analyzed to evaluate key metrics, including reviewer satisfaction (measured through structured postreview surveys), inclusivity ratings (assessed via 5-point Likert scale responses and demographic tracking of reviewer diversity), and manuscript quality (evaluated by independent editorial panels comparing clarity, cultural sensitivity, and scientific rigor between IK-inclusive and conventional reviews).</p>
<h4>Results</h4>
<p> Among the 142 articles reviewed, only 18 (12.7%) referenced IK, and none demonstrated structured inclusion of IK in peer review or evaluation criteria. In total, 32 of 38 IK stakeholders (84.2%) and 50 of 64 researchers (78.1%) perceived current peer review practices as exclusionary, citing language bias, absence of culturally grounded criteria, and a lack of reviewers who were knowledgeable in IK.<sup>3</sup> Pilot testing yielded the following outcomes: 19 of 24 reviewers (79.2%) reported increased clarity and confidence in evaluating IK-informed manuscripts; inclusivity ratings increased from a baseline mean of 2.8 to 4.3 on the 5-point Likert scale; and independent editorial panels reported that IK-inclusive reviews scored 25% to 30% higher in cultural sensitivity and contextual relevance without compromising scientific rigor. Stakeholders reported that integrating IK leads to a more holistic and locally resonant understanding of health and environmental challenges in Nigeria.</p>
<h4>Conclusions</h4>
<p> Integrating IK into peer review can make research validation more equitable and locally relevant, aligning with global calls for inclusivity in scientific processes. This framework, grounded in empirical findings, may serve as a model for adapting peer review standards to respect diverse knowledge systems and improve the societal impact of research.</p>
<h4>References</h4>
<p>1. Agrawal A. Dismantling the divide between indigenous and scientific knowledge. <i>Dev Change</i>. 1995;26(3):413-439. doi:10.1111/j.1467-7660.1995.tb00560.x</p>
<p>2. Smith LT. <i>Decolonizing Methodologies: Research and Indigenous Peoples</i>. Zed Books; 1999.</p>
<p>3. Nakashima D, Roué M. Indigenous knowledge, peoples, and sustainable practice. In: Timmerman P, ed. <i>Encyclopedia of Global Environmental Change</i>. John Wiley &amp; Sons; 2002:314-324.</p>
<sup>1</sup>Department of Sociology, Olabisi Onabanjo University, Ago-Iwoye, Nigeria, solaja.oludele@oouagoiwoye.edu.ng.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>


[V] Integrating Indigenous Knowledge into Peer Review Processes in Nigerian Environmental and Health Research

<p>Hong Chen,<sup>1</sup> Christopher I. Rider,<sup>2</sup> David Jurgens,<sup>1,3</sup> Misha Teplitskiy<sup>1</sup></p>
<h4>Objective</h4>
<p> This study examines which intended scientific contributions become published contributions. While many manuscripts are rejected, there is limited understanding of which researchers successfully navigate rejection and resubmission. We investigate geographical disparities in postrejection publishing outcomes, focusing on differences between authors from Western and non-Western countries. We investigate several potential mechanisms underlying these disparities, especially procedural knowledge of how to interpret editorial decisions for rejection, revision, and resubmission.</p>
<h4>Design</h4>
<p> Partnering with the Institute of Physics Publishing, we collected metadata on 126,000 rejected submissions made in 2018 through 2022 from 62 peer-reviewed journals. We tracked whether, when, and where these rejected manuscripts were eventually published by developing a classifier that we validated with ground-truth data provided by the authors of rejected manuscripts. We defined Western authors as those affiliated with (self-reported) institutions in North America, Europe, or Oceania, and non-Western authors as those from Asia, Africa, and Latin America. The analysis examined the publishing disparities based on author geography while controlling for key confounders—perceived submission quality, proxied by peer reviewer evaluations at the time of rejection (including desk rejection)—as well as time, team composition, and corresponding author characteristics, such as gender, seniority, and current institutional affiliation ranking. The prior Western coauthors and Western affiliations were also included, serving as proxies for access to procedural knowledge. To further probe procedural knowledge as a mechanism, we examined multiple dimensions of how authors interpret and respond to rejection in a survey of corresponding authors whose manuscripts were rejected.</p>
<h4>Results</h4>
<p> Among the authors included, 43% had Western affiliations and 57% had non-Western affiliations. Authors from Western countries had better postrejection outcomes. Despite similar perceived quality at the time of rejection, Western authors were 6.7% more likely to publish the rejected papers elsewhere. Among those who ultimately published (59.3% of all rejected papers), Western authors published approximately 23 days faster and did 5.9% less abstract revising and 12.0% less authorship team change. We also found evidence suggesting that greater access to procedural knowledge, proxied by prior publishing experience and prior Western affiliation and coauthorship, contributed to better postrejection outcomes. These factors explain a substantial portion of the observed disparities in postrejection publication rates, more so than perceived submission quality or institutional resourcefulness. We also surveyed 10,000 corresponding authors of rejected manuscripts in June 2023 and received 287 responses. Limited by the very low response rate, survey responses indicated no significant differences between Western and non-Western authors in their perceptions of feedback fairness or negativity, planned revisions, or target outlet prestige, suggesting that other dimensions of procedural knowledge might be key.</p>
<h4>Conclusions</h4>
<p> This study identified geographical disparities in the file drawer of science. Western authors experienced better postrejection outcomes than non-Western authors, and the findings suggest that differential access to procedural knowledge may contribute to these disparities.</p>
<sup>1</sup>School of Information, University of Michigan, Ann Arbor, MI, US, tepl@umich.edu; <sup>2</sup>Ross School of Business, University of Michigan, Ann Arbor, MI, US; <sup>3</sup>Department of Computer Science and Engineering, University of Michigan, Ann Arbor, MI, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>


[V] Geographical Disparities In Navigating Rejection In Science

<p>Patrick Amboka,<sup>1</sup> Tony Blair Aloo,<sup>1</sup> Daniel Krugman,<sup>1,2</sup> Abel Simiyu,<sup>1</sup> Hiram Kariuki,<sup>1</sup> Benard Ondiek,<sup>1</sup> Nosa Orobaton,<sup>3</sup> Emmy Igonya,<sup>1</sup> Alphonsus Neba,<sup>1</sup> Marta Vicente-Crespo,<sup>1,4</sup> Julius Kirimi Sindi<sup>1</sup></p>
<h4>Objective</h4>
<p> This study explored editorial practices among African journals, examined the factors influencing these practices, and assessed authors’ perspectives and preferences in selecting journals for publication. A large proportion of African journals (63.2%) are neither discoverable via Google Scholar nor included in Scopus.<sup>1</sup> However, their discoverability and inclusion in international platforms are nuanced and not always straightforward; hence, quality markers need to be better aligned.</p>
<h4>Design</h4>
<p> This study triangulated multiple sources of information and used qualitative data gathering techniques to capture nuances and provide deeper insights into the performance and visibility of African journals. Conducted between July and December 2024, the study utilized a cross-sectional design. In-depth interviews (n = 60), key informant interviews (n = 32), and focus group discussions (n = 7) were conducted in Kenya, Ethiopia, Nigeria, and Mozambique. A purposive sampling technique was used to identify participants. Ethical approvals were obtained from relevant institutions. We obtained consent from participants to record and publish the data in anonymized form. Qualitative data from the audio-recorded interviews were transcribed using Microsoft Word and exported to NVivo software for analysis. The data were analyzed using a reflexive thematic analysis.<sup>2</sup> Emerging concepts from the qualitative data were also analyzed using Leximancer software.</p>
<h4>Results</h4>
<p> Four main themes emerged from participant interviews in this study. First, there are challenges in adhering to international editorial standards (“We often want to follow international best practices, but there’s limited guidance and mentorship available to help us meet those expectations consistently”). Second, financial constraints undermine editorial integrity (“Sometimes, we’re forced to accept submissions not based on quality, but because we need the processing fees to keep the journal running. It’s not ideal, but without funding, we have few options”). Third, technological and digital infrastructure gaps exist (“Our journal management system is outdated, and we lack IT support to upgrade it”). Fourth, authors have perceptions of inferiority and trust issues (“Authors often tell us they prefer to submit to Western journals because they assume ours won’t be taken seriously”).</p>
<h4>Conclusions</h4>
<p> Addressing financial constraints, peer review inefficiencies, and infrastructure gaps is critical to strengthening the editorial practices and credibility of African journals. Overcoming historical neglect through sustainable funding, capacity building, and technological advancement will be key to enhancing their global visibility, trust, and academic impact.</p>
<h4>References</h4>
<p>1. Amboka P, Sindi JK, Wamukoya M, et al. Discoverability of African journals by Google Scholar and inclusion in Scopus. VeriXiv. Preprint posted online November 1, 2024. <a href="https://verixiv.org/articles/1-17"><span class="Hyperlink CharOverride-6">https://verixiv.org/articles/1-17</span></a></p>
<p>2. Byrne D. A worked example of Braun and Clarke’s approach to reflexive thematic analysis. Qual Quant. 2022;56:1391-1412. doi:10.1007/s11135-021-01182-y</p>
<sup>1</sup>African Population and Health Research Center, Nairobi, Nairobi County, Kenya, pamboka@aphrc.org; <sup>2</sup>Department of Anthropology, Brown University, Providence, RI, US; <sup>3</sup>Bill &amp; Melinda Gates Foundation, Seattle, WA, US; <sup>4</sup>School of Public Health, University of the Witwatersrand, Johannesburg, South Africa.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Funding/Support</h4>
<p> This study was funded by the Bill and Melinda Gates Foundation (grant INV-034355).</p>
<h4>Role of the Funder/Sponsor</h4>
<p> The funder had no role in the study design, data collection and analysis, decision to publish, or manuscript preparation.</p>


[V] Editorial Landscape of Journals in Kenya, Ethiopia, Nigeria, and Mozambique

<p>James Kiilu,¹ Cecilia Munguti,¹ James Kigera,¹ Michael Mwachiro¹</p>
<h4>Objective </h4>
<p>The peer review process faces the challenges of recruiting reviewers and assessing the quality of the peer review process and manuscripts.<sup>1</sup> This study evaluated trends in reviewer performance (R-scores), manuscript quality (M-scores), and their association with reviewer recommendations and editorial decisions at the <i>Annals of African Surgery</i> over 7 years.</p>
<h4>Design </h4>
<p>This retrospective study analyzed peer review reports at the <i>Annals of African Surgery</i> from 2017 to 2023, using data extracted from the journal’s manuscript management system. Variables included manuscript ID, submission year, revision cycles, reviewer ID, recommendations, R-scores, M-scores, time to review, and editorial decisions. R-scores assessed review timeliness and relevance, while M-scores evaluated the manuscript across dimensions such as interest, quality, and originality. Descriptive statistics summarized reviewer invitation responses, and cross-tabulations were performed to examine trends in R-scores and M-scores against submission year, reviewer recommendations, revision cycles, and editorial decisions. Associations between reviewer recommendations and editorial outcomes were assessed using χ² tests. Continuous data were tested for skewness, with analysis of variance applied to normally distributed data and Kruskal-Wallis tests to skewed data.</p>
<h4>Results </h4>
<p>For the period under review, the journal had a total of 1193 reviewers, with 370 (31%) coming from the international community. The number of reviewers invited per manuscript varied significantly across years, as determined by an independent-samples Kruskal-Wallis test (H<i>2</i> = 18.46; <i>df </i>= 6; <i>P =</i> .005). The mean (SD) number of reviewers invited per manuscript was 6.35 (3.4) in 2017, peaked at 9.81 (9.6) in 2019, then declined to 5.12 (3.5) in 2021 before rising again to 7.02 (4.03) in 2023. There was a significant improvement in the reviewer acceptance rate, increasing from 29.9% in 2019 to 45.7% in 2023 (<i>P</i> &lt; .001) (<b>Table 25-0922</b>). R-scores decreased significantly over the study period, from mean (SD) 2.68 (0.53) in 2017 to 2.29 (0.65) in 2023 (<i>P</i> &lt; .001), while there was no significant change in M-scores. There was no statistically significant change in R-scores across different manuscript revision cycles, but M-scores showed significant improvement (from mean [SD] 2.60 [0.84] in the initial round of review to 2.17 [0.61] after 3 revisions, <i>F</i> = 2.413; <i>P</i> &lt; .05), accompanied by a marked reduction in review turnaround time (from mean [SD] 11.07 [9.9] to 1.29 [1.38] days; <i>P</i> = .01). R-scores and M-scores varied significantly by editorial decision, with the highest R-scores observed for manuscripts receiving a major revision decision (mean = 2.71; <i>P</i> &lt; .001) and the highest M-scores seen in rejected manuscripts (mean = 3.91; <i>P</i> &lt; .001). One-way analysis of variance demonstrated statistically significant differences in mean R-scores awarded by the 15 editors (<i>F</i><i>14, 3986</i> = 17.77; <i>P</i> &lt; .001). Reviewer recommendations were associated with editorial decisions (Pearson χ² = 288.80; <i>df </i>= 9; N = 1173; <i>P</i> &lt; .001).</p>

![](https://assets.underline.io/markdown_image/1/image/54831eb6ac1a19e13b97949096aac091.png)

<h4>Conclusions</h4>
<p> While reviewer acceptance rates improved significantly over time, there was no corresponding improvement in R-scores or M-scores. Future efforts should prioritize not only increasing reviewer participation but also strengthening the quality and consistency of reviewer feedback to more effectively support manuscript development.</p>
<h4>Reference</h4>
<p>1. Peterson CJ, Orticio C, Nugent K. The challenge of recruiting peer reviewers from one medical journal’s perspective. <i>Proc Bayl Univ Med Cent</i>. 2022;35(3):394-396. doi:10.1080/08998280.2022.2035189</p>
<sup>1</sup>Annals of African Surgery, Nairobi, Kenya; james.kiilu@annalsofafricansurgery.com.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> James Kigera is a member of the Peer Review Congress Advisory Board but was not involved in the review or decision for this abstract.</p>


[V] Trends in Peer Review Metrics at the Annals of African Surgery

<p>Jorge Finke,<sup>1</sup> Sumi Sexton<sup>2</sup></p>
<h4>Objective </h4>
<p>The peer review process has been criticized for a lack of evidence that it improves the quality of scientific literature.<sup>1</sup> Some authors have reported peer reviews to be incoherent, lacking constructive feedback, and slow.<sup>2</sup> Importantly, little research has been done on peer review for narrative reviews. We have developed a checklist for peer reviewers to evaluate the effects on the quality of peer reviews and manuscripts.</p>
<h4>Methods</h4>
<p> This was a pseudorandomized nonblinded clinical trial. A 10-question checklist was developed for use by peer reviewers evaluating manuscripts for the medical journal <i>American Family Physician (AFP). </i>The checklist was created through the Delphi method, utilizing a panel of experts including <i>AFP</i> medical editors and editors from other journals who reviewed the proposed checklist. The checklist was internally validated by 2 medical editors independently comparing results of the new checklist to the journal’s existing manuscript quality scoring system. Manuscripts were assigned on alternating months to either the experimental group in which reviewers and editors applied the new checklist or the control group which utilized the traditional system. After peer review, the medical editor assigned to each manuscript rated the quality of each review, using a 0 to 100% scale to determine whether the checklist improved peer reviewer performance. Quality ratings of the initially submitted and finalized manuscripts were also determined by the medical editor using a 0 to100% scale. Changes in initial to finalized manuscript quality ratings were compared between the experimental and control groups.</p>
<h4>Results </h4>
<p>A total of 78 manuscripts were evaluated from December 2022 through March 2024. Forty of these manuscripts were assigned to peer reviewers utilizing the newly developed checklist, and 38 used the traditional review process. No statistically significant improvement was noted in review quality (68.5%; 95% CI, 63.5%-72.8% vs 65.7%; 95% CI, 61.0%-69.8%; <i>P</i> = .40), review word count length (683 words; 95% CI, 608.8-754.5 words vs 616.04 words; 95% CI, 545.8-684.1 words; <i>P</i> = .19) or speed of the peer review process (45.49 days; 95% CI, 37.2-52.8 days vs 37.3 days; 95% CI, 31.9-41.6 days; <i>P</i> = .08). Similarly, no statistically significant improvement was found in manuscript quality rating after revisions based on peer review with new checklist compared with the control group (difference, 15.7%; 95% CI, 9.1%-22.6% vs 12.6%; 95%CI, 6.7%-18.8%; <i>P</i> = .49).</p>
<h4>Conclusions</h4>
<p> Few interventions have had positive impacts in the peer-review process, including the appropriate use of checklists or guidelines.<sup>3</sup> While this study showed no significant improvements with the new checklist in the peer review quality rating, reviewer word counts, speed of reviewer process, and overall manuscript ratings, no parameters were negatively affected. The checklist was still formally adopted as a modified version of the previous format developed based on Delphi feedback from experts in the field.</p>
<h4>References</h4>
<p>1. Kelly J, Sadeghieh T, Adeli K. Peer review in scientific publications: benefits, critiques, &amp; a survival guide. <i>EJIFCC</i>. 2014;25(3):227-243.</p>
<p>2. Sciullo N, Duncan M. Professionalizing peer review suggestions for a more ethical and pedagogical review process. <i>J Schol Pub</i>. 2019;50:248-264. doi:10.3138/jsp.50.4.02</p>
<p>3. Gaudino M, Robinson NB, Di Franco A, et al. Effects of experimental interventions to improve the biomedical peer-review process: a systematic review and meta-analysis. <i>J Am Heart Assoc</i>. 2021;10(15):e019903. doi:10.1161/JAHA.120.019903</p>
<sup>1</sup>Contributing editor, <i>American Family Physician</i>, Leawood, KS, US, jfinke@bidmc.harvard.edu; <sup>2</sup>Editor in chief, <i>American Family Physician</i>, Leawood, KS, US.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None reported.</p>
<h4>Acknowledgment</h4>
<p> This study was conducted with support from American Family Physician as an internal quality improvement project and the intervention developed as adopted for use in the journal’s peer review process. This work was conducted with support from UL1TR002541 award through Harvard Catalyst (Biostatistics/Bioinformatics Consultation Program), The Harvard Clinical and Translational Science Center (National Center for Advancing Translational Sciences, National Institutes of Health), and financial contributions from Harvard University and its affiliated academic health care centers. The content is solely the responsibility of the authors and does not necessarily represent the official views of Harvard Catalyst, Harvard University and its affiliated academic health care centers, or the National Institutes of Health.</p>


[V] Impact of a Novel Checklist on the Peer Review Process

<p>Haley K. Holmer,<sup>1</sup> Edi E. Kuhn,<sup>1</sup> Camber Hansen-Karr,<sup>1</sup> Ed Reid,<sup>1</sup> Mark Helfand<sup>1</sup></p>
<h4>Objective</h4>
<p> To identify methodological concerns in Agency for Healthcare Research and Quality (AHRQ) Evidence-Based Practice Center (EPC) Program reports and assess the comprehensiveness of EPC Methods Guidance using the Systematic Reviewlution (SR) Framework.<sup>1</sup></p>
<h4>Design</h4>
<p> Cross-sectional analysis of reviewer comments from 25 EPC reports submitted between June 1, 2018, and March 1, 2023. The sample included systematic reviews (n = 16), systematic review updates (n = 4), and technical briefs (n = 5). For the internal, pre–peer-reviewed phase, 1 reviewer extracted comments received from associate editors, task order officers, and AHRQ leadership. In the external peer-reviewed phase, we extracted comments from peer reviewers, key informants, technical experts, partners, and public reviewers. A senior reviewer used the SR Framework,<sup>1</sup> a comprehensive typology of 67 systematic review problems, to classify concerns raised in the pre–peer-reviewed and/or peer-reviewed phase and to quantify frequencies. We identified patterns in 2 areas: (1) concerns raised in the pre–peer-review phase that persisted into the peer-reviewed phase and (2) concerns missed in the pre–peer-reviewed phase that were raised in the peer-reviewed phase. Further, we determined whether problems identified in the SR Framework are addressed in EPC Methods Guidance,<sup>2,3</sup> and we identified methodological concerns absent from the SR Framework.</p>
<h4>Results</h4>
<p> Of 5024 total reviewer comments, 15% (252 of 1717) of pre–peer-reviewed comments and 10% (348 of 3307) of peer-reviewed comments addressed methodological concerns. Nearly half of the methodological concerns identified during the pre–peer-review phase pertained to grading strength of evidence (SOE) (n = 116 [46%]), followed by concerns with assessing risk of bias (n = 23 [9%]), and spin (n = 21 [8%]). The most frequent concerns raised by peer reviewers also related to spin (n = 56 [16%]), grading SOE (n = 44 [13%]), and lack of clinical expert or stakeholder perspective (n = 41 [12%]). Spin-related concerns included conclusions that went beyond the evidence, policy implications disconnected from results, failure to incorporate SOE into conclusions, and biased tone in presentation. Comments raised in the peer-reviewed phase that were not raised in pre–peer-review phase included concerns about a priori protocol decisions or inflexible methods for answering review questions. The SR Framework failed to capture several key methodologic issues identified in EPC reports, including concerns about (1) SOE ratings, (2) inadequately defined outcomes, and (3) a priori protocol registration. Over 75% (n = 51) of the 67 problems in the SR Framework were addressed in EPC methods guidance.</p>
<h4>Conclusions</h4>
<p> Overall, EPC reports are of high methodological quality. Notable patterns in methodological concerns emerged, with specific issues pertaining to grading SOE, spin, need for clinical expert or stakeholder perspectives, and balancing adherence to an a priori protocol against potential bias introduced by an adaptive protocol. The persistence of spin-related concerns across both review phases suggests a challenge in maintaining objectivity in evidence synthesis. Notably, the SR Framework did not capture important methodological concerns in EPC reports, suggesting potential gaps in the SR Framework for evaluating the trustworthiness of evidence synthesis products.</p>
<h4><a id="_idTextAnchor075"></a>References</h4>
<p>1. Uttley L, Quintana DS, Montgomery P, et al. The problems with systematic reviews: a living systematic review. <i>J Clin Epidemiol</i>. 2023;156:30-41. doi:10.1016/j.jclinepi.2023.01.011</p>
<p>2. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Page last reviewed October 2022. Effective Health Care Program, Agency for Healthcare Research and Quality. <a href="https://effectivehealthcare.ahrq.gov/products/collections/cer-methods-guide"><span class="Hyperlink CharOverride-6">https://effectivehealthcare.ahrq.gov/products/collections/cer-methods-guide</span></a></p>
<p>3. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. <i>BMJ</i>. 2021;372:n71. doi:10.1136/bmj.n71</p>
<sup>1</sup>Portland VA Research Foundation, Portland, OR, haley.holmer@va.gov.</p>
<h4>Conflict of Interest Disclosures</h4>
<p> None of the authors have any affiliations or financial involvement that conflicts with the material presented in this abstract.</p>
<h4>Funding/Support</h4>
<p> <a id="_idTextAnchor076"></a>This work is funded by the Agency for Healthcare Research and Quality (AHRQ) Effective Healthcare (EHC) Program through a contract to the Scientific Resource Center (Contract No. HHSA 75Q80122C00002). The authors of this abstract are responsible for its content. Statements in the abstract do not necessarily represent the official views of or imply endorsement by the AHRQ, US Department of Health and Human Services.</p>


[V] Identifying Methodological Concerns in Agency for Healthcare Research and Quality Evidence-based Practice Center Reports: Analysis of Editorial Review Comments

<p>Anjali Garg,<sup>1</sup> Preeti Panda,<sup>2</sup> Lydia Furman,<sup>3</sup> Kimberly Montez,<sup>4</sup> Alex R. Kemper,<sup>5</sup> Lewis First<sup>6</sup></p>
<h4>Objective</h4>
<p> To implement and evaluate a pilot mentorship program in which experienced peer reviewers mentor novice reviewers in writing peer reviews.</p>
<h4>Design</h4>
<p> In 2023, editorial board members of <i>Pediatrics</i>, the peer-reviewed journal of the American Academy of Pediatrics, volunteered for a pilot mentorship program to improve the peer review skills of early career physicians (medical students to junior faculty) to enhance the number of early career reviews within the pool present at <i>Pediatrics</i>. Mentors selected 1 or 2 mentees. The target participation for the pilot was 10 dyads. Literature review and expert consensus informed development of peer review mentorship materials. Curricula for mentees included didactics and checklists, and material for mentors provided feedback guidance. For objective assessment, the review quality instrument (RQI) was adapted into an 8-item, 3-point Likert scale questionnaire to score mentee reviews.<sup>1</sup> Mentees needed to complete at least 3 reviews with a score of 3 on at least 6 of 8 RQI items to graduate the program—a metric selected by the program leaders. Mentors and mentees completed a review of a regular article (as defined by the <i>Pediatrics </i>author guidelines) separately, then met together for structured feedback from the mentor to the mentee on the review. Each regular article was sent to the mentor based on their area of expertise; the turnaround time for completion was 3 weeks. The RQI was completed by the mentor, and the combined review was ultimately submitted to the editorial board by the mentor. Qualitative data regarding skills gained and areas for improvement were compiled and analyzed from intake and exit surveys completed by mentors and mentees.</p>
<h4>Results</h4>
<p> Eleven dyads enrolled, with 9 graduating from the program, an 81% graduation rate. Five dyads (55%) required 3 reviews to graduate, 3 (33%) required 4 reviews, and 1 (9%) required 5 reviews. The median number of RQI items scoring a 3 in each dyad’s first reviews increased from 4 (IQR, 2-7) to 7 (IQR, 6-8) by the last review. The range of total RQI scores increased from 16-24 (first reviews) to 23-24 (last reviews) for each dyad. Qualitative analysis indicated that the pilot program was feasible and effective. Notably, mentees reported increased comfort with peer review and improvement in their own manuscript writing skills. Mentors noted an improvement in their own reviewing skills through mentorship. Areas for improvement included targeted communication from the leadership team and allowing separate enrollment of mentees and mentors with matching by the program team to further enhance dissemination of this program.</p>
<h4>Conclusions</h4>
<p> The pilot program was successful in enrolling the target number of participants, graduating over 80% of the mentees with improvement of reviewing skills per increase in RQI scores, and demonstrated feasibility for future program building.</p>
<h4>Reference</h4>
<p>1. van Rooyen S, Black N, Godlee F. Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. <i>J Clin Epidemiol</i>. 1999;52(7):625-629. doi:10.1016/s0895-4356(99)00047-5</p>
<sup>1</sup>Division of Critical Care Medicine, Department of Pediatrics, Ann and Robert H. Lurie Children’s Hospital of Chicago, Chicago, IL, US, anjaliga07@gmail.com; <sup>2</sup>Division of Pediatrics, Department of Emergency Medicine, Stanford University, Stanford, CA, US; <sup>3</sup>University Hospitals Rainbow Babies &amp; Children’s, Cleveland, OH, US; <sup>4</sup>Department of Pediatrics, Wake Forest University School of Medicine, Winston-Salem, NC, US; <sup>5</sup>Division of Primary Care Pediatrics, Nationwide Children’s Hospital, Columbus, OH, US; <sup>6</sup>Department of Pediatrics, University of Vermont Children’s Hospital, Burlington, VT, US. </p>
<h4>Conflict of Interest Disclosures</h4>
<p> All authors are members of the Pediatrics editorial board.</p>
<h4>Acknowledgment</h4>
<p> We acknowledge Kate Larson and the Pediatrics editorial board administrative team for their help with the implementation of this program. </p>
<h4>Additional Information</h4>
<p> Preeti Panda is a co–corresponding author (preetipanda@stanford.edu).</p>


Next from The 10th Peer Review Congress

[V] Trends in Citation Impacts of Original Research in Major Cardiovascular Journals, 2008-2018

Similar lecture

Identifying Potential Duplicate Publications in the Scientific Literature Using Crossref