Prevalence and methodological quality of systematic reviews in Korean medical journals
Article information
Abstract
This study aimed to assess and evaluate the prevalence and methodological quality of systematic reviews (SRs) published in major Korean medical journals (KMJs). The top 15 journals with the highest Korean Medical Citation Index, published between 2018 to 2021, were selected. We assessed the methodological quality of SRs using A Measurement Tool to Assess Systematic Reviews 2 (AMSTAR 2). In total, 126 SRs were included, with an average of 32 SRs being reported annually. The overall prevalence of SRs in KMJs was 2.8%, with an increase from 2.6% in 2018 to 3.4% in 2021. Overall, the methodological quality of SRs was low (9.5% low, 90.5% critically low). More than 80% of the studies adhered to critical domain items such as a comprehensive literature search and risk of bias assessment, but for items such as protocol registration and listing excluded studies and the justification for exclusion, the adherence rate was less than 15%. While the number of SRs in KMJs steadily increased, the overall confidence in the methodological quality was low to critically low. Therefore, in order to provide the best evidence for decision-making in clinical and public health areas, editors, reviewers, and authors need to pay more attention to improving the quality of SRs.
INTRODUCTION
Systematic reviews (SRs) are one of the most valuable study designs, providing the highest level of evidence for various research questions, including those for public health, and clinical research and practice. A previous study conducted in 2014 reported that more than 8,000 SRs were indexed in MEDLINE annually, with a three-fold increase over the last decade [1].
The number of SRs published worldwide has continued to rise, with Korea being one of the top countries publishing core clinical journals [2]. In addition, many Korean medical journals (KMJs) have gone global in recent years [3]. However, the prevalence and trends of SRs published in major KMJs have not been thoroughly reported.
The validity and reliability of the evidence from SRs depend on the methodology used, which is also essential for a valid interpretation and application of the findings [4]. Even though previous studies have assessed the methodological quality of SRs in specific medical fields [4-6] or countries [7,8], limited information is available on SRs in KMJs. This study assessed the prevalence of SRs in major KMJs published in recent years and evaluated the methodological quality of those SRs.
MATERIALS AND METHODS
Literature search strategy
We selected major KMJs from KoreaMed (https://koreamed.org), which is a service of the Korean Association of Medical Journal Editors (KAMJE). The Korean Medical Citation Index (KoMCI) is a citation database maintained by the Korean Academy of Medical Science. It indexes KoreaMed journals and publishes annual citation reports, which analyze citations of articles published in Korean journals. The KoMCI 2018 list contained 254 journals. The top 15 journals published from 2018 to 2021 with the highest KoMCIs were selected. We searched the literature in KoreaMed on October 18, 2021. Supplementary Material 1 presents a list of the selected journals and the search strategy.
Inclusion criteria
We included SRs regardless of whether they included a metaanalysis. A study was considered to be an SR if the authors explicitly reported that they conducted it using terms such as “systematic” or “systematically.” We also considered a study to be an SR if the authors described some of the standard steps of SRs, such as the study search and study selection process, which tend to be reproducible.
We classified the remaining studies into other types of reviews (narrative and non-systematic), primary studies (randomized controlled trials, case-control studies, cross-sectional studies, etc.), and others (correspondence, editorials, commentaries, etc.). We regarded a review as a non-systematic if it did not report the method used to perform the review, such as the study search and selection.
Study selection
Teams of two reviewers screened information from the title and abstract from each reference independently to create duplicate records, which were then calibrated, and disagreements, if any, were resolved by discussion or consultation with a third reviewer whenever necessary. For references identified as potentially eligible, we obtained the full text. Pairs of reviewers independently conducted full-text screening and resolved any discrepancies by discussion. At each stage, we conducted calibration exercises to increase reviewers’ understanding and improve consistency among them.
Data extraction
We extracted data using a pre-piloted data extraction form after the calibration exercises. Teams of two reviewers extracted the following data independently and in duplicate.
(1) Study characteristics: first author, number of authors, number of affiliations, international collaborative authorship, publication year, language of publication, type of question, study design of primary studies, number of primary studies, number of participants, type of intervention/exposure, type of outcome, and meta-analysis.
(2) Methodological quality assessment: We used A Measurement Tool to Assess Systematic Reviews 2 (AMSTAR 2) [9] to evaluate the methodological quality of the included SRs. The AMSTAR 2 checklist consists of a total of 16 items, including 7 critical and 9 non-critical items. SRs can be categorized based on critical weakness as high (none or 1 non-critical weakness), moderate (more than 1 non-critical weakness), low (1 critical flaw with or without non-critical weaknesses), and critically low (more than 1 critical flaw with or without non-critical weaknesses) depending on the rating.
Statistical analysis
The SRs selected from among the studies published were categorized by the type of journal articles and the year of publication. The study characteristics were presented in terms of frequencies and percentages. The proportion of reviews was measured against each item of the AMSTAR-2, classifying the methodological quality into 4 categories. All statistical analyses were performed using SAS version 9.4 (SAS Institute Inc., Cary, NC, USA).
Ethics statement
This study was based on published studies and did not require ethical approval.
RESULTS
Characteristics of the included systematic reviews
In total, 4,526 articles were identified from the top 15 KMJs published between 2018 to 2021. Among them, 3,288 (72.6%) were primary studies, and 494 (10.9%) were other types of reviews. The remaining 126 (2.8%) articles were SRs and were included in this study (Table 1, Supplementary Material 2). Supplementary Material 3 presents the detailed characteristics of the SRs included.
Prevalence of systematic reviews in Korean medical journals
An average of 32 (29-37) SRs were reported annually, with the prevalence of SRs increasing from 2.6% in 2018 to 3.4% in 2021 (Table 1). International collaborative studies were rare, with only 7 (5.6%) articles published collaboratively between Korean researchers and those from other countries. We identified 76 (60.3%) SRs focusing on intervention-related questions, 16 (12.7%) investigating questions related to prognosis, and 15 (11.9%) that investigated questions that pertained to diagnosis. Furthermore, 96 (76.2%) of the SRs involved a meta-analysis (Table 2).
Characteristics of the included studies (n=126)
Methodological quality of systematic reviews
There were 9 items to which more than 70% of SRs complied (1, 4, 5, 6, 8, 9, 11, 15, 16), but 5 items with an adherence rate lower than 50% (2, 3, 7, 10, 13). Furthermore, fewer than 15% of articles adhered to items 2 and 7, which fall under critical domains (Table 3). Therefore, the overall confidence in the methodological quality of the SRs was low (9.5%) or critically low (90.5%) (Table 4).
Quality assessment according to the AMSTAR 2 tool (n=126)
Summary quality assessment according to the AMSTAR 2 tool (n=126)
DISCUSSION
This study investigated the prevalence and methodological quality of SRs recently published in major KMJs. A total of 126 SRs were investigated, and the prevalence of SRs increased from 2.6% in 2018 to 3.4% in 2021. However, unlike international trends [2], only 7 SRs were conducted through international collaboration. A well-conducted international collaboration tends to improve the completeness and quality of studies [10-12]. However, studies conducted through international collaboration might be submitted to international journals, instead of journals based in Korea.
The methodological quality of SRs, as determined using AMSTAR 2, only showed low to critically low confidence. Significant contributors to low confidence were less than 50% adherence to domains, such as protocol registration (item 2), explanation of study designs selection (item 3), provision of a list of excluded studies (item 7), reporting of funding sources of primary studies (item 10), risk of bias interpretation (item 13), and failure to adhere to critical domains. Most previous studies using AMSTAR showed methodologically moderate quality overall [7,13-19], whereas studies using AMSTAR 2 showed confidence levels ranging from low to critically low [5,6,8,20-23]. Unlike AMSTAR, AMSTAR 2 does not classify quality based on a total score, but divides the items into 7 critical and 9 non-critical domains according to importance, and then classifies the quality according to each item rating [9]. Therefore, if the critical domains are not adhered to, the quality of SRs is classified as low or critically low confidence. In this study, the proportion of adherence to items 2 and 7, which are critical domains, did not even reach 15%; this may have resulted in the low confidence.
Similarly low adherence to items 2 and 7 was reported in previous studies [6,21-23]. Regarding item 2, protocol registration, only 14.3% of SRs in major KMJs registered or published their study protocol prior to conducting the SRs. Previous studies assessing the methodological quality of SRs in clinical areas have reported adherence to this item as 0% [6,22]. A written protocol with independent verification is recommended to reduce the risk of bias that may creep in during the process of conducting research [9].
In addition, adherence to item 7 (i.e., providing a list of excluded studies and justifying their exclusion), was rated as 13.5%. Similarly low adherence (0.0-6.3%) has been reported in previous studies [22,23]. Most SRs in this study described inclusion and exclusion criteria explicitly, but a list of excluded studies was not provided. Providing a complete list of potentially relevant studies would guarantee unbiased study selection and reproducibility of the study results [9,22,24]. However, the authors might not provide a complete list due to the relative importance of the list of included studies or words, and the space limitations of the journal. Therefore, all the authors’ reports, including appendices and supplementary materials, have been reviewed wherever necessary.
Item 10 (reporting funding sources of primary studies) was reported in fewer than 5% of SRs. This finding is similar to the proportions of 0.0-30.3% reported in previous studies [6,21-23,25]. Industry-funded studies sometimes show favor toward sponsors, due to which they are less likely to be published than independently funded studies [9,25,26]. Since this information may not be clearly visible in the design or methodology of a study, the funding source for each study should be well described, separately.
Although more than 80% of SRs included assessed the risk of bias (RoB; item 9), item 13 (i.e., review authors accounting for RoB in the individual studies when interpreting/discussing the results of the review) was reported completely, in fewer than 50% of studies. However, other studies published since 2020, assessing methodology and reporting quality of SRs or meta-analyses in various fields of medical treatment using AMSTAR 2, have reported satisfactory quality between 60% and 75% [6,20,22,23]. Accurately evaluating and interpreting the RoB of primary studies is important because combining low-quality primary studies increases bias, resulting in pooled estimates that are misleading [9]. Therefore, KMJs’ editors, reviewers, and researchers should pay more attention to these items in the future to increase the validity of SRs.
Implications
Overall, the methodological quality of SRs published in major KMJs was low to critically low. Even if we were to expand the scope of our study to include other journals in Korea, it would not improve the methodological quality of SRs. Journal editors and reviewers need to be aware of these findings and use them to educate reviewers and guide authors to adhere to the standards of SRs. They also need to improve the methodological quality and ensure that SRs are conducted specifically by reviewers with relevant experience in the SR process.
Authors should be aware of methodological quality evaluation standards such as AMSTAR 2 when conducting or reporting SRs, as this would facilitate a transparent and systematic delivery of their study results. When using the results of published SRs, readers or institutions should keep in mind that the credibility of the results may vary, be alert to limitations when interpreting results or applying them in the field, and should review the quality of the SRs.
Strengths and limitations
To the best of our knowledge, this study was the first to apply AMSTAR 2, the most widely used international evaluation standard, to evaluate the methodological quality of SRs in KMJs. In addition, the present study included the top 15 representative KMJs covering general and specialized medical areas. We complied with standard methods in this field, such as independent and duplicate reference screening, selection, and data extraction. We tried to ensure reliability and consistency by performing calibration exercises in all processes.
As limitations, due to the low to very low confidence of SRs in this study, we could not perform a comparative analysis to identify the study characteristics related to the methodological quality. We used AMSTAR 2 for a methodological quality assessment; however, this tool is primarily designed to evaluate SRs addressing intervention-related questions. Therefore, this study may not reflect the special requirements of reviews with other types of questions, such as prognosis or diagnostic performance.
The overall methodological quality of SRs in major KMJs had low to critically low confidence. Given the increasing prevalence and importance of SRs in providing the best evidence for decisionmaking in clinical and public health areas, stakeholders need to use our findings and pay more attention to improving the quality of SRs. In addition, future research on the current practices of the research community in Korea, including awareness of editors, reviewers, and authors, and authors’ guidance provided by journals should focus on providing additional evidence for improving the quality of SRs.
SUPPLEMENTARY MATERIALS
Supplementary materials are available at https://www.e-epih.org/.
Supplementary Material 1.
The list of selected journals and search strategy
Supplementary Material 2
Study selection process
Supplementary Material 3
General characteristics of systematic reviews included
Notes
CONFLICT OF INTEREST
The authors have no conflicts of interest to declare for this study.
FUNDING
This research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF), funded by the Ministry of Education (2021R1I1A3041301) and the Ministry of Science and ICT (NRF-2022R1A5A2030454).
AUTHOR CONTRIBUTIONS
Conceptualization: Kim SJ, Han MA, Jung JH, Hwang EC. Data curation: Kim SJ, Han MA. Formal analysis: Han MA. Funding acquisition: Han MA. Methodology: Kim SJ, Han MA, Jung JH, Hwang EC. Project administration: Kim SJ, Han MA, Jung JH, Hwang EC, Kim HR, Yoon SE, Kim SH, Kim P, Kim SY. Visualization: Kim SJ, Han MA. Writing – original draft: Kim SJ, Han MA. Writing – review & editing: Kim SJ, Han MA, Jung JH, Hwang EC, Kim HR, Yoon SE, Kim SH, Kim P, Kim SY.
Acknowledgements
We registered the study protocol in PROSPERO (CRD42021283214).