Quality of Cohort Studies Reporting Post the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement

Article information

Epidemiol Health. 2011;33.e2011005

Publication date (electronic) : 2011 June 07

doi : https://doi.org/10.4178/epih/e2011005

Jalal Poorolajal ¹, Zahra Cheraghi ², Amin Doosti Irani ², Shahab Rezaeian ²

¹Research Center for Health Sciences and Department of Epidemiology & Biostatistics, School of Public Health, Hamadan University of Medical Sciences, Hamadan, Iran.

²Department of Epidemiology & Biostatistics, School of Public Health, Hamadan University of Medical Sciences, Hamadan, Iran.

Correspondence: Jalal Poorolajal, MD, PhD. Department of Epidemiology & Biostatistics, School of Public Health, Hamadan University of Medical Sciences, Shahid Fahmideh Ave., Hamadan 65157838695, Iran. Tel: +98-811-8260661, Fax: +98-811-8255301, poorolajal@umsha.ac.ir

Received 2011 March 15; Accepted 2011 May 11.

Abstract

The quality of reporting of cohort studies published in the most prestigious scientific medical journals was investigated to indicate to what extent the items in the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) checklist are addressed. Six top scientific medical journals with high impact factor were selected including New England Journal of Medicine, Journal of the American Medical Association, Lancet, British Medical Journal, Archive of Internal Medicine, and Canadian Medical Association Journal. Ten cohort studies published in 2010 were selected randomly from each journal. The percentage of items in the STROBE checklist that were addressed in each study was investigated. The total percentage of items addressed by these studies was 69.3 (95% confidence interval: 59.6 to 79.0). We concluded that reporting of cohort studies published in the most prestigious scientific medical journals is not clear enough yet. The reporting of other types of observational studies such as case-control and cross-sectional studies particularly those being published in less prestigious journals expected to be much more imprecise.

Keywords: STROBE statement; Cohort studies; Observational studies; Reporting

INTRODUCTION

Randomized trials are generally considered the gold standard for evaluating both the efficacy and the potential side effects of new therapeutic or preventive interventions in both clinical medicine and public health [1]. However, much of the biomedical research into the cause of diseases comes from observational studies [2]. The results of these studies should be reported as transparently as possible "so that readers can follow what was planned, what was done, what was found, and what conclusions were drawn" [3]. Unfortunately, reporting of observational research is neither precise nor clear enough to assess the strengths and weaknesses of the evidence [4, 5].

To improve the reporting of observational research, a group of experts developed a checklist of items known as the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement [6]. The STROBE statement was developed in 2007 to assist authors in report writing of observational studies, including cohort, case-control, and cross-sectional studies, to support editors and reviewers in considering such manuscript for publication, and to help readers in appraising published articles critically [3].

Concerning valuable recommendations made in this statement, expected that reporting of observational studies published after 2007 being improved enough to assess the strengths and weaknesses of the evidence. The present study was set up to investigate the quality of reporting of cohort studies published in the most prestigious scientific medical journals and to indicate to what extent the items in STROBE checklist are noticed by both authors and publishers.

METHODS

We conducted a cross-sectional study and selected six top scientific medical journals with high impact factor (IF) among the most prestigious and important medical journals indexed in international databases. These included New England Journal of Medicine (N Engl J Med) (IF 50.017), Journal of the American Medical Association (JAMA) (IF 31.718), Lancet (IF 28.409), British Medical Journal (BMJ) (IF 12.827), Archives of Internal Medicine (Arch Intern Med) (IF 9.110), and Canadian Medical Association Journal (Can Med Assoc J) (IF 7.464). We were to include Nature and Annals of Internal Medicine in this survey, but they were not open access journals.

Among the observational studies, cohort studies are much more expensive and take longer follow-up time than case-control and cross-sectional studies. Therefore, the results obtained from cohort studies are of substantially superior quality to other observational studies. In fact, cohort studies are considered the gold standard in observational epidemiology. Accordingly, we selected cohort studies, either prospective or retrospective, for this evaluation.

We randomly selected 10 cohort studies published in each of the six medical journals from January to August 2010. To do so, we sorted the articles from newly published articles to the old ones. Then, looked for the cohort studies to find ten eligible articles. If there were not enough cohort studies, we searched second half of 2009 to obtain additional studies. Accordingly, we enrolled 60 cohort studies from six prestigious scientific medical journals. The studies were randomly assigned to four reviewers through drawing lots. The reviewers independently made the decisions on the number of items, from the STROBE checklist, which were addressed in the selected studies. The reviewers were not blinded to the names of the studies' authors and journals.

In order to check the reliability of the four reviewers' judgment on the quality of reporting of cohort studies, we conducted two consecutive pilot studies as follows. First, an article was randomly selected from JAMA. All four reviewers assessed the strengths and weaknesses of the evidence reported in the same article using the predetermined checklist of items. The disagreements were discussed to reach the same perception of the checklist in order to increase the between reviewers reliability. Then, the four reviewers evaluated the quality of reporting of another article that was randomly selected. There was no significant differences between the reviewers' judgment in the second pilot study (p=0.823).

In order to assess the validity of the reviewers' judgments, two reviewers made decision on the quality of reporting of each cohort study independently. Any disagreements were resolved by adjudication with a third author.

The STROBE statement included a checklist of 22 items. In order to estimate the quality of reporting of cohort studies more accurately, we divided the items into 47 sub-items (Table 1). We considered three choices for each sub-item (reported/not reported/not applicable). The percentage of each sub-item addressed in the selected articles was estimated. The total percentage for all sub-items was reported.

Table 1

Percentage of items in STROBE checklist which were addressed in reports of cohort studies published in six top scientific medical journals in 2010 (n (%))

All statistical analysis was performed at 95% significant level using statistical software Stata version 10.0 (StataCorp, College Station, TX, USA).

RESULTS

In this survey, 60 cohort studies were selected for evaluation including 10 studies from the six prestigious scientific medical journals including: N Engl J Med, JAMA, Lancet, BMJ, Arch Intern Med, and Can Med Assoc J. From these, 56 studies were published in 2010 and four studies in 2009. The percentage of items and sub-items addressed by these studies are summarized in Table 1. The sub-items were not applicable in 7.1% (95% confidence interval [CI], 1.3% to 13.0%), were not reported in 23.6% (95% CI, 15.3% to 31.5%), and were reported in 69.3% (95% CI, 59.6% to 79.0%) of the cohort studies.

Of 47 sub-items investigated in this survey, nine sub-items were reported 100%, 22 sub-items were addressed in more than 90% of the studies, 28 sub-items were included in more than 75% of the studies, and 32 sub-items were addressed in more than 50% of the studies.

DISCUSSION

STROBE checklist of items provides valuable recommendations for both authors to report the results of observational studies clearly as well as for editors, reviewers, and readers to appraise such reports critically [3]. On overall, almost 69.3% of the items and sub-items in STROBE checklist were addressed by cohort studies published in six top scientific medical journals three year after dissemination of STROBE statement.

However, what has happened to the reports of other types of observational studies? The results of the present study represent the reporting of cohort studies published in six prestigious scientific medical journals that generally accept the well-done and well-written studies. However, there are numerous observational studies, the results of which are published in other less fastidious peer-reviewed medical journals. Thus, it is expected that the quality of reporting of such studies is much poorer than what reported in the present study, although the result of present study is not desirable enough. Furthermore, cohort studies are much more expensive and take longer follow-up time than other types of observational studies such as case-control and cross-sectional design. Hence, the reporting of cohort studies is generally expected to be of substantially superior quality to other observational studies. Accordingly, if this survey had been planned to assess reporting of case-control or cross-sectional studies, the estimated result would be much more undesirable.

We found no similar studies but one. Poorolajal et al. [7] conducted a similar study in 2007 to assess the reporting of cohort studies before STROBE statement being issued. However, the design, the results, and the number of sub-items, which were evaluated in that study, was not comparable with that of the present study.

The present study had a number of limitations. First, the limited number of studies evaluated in the present study may increase the possibility of random error. Second, randomly selection of cohort studies from a few prestigious medical journals may increase the possibility of selection bias. Third, the value of all sub-items was not really the same. Hence, adding up all percentages to estimate a summary measure might not be reasonable, although was done to help the readers make an overall judgment.

This result of the present survey represents the quality of the reporting of cohort studies in top scientific medical journals. Hence, we can generalize the results of this survey neither to other types of observational studies nor to the publication of less fastidious peer-reviewed medical journals.

We concluded that reporting of cohort studies published in the most prestigious scientific medical journals is not clear and desirable enough yet. The reporting of other types of observational studies such as case-control and cross-sectional studies particularly those being published in less fastidious peer-reviewed journals is expected to be much more imprecise. This issue should be the focus of the both authors' and editors' special attention when reporting and/or reviewing the reports of observational studies.

Notes

The authors have no conflicts of interest to declare for this study.

This article is available from: http://e-epih.org/.

References

1. Gordis L. Epidemiology 2008. 4th edth ed. Philadelphia: Saunders.

2. Glasziou P, Vandenbroucke JP, Chalmers I. Assessing the quality of research. BMJ 2004;328:39–41. 14703546.

3. von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. PLoS Med 2007;4:e296. 17941714.

4. Pocock SJ, Collier TJ, Dandreo KJ, de Stavola BL, Goldman MB, Kalish LA, et al. Issues in the reporting of epidemiological studies: a survey of recent practice. BMJ 2004;329:883. 15469946.

5. Tooth L, Ware R, Bain C, Purdie DM, Dobson A. Quality of reporting of observational longitudinal research. Am J Epidemiol 2005;161:280–288. 15671260.

6. Vandenbroucke JP, von Elm E, Altman DG, Gøtzsche PC, Mulrow CD, Pocock SJ, et al. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. PLoS Med 2007;4:e297. 17941715.

7. Poorolajal J, Tajik P, Yazdizadeh B, Sehat M, Salehi AR, Rezaei M, et al. Quality assessment of the reporting of cohort studies before STROBE statement. Iranian J Epidemiol 2009;5:17–26. (Persian).

Article information Continued

(open-access, http://creativecommons.org/licenses/by/3.0/) :

This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Table 1

Percentage of items in STROBE checklist which were addressed in reports of cohort studies published in six top scientific medical journals in 2010 (n (%))

Item	Recommendation	Not applicable	Not reported	Reported
1a	Indicate the study’s design in the title/abstract	0 (0.0)	15 (25.0)	45 (75.0)
1b	Provide in the abstract an informative and balanced summary	0 (0.0)	3 (5.0)	57 (95.0)
Introduction
2	Explain the scientific background and rationale for the investigation	0 (0.0)	0 (0.0)	60 (100.0)
3	State specific objectives, including any pre-specified hypotheses	0 (0.0)	1 (1.7)	59 (98.3)
Methods
4	Present key elements of study design early in the paper	0 (0.0)	21 (35.0)	39 (65.0)
5	Describe locations	0 (0.0)	2 (3.3)	58 (96.7)
5	Describe recruitment dates	0 (0.0)	2 (3.3)	58 (96.7)
5	Describe periods of follow-up	0 (0.0)	4 (6.7)	56 (93.3)
6a	Give the eligibility criteria	0 (0.0)	4 (6.7)	56 (93.3)
6a	Describe the methods of follow-up	0 (0.0)	2 (3.3)	58 (96.7)
6b	Give matching criteria	54 (90.0)	0 (0.0)	6 (10.0)
6b	Give number of exposed and unexposed in matched studies	52 (86.7)	5 (8.3)	3 (5.0)
7	Clearly define all outcomes	0 (0.0)	0 (0.0)	60 (100.0)
7	Clearly define all exposures	0 (0.0)	6 (10.0)	54 (90.0)
7	Clearly define all predictors	0 (0.0)	2 (3.3)	58 (96.7)
7	Clearly define all potential confounders	0 (0.0)	9 (15.0)	51 (85.0)
7	Clearly define all effect modifiers	0 (0.0)	41 (68.3)	19 (31.7)
8	Give sources of data	0 (0.0)	0 (0.0)	60 (100.0)
8	Method of measurement	0 (0.0)	1 (1.7)	59 (98.3)
9	Describe any efforts to address potential sources of bias	0 (0.0)	51 (85.0)	9 (15.0)
10	Explain how the study size was arrived at	33 (55.0)	15 (25.0)	12 (20.0)
11	Describe which groupings were chosen (if applicable)	10 (16.7)	8 (13.3)	42 (70.0)
12a	Describe all statistical methods	0 (0.0)	0 (0.0)	60 (100.0)
12a	Describe all statistical software	0 (0.0)	14 (23.3)	46 (76.7)
12b	Describe any methods used to examine subgroups and interaction	0 (0.0)	42 (70.0)	18 (30.0)
12c	Explain how missing data were addressed	0 (0.0)	33 (55.0)	27 (45.0)
12d	explain how loss to follow-up was addressed	3 (5.0)	42 (70.0)	15 (25.0)
12e	Describe any sensitivity analyses	3 (5.0)	44 (73.3)	0 (21.7)
Results
13a	Report numbers of individuals at each stage of study	5 (8.3)	29 (48.4)	26 (43.3)
13b	Give reasons for non-participation at each stage	5 (8.3)	42 (70.0)	13 (21.7)
13c	Consider use of a flow diagram	0 (0.0)	51 (85.0)	9 (15.0)
14a	Give characteristics of study participants	0 (0.0)	5 (8.3)	55 (81.7)
14b	Indicate the number of participants with missing data	4 (6.7)	37 (61.6)	19 (31.7)
14c	Summarize follow-up time	2 (3.3)	32 (53.4)	26 (43.3)
15	Report numbers of outcome events or summary measures	0 (0.0)	0 (0.0)	60 (100.0)
16a	Give unadjusted estimates	0 (0.0)	23 (38.3)	37 (61.7)
16a	Give confounder-adjusted estimates	1 (1.7)	6 (10.0)	53 (88.3)
16a	Give estimates precision/confidence interval	0 (0.0)	2 (3.3)	58 (96.7)
16b	Report category boundaries when continuous variables were categorized	8 (13.3)	7 (11.7)	45 (75.0)
16c	Consider translating estimates of relative risk into absolute risk (If relevant)	21 (35.0)	32 (53.3)	7 (11.7)
17	Report other analyses done e.g., subgroups analysis and sensitivity analyses	0 (0.0)	19 (31.7)	41 (68.3)
Discussion
18	Summarize key results with reference to study objectives	0 (0.0)	0 (0.0)	60 (100.0)
19	Discuss limitations of the study	0 (0.0)	0 (0.0)	60 (100.0)
20	Give a cautious interpretation of results considering objectives	0 (0.0)	0 (0.0)	60 (100.0)
20	Explain results from similar studies	0 (0.0)	0 (0.0)	60 (100.0)
21	Discuss the generalizability of the study results	0 (0.0)	6 (10.0)	54 (90.0)
Other information
22	Give the source of funding and the role of the funders	0 (0.0)	1 (1.7)	59 (98.3)
1-22	Total	(7.1)	(23.6)	(69.3)

STOBE, The Strengthening the Reporting of Observational Studies in Epidemiology.