Take a look at the Recent articles

Abstract

The increased availability of databases for orthopaedic research purposes allows researchers to easily conduct studies and obtain large sample sizes. However, the scientific utility of these databases is in question. The purpose of this study is to determine how these databases impact the quality of podium abstracts, as evaluated by level of evidence, at the American Academy of Orthopaedic Surgeons Annual Meeting. Paper abstracts (n=8,866) from 2008 to 2018 were obtained from the Meeting Committee. Five reviewers independently determined study design, data source, and focus of each abstract. The level of evidence was assigned using the Oxford Centre for Evidence-Based Medicine scheme. The number of abstracts accepted at each Annual Meeting increased substantially (582 in 2008; 918 in 2018). There was a significant increase in the overall rate of studies utilizing large databases (5.3% in 2008; 17.1% in 2018) and a decrease in the rate of studies with single-center data (80.2% in 2008; 69.4% in 2018). There was an overall increase in the rate of Level II (18.2% in 2008; 25.5% in 2018) and Level III (19.6% in 2008; 26.9% in 2018) studies with a decrease in Level IV (39.2% in 2008; 28.4% in 2018) studies. In studies specifically utilizing large databases, the percentage of Level II evidence increased (16.1% in 2008; 29.3% in 2018). Overall, the use of these databases has contributed to the improved quality of higher level (Level II and III) research presented at the Annual Meetings.

Key words

AAOS, annual meeting, level of evidence, database, abstract

Introduction

Clinicians and researchers have experienced a significant change in research being published and presented over the last decade. Large-scale data has become more accessible over the years due to the rapidly expanding volume of computerized databases, with studies showing that orthopaedic researchers have increasingly utilized these databases [1-5]. Meta-analyses have also become popular as a statistical method for combining the data from multiple studies in order to hopefully reduce outcome bias by combining large groups of patients from multiple centers. Researchers have also witnessed the evolution of medicine where quality, outcomes, and cost-effectiveness of care is increasingly demanded, which may alter the focus of research studies in other ways [6-8]. Some of these changes have been controversial as journals and professional organizations seek to navigate these new opportunities and present high-quality research.

Although designed primarily for tracking, insurance, coding, or billing purposes, state and national administrative databases of medical records have been made available for research purposes [1,3,5]. These databases provide substantial health care data on large populations of patients and present a relatively efficient method for researchers to obtain a large sample size and conduct studies on a larger scale. Subsequently, they allow for easy evaluation of hospital or patient outcomes and provide the benefit of increased statistical strength of conclusions [9]. However, it is important to consider the limitations both within and between databases that may bias results, including lack of standardization, varied patient data entry and collection methods, data inaccuracy, and validity [3,10-12]. Consideration must be given to the utility of these databases and what role they should play in the body of orthopaedic research being presented to the orthopaedic community. Since studies that employ administrative databases remain controversial, it is worthwhile to analyze the extent to which these databases are emerging within the orthopaedic literature.

With the expansion of types of research including administrative database analysis and meta-analysis, as well as the evolution of healthcare which demands different types of studies such as practice management, a concern can be raised about how this will impact the quality of research. Level of evidence (LOE) designations offer a well-accepted and systematic method for clinicians and patients to evaluate research studies for quality and usefulness in order to arrive at a medical decision [13]. More rigorous orthopaedic research with a higher LOE is associated with higher rates of citations and publications [14,15]. It follows that certain studies are inherently more influential and applicable to practice. With this in mind, one would hope that annual meeting reviewers select abstracts of the highest quality to improve upon the value of the meeting. This idea is supported by recent improvement in the overall strength of evidence of studies disseminated through various orthopaedic journals and annual meetings [16-21].

The American Academy of Orthopaedic Surgeons (AAOS) Annual Meeting integrates innovators from around the world and serves as a leading forum for exchanging the latest in orthopaedic research. A widely attended event, the AAOS Annual Meeting provides both significant educational and clinical value. As a result, the authors hypothesize that the Annual Meeting is a reliable measure of trends within orthopaedic research over time. Although the quality and impact of a study may be subjective, the use of level of evidence (LOE) helps assess each study in a more objective and systematic fashion.

The purpose of this study is to evaluate the quality and types of studies that have been accepted to AAOS Annual Meeting. Specifically, the change in utilization of large administrative databases will be analyzed and compared with other study architectures including single and multi-center studies, along with trends in LOE as a measure of quality in orthopaedic research. The authors hypothesize that there has been an increase in large administrative databases at the AAOS Annual Meetings. However, to our knowledge, no prior study analyzes how large databases impact the scientific quality of the research presented at the AAOS Annual Meetings. Several variables in addition to data source, including study design, will be quantified to define other study features associated with LOE. This will help elucidate trends in orthopaedic research and subsequently determine if certain features are associated with changes in the clinical quality of research over time.

Methods

The study group chosen was the AAOS Annual Meeting research abstracts that were accepted for podium presentation. Podium abstracts have been shown to be of higher quality and are more likely to be subsequently published in refereed journals [22-24]. Thus, poster presentations were excluded from this study in order to provide a more uniform selection of higher quality studies that are likely to represent the state of research being performed annually and reported in the orthopaedic literature. Electronic versions of the AAOS Proceedings were obtained courtesy of the AAOS from 2008-2018 for a total of 11 consecutive years. Each paper was independently assessed by five reviewers, and the results were tabulated. None of the reviewers had formal epidemiological training, but each reviewer was familiar with the general principles of research methods and designs. In order to systemize the approach to assigning the LOE and to ensure consistency between reviewers, the study utilized the 2011 LOE scheme designed by the Oxford Centre for Evidence-Based Medicine (OCEBM) [25]. Many LOE systems focus on treatment effects and harms. However, the design of the OCEBM levels system is based on the entire progression of a clinical encounter (i.e. diagnosis, prognosis, treatment, benefits, and harms), which allows for more comprehensive assessments of each individual study [13]. The benefit of the OCEBM system is that the study can be evaluated and applied to more specific aspects of patient care.

The following variables were recorded: (1) study design (case series/reports, case-control, cross-section, retrospective, prospective, randomized control, systematic review, meta-analysis, other); (2) the number of institutions involved (single-center, multi-center, state/national database); (3) study focus (basic science, clinical practice, practice management, educational); and (4) level of evidence (I-V). Reviewers also had the opportunity to review the authors’ published manuscript if further explanation of the method or design was necessary.

The data was extracted from each study and recorded in Microsoft Excel. Nonparametric Spearman rank correlation was used to assess the relationship between each variable and the year. Significance for statistical analysis was set to p<0.05.

Results

In total, 8,866 AAOS paper abstracts between 2008-2018 were analyzed. The total number of paper abstracts accepted each year increased significantly over the study period (Figure 1), although they did seem to level off in the final four years. This is a clearly a function of the AAOS meeting schedule which has included greater time for presentation of research projects at the podium throughout the course of the study. While the percentage of studies with Level I evidence remained stable, there was an increase in the percentages of Level II (rs=0.9, p<0.001) and Level III (rs=0.89, p<0.001) evidence with a dramatic decrease in Level IV (rs=-0.92, p<0.001) and a statistically significant but small decrease in Level V (rs=-0.62, p<0.05) evidence (Table 1 and Figure 2).

Table 1. Trends in overall level of evidence

	Level I		Level II		Level III		Level IV		Level V
Year	#	%	#	%	#	%	#	%	#	%
2008	60	10.3	106	18.2	114	19.6	228	39.2	74	12.7
2009	70	10.4	124	18.4	140	20.8	277	41.2	62	9.2
2010	74	10.3	130	18.0	138	19.2	305	42.4	73	10.1
2011	88	11.8	136	18.2	139	18.6	293	39.2	91	12.2
2012	81	10.0	158	19.5	176	21.7	299	36.9	96	11.9
2013	84	10.2	173	20.9	190	23.0	294	35.6	85	10.3
2014	70	8.4	190	22.9	203	24.5	266	32.0	101	12.2
2015	87	9.5	216	23.5	233	25.4	277	30.1	106	11.5
2016	95	10.3	204	22.1	251	27.3	292	31.7	79	8.6
2017	95	10.3	223	24.2	251	27.3	273	29.7	78	8.5
2018	98	10.7	234	25.5	247	26.9	261	28.4	78	8.5

Figure 1. Number of podium abstracts per year

Figure 2. Trends in overall level of evidence

Analysis of study designs revealed a significant percentage decrease in case series (rs=-0.95, p<0.001) with an increase in both case-control (rs=0.69, p<0.02) and cohort (rs=0.94, p<0.001) designs. In addition, there was a decrease in the percentage of randomized controlled trials (rs=-0.86, p<0.001) over the years, although the total number of RCT’s presented did not change (n=66 in 2008; n=65 in 2018) (Table 2).

Table 2. Study design

	Case series		Case control		Cohort		RCT
Year	#	%	#	%	#	%	#	%
2008	249	42.8	26	4.5	179	30.8	66	11.3
2009	268	39.8	26	3.9	243	36.1	60	8.9
2010	286	39.7	24	3.3	260	36.1	64	8.9
2011	306	41.0	23	3.1	253	33.9	76	10.2
2012	303	37.4	29	3.6	303	37.4	79	9.8
2013	296	35.8	46	5.6	326	39.5	73	8.8
2014	274	33.0	65	7.8	324	39.0	67	8.1
2015	276	30.0	68	7.4	398	43.3	55	6.0
2016	286	31.1	79	8.6	374	40.6	77	8.4
2017	262	28.5	50	5.4	384	41.9	72	7.8
2018	263	28.6	69	7.5	414	45.1	65	7.1

The percentage of basic science studies significantly decreased (rs=-0.64, p<0.04) although the total number did not appreciably change, while studies involving practice management (rs=0.74, p<0.01) dramatically increased in total number and percentage (Table 3). LOE distribution of studies classified as practice management showed a significant percentage increase in Level II evidence (rs=0.77, p<0.01) (Table 4). There were no significant changes for studies focused on either clinical practice or education.

Table 3. Study focus

	Basic science		Practice management
Year	#	%	#	%
2008	96	16.5	5	0.86
2009	82	12.2	18	2.7
2010	108	15.0	35	4.9
2011	107	14.3	54	7.2
2012	147	18.1	46	5.7
2013	116	14.0	72	8.7
2014	135	16.3	44	5.3
2015	119	12.9	84	9.1
2016	66	7.2	79	8.6
2017	88	9.6	75	8.2
2018	105	11.4	69	7.5

Table 4. LOE distribution for practice management studies

	Level I		Level II		Level III		Level IV		Level V
Year	#	%	#	%	#	%	#	%	#	%
2008	0	0	1	20	2	40	1	20	1	20
2009	3	16.7	2	11.1	7	38.9	2	11.1	4	22.2
2010	3	8.6	6	17.1	2	5.7	22	62.9	2	5.7
2011	4	7.4	9	16.7	17	31.5	11	20.3	13	24.1
2012	3	6.5	9	19.6	14	30.4	13	28.3	7	15.2
2013	4	5.6	12	16.7	25	34.7	17	23.6	14	19.4
2014	3	6.8	10	22.7	15	34.1	14	31.8	2	4.6
2015	5	6.0	18	21.4	27	32.1	3	3.6	31	36.9
2016	6	7.6	37	46.8	13	16.5	20	25.3	3	3.8
2017	5	6.7	19	25.3	21	28.0	15	20.0	15	20.0
2018	5	7.2	20	29.0	23	33.3	11	16.0	10	14.5

There was a dramatic increase in the percentage (rs=0.9, p<0.001) and absolute number (n=31 in 2008; n=157 in 2018) of studies utilizing national/state databases while the percentage of studies using single-center data decreased (rs=-0.92, p<0.001). There was no change in the percentage of studies conducted with multi-center data (Table 5).

Table 5. Data source

	Single-center data		Multi-center data		National/state databases
Year	#	%	#	%	#	%
2008	467	80.2	84	14.4	31	5.3
2009	547	81.3	89	13.2	37	5.5
2010	597	82.9	67	9.3	56	7.8
2011	620	82.9	94	12.6	33	4.4
2012	648	80.0	106	13.1	56	6.9
2013	633	76.6	111	13.4	82	9.9
2014	614	73.9	113	13.6	103	12.4
2015	662	72.0	125	13.6	132	14.4
2016	640	69.5	118	12.8	163	17.7
2017	643	69.9	137	14.9	140	15.2
2018	637	69.4	124	13.5	157	17.1

Further analysis of the LOE distribution based on data source revealed that studies using national and state databases had significant percentage increase in Level II evidence (rs=0.85, p<0.001) (Table 6). There were, in addition, significant percentage increases in both Level II (rs=0.78, p<0.01) and Level III evidence (rs=0.78, p<0.01) as well a significant decrease in Level IV evidence (rs=-0.90, p<0.001) in studies using single-center data (Table 7). There were no significant changes in any LOE of studies using multi-center data.

Table 6. LOE Distribution for national/state database studies

	Level I		Level II		Level III		Level IV		Level V
Year	#	%	#	%	#	%	#	%	#	%
2008	1	3.2	5	16.1	11	35.5	8	25.8	6	19.4
2009	1	2.7	5	13.5	15	40.5	6	16.2	6	16.2
2010	0	0	10	17.9	15	26.8	30	53.6	1	1.7
2011	7	21.2	4	12.1	13	39.4	8	24.2	1	3.1
2012	7	12.5	10	17.9	20	35.7	14	25.0	5	8.9
2013	8	9.8	17	20.7	27	32.9	23	28.0	7	8.6
2014	4	3.9	28	27.2	41	39.8	28	27.2	2	1.9
2015	4	3.0	53	40.2	46	34.8	11	8.3	18	13.7
2016	3	1.8	50	30.7	75	46.0	29	17.8	6	3.7
2017	7	5.0	45	32.1	67	47.9	14	10.0	7	5.0
2018	6	3.8	46	29.3	67	42.7	22	14.0	16	10.2

Table 7. LOE distribution for single-center studies

	Level I		Level II		Level III		Level IV		Level V
Year	#	%	#	%	#	%	#	%	#	%
2008	44	9.4	77	16.5	79	16.9	202	43.3	65	13.9
2009	51	9.3	100	18.3	102	18.6	241	44.1	53	9.7
2010	67	11.2	107	17.9	113	18.9	245	41.1	65	10.9
2011	65	10.5	115	18.5	103	16.6	252	40.6	85	13.7
2012	64	9.9	129	19.9	128	19.7	245	37.8	82	12.7
2013	53	8.4	138	21.8	143	22.6	234	36.9	65	10.3
2014	50	8.1	133	21.7	127	20.7	208	33.9	96	15.6
2015	70	10.6	127	19.1	154	23.2	233	35.2	78	11.9
2016	74	11.6	124	19.4	143	22.3	228	35.6	71	11.1
2017	80	12.4	139	21.6	129	20.1	230	35.8	65	10.1
2018	73	11.4	154	24.2	147	23.1	208	32.7	55	8.6

Discussion

The fundamental finding of this study is that the quality of research measured at the AAOS meeting is not only being maintained but is actually improving as measured by the LOE. The greatest impact appears to be in the Level IV studies which have dramatically declined as a percentage over the 11-year period measured to the benefit primarily of Level II and Level III studies. During this period, the total number of abstracts accepted for presentation has increased by over 40% reflecting schedule changes on the part of the AAOS program committee. It would be logical to raise the concern that the increase in presentation opportunities could lead to dilution of the quality of the studies; however, the opposite appears to be the case. The authors did not evaluate information about number of abstracts submitted for evaluation, but this data would suggest that the number of quality abstracts submitted is keeping pace and even exceeding the acceptance opportunities.

The finding that research based upon administrative databases has risen to over 17% of podium presentations is consistent with previous studies and clearly indicates the prominence of these studies in use today as well as some measure of their utility and general acceptance [1-3]. The ease with which researchers can access large sample sizes makes these databases appealing for researchers who wish to conduct studies in a timely manner while strengthening conclusions. This study demonstrates a concomitant increase in the use of large databases and a decrease in studies performed at a single institution. It is likely that the overall validity of these databases will remain controversial for some time. Many authors have cautioned about the generalizability of this type of data and the potential for significant bias [1-5]. The fact that these types of studies seem to be replacing single center studies is difficult to evaluate, but in the most recent year, there are still over 4 times as many single center studies as database studies reflecting a continued preference on the part of researchers or reviewers for the latter.

It is important to note that the percentage of randomized control trials (RCTs) performed has decreased over the last decade, but the absolute number has not changed very much. Although no study is completely free of confounding or bias, well-conducted RCTs are the hallmark of evidence-based medicine. However, RCTs tend to be complex, burdened by regulation, time-consuming, and often require extensive funding and outside resources, which may explain this trend [3,26,27]. Despite this decrease in randomized designs, the rate of level I designated studies accepted by the Annual Meeting committee remained consistent over the study period. Based on the findings, the absence of a significant change in the percentage of level I studies was due to the implementation of high-quality cohort, systematic review, and meta-analysis studies. Historically, the consensus has been that study designs in clinical research are hierarchical, with RCTs remaining the most prominent tool. Although RCTs are likely to provide the best possible evidence, well-designed cohort studies have demonstrated remarkably similar summary results when compared to RCTs [28]. The stability of studies with level I evidence reassuringly demonstrates that the gradual change in study methods by researchers over the years has not compromised the reliability of the highest-quality evidence presented at the Annual Meetings as measured by LOE. It will be worth evaluating in the future the role of the meta-analysis as a relatively recent addition to the armamentarium of researchers that seems to be gaining in prominence although the total numbers remain small. It is also impossible to know whether this reflects the preference of authors or the bias of reviewers.

The increase in the rate of acceptance of Level II studies is encouraging since it reflects the improved quality of the studies being accepted by the committee. Based on the grading criteria, level II studies in particular are represented by high quality diagnostic and prognostic cohort studies in addition to systematic reviews [25]. This is reflected by the significant increase in cohort studies throughout the study period. The only other variable measured in this study with significant increases were the use of large databases. In fact, when further analyzing the LOE distribution based on each data source, results showed that studies specifically utilizing large administrative databases were associated with a statistically significant increase in Level II studies. This may be attributed to the large effect sizes that can be obtained from these databases. Although the overall percentage of studies utilizing single-center data decreased over time, LOE distribution analysis of single-center studies also revealed an increase in the percentage of both Level II and Level III evidence. The fact that single-center studies are associated with improvements in scientific research despite a relative decrease in percentage of this data source highlights the continued importance of these small, but high-quality studies in research.

Likewise, the rate of acceptance of level III studies by the committee increased. Similar to level II studies, level III studies are largely represented by non-randomized controlled cohort studies [25]. However, level III studies typically have smaller sample and effect sizes than level II studies and do carry a greater risk of confounding. Based on the study’s findings and the OCEBM’s classification system, the increase in level III studies is driven by these cohort studies and smaller single-center studies. Though the evidence may not be as strong as a level I or II study, the level III studies still present sufficiently strong data to come to a decision and are occurring at the expense of less well controlled studies (Level IV and V).

The increase in level II and III evidence demonstrates the improved quality of the abstracts accepted by the committee. Nevertheless, it is equally important to address the decrease in level IV studies. Level IV studies are largely represented by case-series or otherwise poorly controlled cohort studies. Generally speaking, case-series are associated with weaker forms of evidence. As mentioned previously, an increase in large cohort studies was likely the driving force behind the increase in level II and III studies. Similarly, the yearly decrease in the rate of case series designs is mirrored by a decrease in level IV studies. LOE distribution analysis also showed that there was a significant percentage decrease in Level IV evidence in studies using single-center data. There appears to be a shift in focus away from low-quality study designs at these smaller centers. Another low-quality form of evidence according to OCEBM’s system, level V studies are largely observations utilizing mechanism-based reasoning. The absolute number did not change over time, but the percentage of Level V observation studies, certainly the weakest of all study designs, decreased significantly. The relative decrease in level IV and V studies again supports how the Annual Meeting continues to improve the overall quality of studies from top to bottom.

It is not particularly surprising that research is, in part, affected by greater societal concerns and an increase in studies looking at practice management was demonstrated. In the context of alternative health care models including prospective payments and bundled payments, many societies have focused their offerings of seminars and presentations on the orthopaedic surgeons’ role in these new systems. Clearly, a significant reason for the increase in focus on practice management may be attributed to the relative lack of literature on this topic and a decision on the part of the clinicians and society to increase the opportunities for presentation [7,8]. One can reasonably assume that this emphasis is commensurate with an interest on the part of researchers or meeting attendees, or both, to be educated in this fashion. The distribution of LOE for practice management studies showed an increase in high-quality Level II evidence. As information becomes computerized, outcomes, quality of care, and value-based data can be gathered rather efficiently at the single-center level or with the use of large databases, both of which were also associated with increases in Level II evidence. Regardless of the reason for the increased popularity of these types of studies, the overall quality of research being presented as measured by the present analysis has not declined.

Despite this advancement of LOE, there still remains room for improvement. It is reassuring to observe that the rate of level I studies has not decreased over time. However, because level I studies are the most rigorously conducted and thus the most reliable, it would benefit orthopaedics as a specialty to strive to increase the number of such studies. Based on this study’s findings and the OCEBM levels, improving the rate of level I studies does not solely rely upon RCTs. Large cohort studies, systematic reviews, and meta-analyses that are well-conducted utilizing already available databases can help increase the rate of level I studies.

This study has several limitations. A potential confounder is that the authors reviewed abstracts accepted rather than abstracts submitted. This creates potential that there is bias among the reviewers for certain types of papers. However, given the large number of reviewers and the large number of paper presentations that are ultimately published in journals, this seems reasonable for evaluation. In order to create a more powerful analysis, the authors decided to review paper abstracts for every Annual Meeting from 2008-2018. While LOE is an excellent method of systematically evaluating the quality of research articles, the sheer number of abstracts required several reviewers. Having several reviewers reduces the risk of grader bias, but evaluating LOE can still be a subjective matter, which could result in minor inconsistencies between observers. In theory, using OCEBM’s classification system reduces the risk of inter-grader bias. Still, it is important to consider that focusing on specific criteria for each level entails a risk that graders assign a level without critically analyzing the study as a whole [29]. The increased focus on research practices and evidence-based medicine today warrants critical analysis of medical literature. The explosion of research evidence, partly due to these large databases, has led to a multitude of innovative diagnostic or treatment methods, many of which may provide marginal benefit at best even though the presented data portends otherwise [30]. With the development of many alternative clinical choices, physicians must be able to incorporate both literature and patient preferences when making medical decisions or recommendations. It is important for modern-day clinicians to utilize their own judgment and evaluate studies within the context of good patient care independent of an assigned LOE [30,31]. Ultimately, the OCEBM levels do not provide a recommendation even if the study is supported by strong evidence. The responsibility lies with the clinician to be aware of different study designs and to determine how to best apply the findings [13,28].

Further, grading of each study was largely based on the information provided within the limitations of each abstract. If there was uncertainty regarding a particular feature of the study design, the reviewers evaluated the published manuscript if it was available. Despite these limitations, the sheer number of abstracts reviewed and the subsequent statistically significant trends found in this study confirms the continuous improvement of the abstracts accepted and distributed at the AAOS Annual Meetings.

In conclusion, the quality of research in orthopaedics as measured by the LOE of paper presentations at the AAOS annual meeting has improved over an 11-year period, in spite of a considerable increase in the number of presentations, which alleviates the concern that more papers could be associated with lower quality. During this same time period, the type of research has evolved including an increase in studies using large databases and meta-analyses, as well as a significant increase in practice management research. This has all occurred without negatively affecting quality. Large databases provide researchers with a powerful tool that enables the utilization of large study populations to facilitate the study of clinical questions. However, there is limited research regarding the impact of these databases on the scientific quality of abstracts. The findings in this study validate the hypothesis and show that the growing use of large-scale databases by researchers has been associated with increased volume and quality of studies accepted to the AAOS Annual Meeting over the last decade, although this does not prove cause and effect. In spite of this favorable finding, the authors caution that conclusions drawn from database studies must continue to be carefully scrutinized on an individual basis as the database, methodology, accuracy, and scope of study can impact not only study results, but also patient care.

Acknowledgements

Annual Meeting Proceedings containing all paper abstracts from 2008 to 2018 were obtained courtesy of Conventions, Course, and Exhibits at AAOS. The authors received no financial support for the research, authorship, or publication of this article. The interpretations and conclusions of this paper are solely that of the listed authors.

Disclosure

None of the authors involved with this study have affiliations with or involvement in any organization or entity with either financial or non-financial interest in the subject matter discussed in this manuscript.

Author contributions

R.H., A.B., and A.S. designed the study and conceived the presented idea. R.H., E.W., and M.R. carried out the study, performed the statistical analyses, and drafted the manuscript with support and contributions from A.B. and A.S. A.S. supervised the entirety of the project including the approval of the final version. All authors have read and approved the final submitted manuscript.

References

1. Bohl DD, Singh K, Grauer JN (2016) Nationwide databases in orthopaedic surgery research.J Am Acad Orthop Surg24: 673-682. [Crossref]
2. Karlson NW, Nezwek TA, Menendez ME, Tybor D, Salzler MJ (2018) Increased utilization of American administrative databases and large-scale clinical registries in orthopaedic research, 1996 to 2016. J Am Acad Orthop Surg Glob Res Rev 2: e076. [Crossref]
3. Pugely AJ, Martin CT, Harwood J, Ong KL, Bozic KJ, et al. (2015) Database and registry research in orthopaedic surgery: Part 2: Clinical registry data.J Bone Joint Surg Am97: 1799-1808. [Crossref]
4. Sivasundaram L, Pannell W, Heckmann N, Ram Kiran A, Omid R, et al. (2016) Database studies: An increasing trend in the United States orthopaedic literature. Curr Orthop Pract 27: 673-679.
5. Hashimoto R, Brodt E, Skelly A, Dettori J (2014) Administrative database studies: Goldmine or goose chase? Evid Based Spine Care J 5: 74-76. [Crossref]
6. Brauer CA, Rosen AB, Olchanski NV, Neumann PJ (2005) Cost-utility analyses in orthopaedic surgery.J Bone Joint Surg Am87: 1253-1259. [Crossref]
7. Lansky D, Nwachukwu BU, Bozic KJ (2012) Using financial incentives to improve value in orthopaedics. Clinical Orthopaedics and Related Research 470: 1027-1037.
8. Nwachukwu BU, Schairer WW, Bernstein JL, Dodwell ER, Marx RG, et al. (2015) Cost-effectiveness analyses in orthopaedic sports medicine: A systematic review. Am J Sports Med 43: 1530-1537. [Crossref]
9. Romano P, Hussey P, Ritley D (2010) Selecting quality and resource use measures: A decision guide for community quality collaboratives. AHRQ Publ 9: 1-106.
10. Moore L, Clark DE (2008) The value of trauma registries.Injury39: 686-695. [Crossref]
11. Bohl DD, Russo GS, Basques BA, Golinvaux NS, Fu MC, et al. (2014) Variations in data collection methods between national databases affect study results: A comparison of the nationwide inpatient sample and national surgical quality improvement program databases for lumbar spine fusion procedures. J Bone Jt Surg 96: e193. [Crossref]
12. Somani S, Di Capua J, Kim JS, Kothari P, Lee NJ, et al. (2017) Comparing national inpatient sample and national surgical quality improvement program: An independent risk factor analysis for risk stratification in anterior cervical discectomy and fusion. Spine 42: 565-572. [Crossref]
13. OCEBM levels of evidence working group, Howick J, Chalmers I, Glasziou P, Greenhalgh T, et al. (2011) Explanation of the 2011 oxford centre for evidence-based medicine (OCEBM) levels of evidence (background document). Oxford Cent Evidence-Based Med 1: 5653 Available from:http://www.cebm.net/index.aspx?o=1025.
14. Okike K, Kocher MS, Torpey JL, Nwachukwu BU, Mehlman CT, et al. (2011) Level of evidence and conflict of interest disclosure associated with higher citation rates in orthopedics. J Clin Epidemiol 64: 331-338. [Crossref]
15. Voleti PB, Donegan DJ, Kim TWB, Lee GC (2013) Level of evidence: Does it change the rate of publication and time to publication of American Academy of Orthopaedic Surgeons presentations? J Bone Jt Surg 95: 1-5.
16. Hanzlik S, Mahabir RC, Baynosa RC, Khiabani KT (2009) Levels of evidence in research published in the Journal of Bone and Joint Surgery (American volume) over the last thirty years. J Bone Jt Surg 91: 425-428.
17. Slobogean GP, Dielwart C, Johal HS, Shantz JA, Mulpuri K (2013) Levels of evidence at the orthopaedic trauma association annual meetings. J Orthop Trauma 27: e208-e212. [Crossref]
18. Grant HM, Tjoumakaris FP, Maltenfort MG, Freedman KB (2014) Levels of evidence in the clinical sports medicine literature. Am J Sports Med 42: 1738-1742.
19. Kay J, Memon M, De SA D, Simunovic N, Athwal GS, et al. (2016) Level of clinical evidence presented at the open and closed American Shoulder and Elbow Surgeons annual meeting over 10 years (2005-2014). BMC Musculoskelet Disord 17: 1-6.
20. Kay J, de SA D, Shallow S, Simunovic N, Safran MR, et al. (2015) Level of clinical evidence presented at the international society for hip arthroscopy annual scientific meeting over 5 years (2010-2014). J Hip Preserv Surg 2: hnv059.
21. Kay J, Memon M, Simunovic N, de Sa D, Ayeni OR (2016) Level of clinical evidence presented at the arthroscopy association of north America annual meeting over 10 years (2006-2015). Arthroscopy 32: 686-691. [Crossref]
22. Preston CF, Bhandari M, Fulkerson E, Ginat D, Koval KJ, et al. (2005) Podium versus poster publication rates at the orthopaedic trauma association. Clin Orthop Relat Res 437: 260-264. [Crossref]
23. Kinsella SD, Menge TJ, Anderson AF, Spindler KP (2015) Publication rates of podium versus poster presentations at the American orthopaedic society for sports medicine meetings: 2006-2010. Am J Sports Med 43: 1255-1259. [Crossref]
24. Frank RM, Cvetanovich GL, Collins MJ, Arns TA, Black A, et al. (2017) Publication rates of podium versus poster presentations at the arthroscopy association of north America meetings 2008-2012. Arthroscopy 33: 6-11. [Crossref]
25. OCEBM levels of evidence working group, Howick J, Chalmers I, Glasziou P, Greenhalgh T, et al. (2011) The oxford levels of evidence 2. Oxford Cent Evidence-Based Med 1: 5653 Available from:https://www.cebm.net/index.aspx?o=5653.
26. Emanuel EJ, Schnipper LE, Kamin DY, Levinson J, Lichter AS (2003) The costs of conducting clinical research.J Clin Oncol21: 4145-4150. [Crossref]
27. Vickers AJ, Scardino PT (2009) The clinically-integrated randomized trial: Proposed novel method for conducting large trials at low cost. Trials 10.
28. Concato J, Shah N, Horwitz RI (2000) Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med 342: 1887-1892.
29. Evidence-based medicine working group (1992) Evidence-based medicine. A new approach to teaching the practice of medicine.JAMA268: 2420-2425. [Crossref]
30. Haynes RB, Devereaux PJ, Guyatt GH (2002) Clinical expertise in the era of evidence-based medicine and patient choice. Evid Based Med 7: 36-38.
31. Mir T (1999) Evidence-based medicine: Transferring evidence from research into practice: The role of clinical care research evidence in clinical decisions. JK Pract 6: 24-27.