Advertisement

Radiation Therapy Research: A Global Analysis 2001-2015

Open AccessPublished:March 20, 2018DOI:https://doi.org/10.1016/j.ijrobp.2018.03.009
      Radiation therapy is a core modality of cancer treatment; however, concerns have been expressed regarding its underutilization and its lack of prioritization as a research domain relative to other cancer treatment modalities, despite its rapid technical evolution. It is therefore important to understand, from a public policy perspective, the evolution of global radiation therapy research, to identify strengths, weaknesses, and opportunities. This study used a bibliometric approach to undertake a quantitative analysis of global radiation therapy research published between 2001 and 2015 and available in the Web of Science (Wos) database, with particular focus on the 25 leading research-active countries. A total of 62,550 radiation therapy research articles from 127 countries, published in 2531 international journals, were analyzed. The United States was responsible for 32.3% of these outputs, followed by Japan (8.0%) and Germany (7.7%). Nearly half of all publications related to preparation and delivery of radiation therapy, combined-modality regimens, and dose fractionation studies. Health services research, palliative care, and quality of life studies represented only 2%, 5%, and 4% of all research outputs, respectively. Countries varied significantly in their commitment to different research domains, and trial-related publications represented only 5.1% of total output. Research impact was analyzed according to 3 different citation scores, with research outputs from Denmark, The Netherlands, and the United States consistently the highest ranked. Globally, radiation therapy publication outputs continue to increase but lag behind other spheres of cancer management. The types of radiation therapy research undertaken appear to be regionally patterned, and there is a clear disconcordance between the volume of research output from individual countries and its citation impact. Greater support for radiation therapy research in low- and middle-income countries is required, including international collaboration. The study findings are expected to provide the requisite knowledge to guide future radiation therapy research programs.

      Introduction

      Cancer research is one of the most globally active domains of science, with more than $14 billion per annum in public and private expenditure (
      • Eckhouse S.
      • Lewison G.
      • Sullivan R.
      Trends in the global funding and activity of cancer research.
      ). Research is integral to improving outcomes from cancer, be it through an improved understanding of the etiology of disease, identifying new treatment targets or modalities, or by the provision of information on how best to coordinate cancer services to deliver affordable and equitable cancer care.
      Radiation therapy is one of the main modalities of control, cure, and palliation for cancer, with approximately 50% of cancer patients requiring radiation therapy during their disease course (
      • Delaney G.P.
      • Barton M.B.
      Evidence-based estimates of the demand for radiotherapy.
      ,
      • Shack L.
      • Lu S.
      • Weeks L.A.
      • et al.
      Determining the need and utilization of radiotherapy in cancers of the breast, cervix, lung, prostate and rectum: A population level study.
      ,
      • Borras J.M.
      • Lievens Y.
      • Dunscombe P.
      • et al.
      The optimal utilization proportion of external beam radiotherapy in European countries: An ESTRO-HERO analysis.
      ). Given the outcomes it can deliver with respect to survival and quality of life, its under-utilization as a clinical modality (
      • Shack L.
      • Lu S.
      • Weeks L.A.
      • et al.
      Determining the need and utilization of radiotherapy in cancers of the breast, cervix, lung, prostate and rectum: A population level study.
      ) and the lack of prioritization it has been given as a research domain within the cancer spectrum remain major concerns. A recent study within lung cancer has found that only 8% of lung cancer research is devoted to radiation therapy research, compared with 20% for genetics, 17% for systemic therapies, and 16% for prognostic biomarkers (
      • Aggarwal A.
      • Lewison G.
      • Idir S.
      • et al.
      The state of lung cancer research: A global analysis.
      ). This undoubtedly will influence patterns of care for particular disease entities and the potential for new developments that will ultimately improve patient outcomes.
      It is therefore important to understand from a public policy perspective how, why, and which particular research domains evolve and have an impact on outcomes. For example, how do different countries influence the radiation therapy research agenda, either through the volume of research they publish, the citation impact of their articles, or their commitment to particular research domains (eg, basic science, health services research)?
      An empirical analysis of research outputs would also highlight gaps and provide direction as to which research areas should be prioritized to meet current and future challenges. For example, there is currently an ongoing debate (
      • Wallner P.E.
      • Steinberg M.L.
      • McBride W.H.
      • et al.
      A fork in the road: Choosing the path of relevance.
      ) as to whether the increasing focus on technological innovation within radiation therapy to improve the therapeutic ratio has been at the cost of not developing a greater understanding of the potential role for radiation therapy in exploiting cancer weaknesses based on the biological hallmarks of cancer. It is thought that further research into these areas could unlock new and more cost-effective opportunities for radiation therapy to improve outcomes from cancer. In addition, it would help to ensure that radiation therapy continues to be relevant and that research in this area is prioritized in an era of precision medicine, which is increasingly drug-focused (
      • Zietman A.
      The future of radiation oncology: The evolution, diversification, and survival of the specialty.
      ,
      • Lawler M.
      • French D.
      • Henderson R.
      • et al.
      Shooting for the moon or flying too near the sun? Crossing the value Rubicon in precision cancer care.
      ).
      Using a bibliometric approach, we present an analysis of global research on cancer radiation therapy between 2001 and 2015. This type of analysis is now used routinely in public policy analysis to study research domains (
      • Derrick G.E.
      • Pavone V.
      Democratising research evaluation: Achieving greater public engagement with bibliometrics-informed peer review.
      ,
      • Hood W.W.
      • Wilson C.S.
      The literature of bibliometrics, scientometrics, and informetrics.
      ). We examine the growth in research output from 25 leading countries, the volume of research produced relative to their wealth, the main radiation therapy research domains these countries prioritize, and the citation impact of radiation therapy research stratified according to country and research domain.

      Methods and Materials

      We performed a bibliometric analysis of radiation therapy research outputs during 2001-2015, based on articles and reviews in the Web of Science (WoS) database. This contains full bibliographic information about the articles, including all addresses as well as the numbers of citations received by each article in each year. Of note, the use of additional biomedical databases does not significantly increase the yield of relevant journals.
      We used filters (macro algorithms based on logic functions) developed by an expert bibliometrician (G.L.) in conjunction with the team's expertise in radiation therapy and radiation research to identify relevant articles in the WoS. The search for radiation therapy articles included articles in ANY journal, including general medical and basic science journals, provided that they had a title term indicative of cancer (composed of 323 words and short phrases (
      • Lewison G.
      Definition of cancer research: Journals, titles, abstracts or keywords?.
      )) AND had the word “radiation or radiotherapy” including wildcard “rad*,” OR contained 1 of the 12 radiation therapy–specific title words, such as “brachytherapy” (Supplementary Appendix 1; available online at www.redjournal.org). It also included articles in 7 specialist cancer radiation therapy journals identified by the study authors (Supplementary Appendix 1; available online at www.redjournal.org) and in 185 specialist cancer journals, provided that articles in the latter contained 1 of the radiation therapy filter terms. In addition, there were 20 more journals identified by the authors that also covered both cancer and noncancer radiation topics (Supplementary Appendix 1; available online at www.redjournal.org). Articles in these journals were retained ONLY if they had one of the 323 cancer filter terms.
      All these filters were developed through iterative rounds, which involved creating datasets and having these manually coded by clinical experts as to their relevance to the research fields being sought (using methods previously described (
      • Lewison G.
      The definition and calibration of biomedical subfields.
      )). This process resulted in both a precision or specificity and a recall or sensitivity for identifying radiation therapy research articles of 0.95, which is considered very high (
      • Lewison G.
      Definition of cancer research: Journals, titles, abstracts or keywords?.
      ).
      There were 127 countries that contributed to these radiation therapy articles. However, the results presented in this study will primarily focus on the 25 leading research-active countries that are responsible for 97% of the total.
      The counts of the numbers of publications per year were obtained as both integer and fractional counts using the article's addresses. For example, if an article has 2 addresses in Germany and 1 in France it would be counted as 1 for each on an integer count basis, but as 0.67 and 0.33, respectively, on a fractional count basis. Fractional counts sum to less than the total partly because of the outputs of the other countries and partly because some articles had no addresses. Integer counts summed to more than the total because of international co-authorship. Unless otherwise mentioned, our analysis is based on fractional counts, which give a much better impression than integer counts of the relative research effort by each country (
      • Waltman L.
      • van Eck N.J.
      Field-normalized citation impact indicators and the choice of an appropriate counting method.
      ,
      • Huffman M.D.
      • Baldridge A.
      • Bloomfield G.S.
      • et al.
      Global cardiovascular research output, citations, and collaborations: A time-trend, bibliometric analysis (1999-2008).
      ).

       Volume of research

      For each of the 25 leading countries, we analyzed the numbers of published radiation therapy research articles for each year from 2001 to 2015 and calculated the annual average percentage growth rate (AAPG) and the ratio of outputs between 2011-2015 and 2001-2005. We also calculated the commitment of the 25 countries to radiation therapy research relative to their output of cancer research overall. For example, between 2001 and 2015, Canada published 3956 articles out of a world total of 62,550 (6.3%). In the same period, Canada contributed to 4.2% of all oncology research outputs worldwide. Canada's relative commitment to radiation therapy research was therefore 6.3/4.2 = 1.50. For selected high- and middle-income countries we also analyzed the association between radiation therapy research output and each country's gross domestic product (GDP). The GDP is a measure of a country's economy and is the total market value of all consumer goods and services produced by all the people and companies in the country in a period of time (quarterly or yearly).

       Radiation therapy research domains

      Radiation therapy research publications were categorized into 10 research domains. These were defined using subfilters that contained a set of title words and strings to categorize relevant articles into particular domains. These subfilters were all developed as part of an iterative process by A.A. and Y.L. in collaboration with G.L., with additional terms being added to each of the sufilters to capture many of the articles not yet classified. We also used a complex logic process as the individual subfilters were applied to the spreadsheet, so that some articles were classified if they were identified by particular subfilters but NOT by others.
      The research domains and individual codes used for analysis were as follows: BIOL ((radio)biology); PHYS (physics); ASSU (quality assurance); FRAC (dose fractionation and sequencing studies); COMB (multimodality studies involving radiation therapy); PRED (preparation and delivery of radiation therapy); PALL (palliative care); QUAL (quality of life); HESR (health services research); and REVS (review articles). Of note, the PRED domain also included studies evaluating particle therapy in the clinical setting. The domains were not mutually exclusive, and therefore articles reviewing quality of life in relation to clinical dose fractionation studies would be included in both categories (ie, QUAL and FRAC). In addition, articles classified solely within either BIOL, PHYS, or ASSU were treated as nonclinical basic science studies. The PRED, FRAC, and COMB domains included studies that measured clinical endpoints such as morbidity. Eventually, all but 6465 articles were classified into 1 or more of these categories (10.3%).
      Fractional counts were calculated for each country and research type, and the results compared with the world totals so as to show whether a country was over- or underrepresented relative to the world average for each research domain. We also analyzed the number of articles describing radiation therapy trials (phases 1-4) and the fractional counts of these articles for each country.

       Citation impact by country and research domain

      The citation counts for each article from 2001-2011, year by year, were downloaded from the WoS. The 5-year citation counts (actual citation impact, ACI) beginning in the publication year were calculated. A 5-year window was used as a compromise between the need for immediacy (ie, citations to recent articles) and stability (ie, inclusion of the peak year for citations, usually the second or third year after publication). It is best determined for a country on the basis of fractional counts, because many of the most cited articles are multinational. Altogether, citation counts were determined for 32,162 articles.
      For each country we calculated an arithmetic mean citation score as a measure of the impact of their radiation therapy–related research. This was based on the ACI of each of their articles during the study period. To calculate this, the citation score for each article was multiplied by the country's fractional contribution to that article and the products summed, and the total divided by the sum of the country's fractional counts for the relevant years. For example, for Germany, the top-cited article, with 1360 citations, had a German presence among the addresses of 0.133. Germany was therefore credited with 1360 × 0.1333 = 181.3 citations. All the products for individual articles were summed, to give the fractional German citation total of 44,341.4 citations, which when divided by the German fractional count citable total of 3306.9 articles gives an arithmetic mean citation score of 13.41 citations per article. A geometric mean was also calculated on the basis of the logarithms of the actual citation counts. This value is considered to be a better indicator because it is less influenced by a few very high citation counts (
      • Thelwall M.
      Interpreting correlations between citation counts and other indicators.
      ).
      Another measure of citation impact is the number of a country's articles that receive enough citations to put them in the top 1%, 2%, or 5% of all cited radiation therapy articles. During the period of analysis (2001-2011), an article would have had to receive 93, 68, and 44 citations, respectively, to be in these centiles. A WorldScale (WS) value at a particular centile was calculated according to the ratio of the proportion of articles from a selected country in the top x% of cited articles compared with the proportion of all articles in that particular centile. So for Germany, with 3306.9 citable articles in the 11 years, because it published 136.5 articles with 44 or more citations, its WS value at 5% was (136.5/3306.9)/0.05 × 100 = 83. This value is lower than 100 and indicates it has fewer than the expected number of articles in this centile. The 3 WS values at 1%, 2%, and 5% for each country were averaged to give a composite value for a country's highly cited articles—the WS mean.
      The 25 leading countries were ranked separately on these 3 citation indicators (arithmetic, geometric, and WS means), and the rankings were then averaged. Using similar methods, we also determined the citation impacts for articles in the 10 research domains.
      Finally, an analysis of the proportion of publications that were open access between 2001 and 2015 was undertaken, as well as a comparative analysis of 5-year citation scores for open access versus non–open access articles published between 2010 and 2012.

      Results

      In total, 62,550 articles were identified from 2531 international journals and from 127 countries. Nearly all of the articles were in English (60,494 articles, 96.7%), but others were in 20 different languages, led by French (1193, 1.9%), German (598, 0.96%), and Spanish (76, 0.12%). A few were in Chinese, Japanese, or Korean.
      The 25 leading research-active countries, which accounted for 96.9% (or 60,673 articles) of the total in the file, included (in alphabetical order): Australia, Austria, Belgium, Brazil, Canada, China, Denmark, France, Germany, Greece, India, Iran, Italy, Japan, Netherlands, Norway, Poland, Republic of Korea, Spain, Sweden, Switzerland, Taiwan, Turkey, the United Kingdom, and the United States.

       Volume of research

      Between 2001 and 2015 the volume of radiation therapy research increased year on year, with a doubling in world outputs (Fig. 1). The United States was responsible for 32.3% of these outputs (Table 1), followed by Japan (8.0%) and Germany (7.7%). However, there have been significant changes in the volume of research produced from some countries, with Iran, China, Brazil, and South Korea showing the highest AAPG values and with output ratios between 2011-2015 and 2001-2005 of 19.3, 10.4, 4.8, and 4.5, respectively.
      Figure thumbnail gr1
      Fig. 1Increase in radiation therapy research publication output between 2001 and 2015 (n = 62,550).
      Table 1Outputs of radiation therapy articles from the world and 25 leading countries, 2001-2015
      CountryAll years%AAPGRatioR.C.
      Worldwide62,5506.21.87
      United States20,09732.23.71.471.07
      Japan50198.05.41.750.92
      Germany47827.72.71.341.21
      China31225.023.310.40.53
      United Kingdom29864.83.11.390.92
      Canada29444.76.61.911.51
      France29314.78.92.351.09
      Italy26614.35.11.710.81
      Netherlands24483.95.51.741.69
      Republic of Korea21403.416.04.530.99
      Australia14642.311.53.141.13
      Taiwan9231.510.52.640.78
      Spain8791.49.92.640.65
      India8551.411.93.140.73
      Sweden8341.31.71.150.96
      Belgium8231.34.01.471.45
      Turkey7961.38.82.240.85
      Switzerland7691.24.61.541.32
      Denmark5410.911.13.011.21
      Poland4970.89.62.570.78
      Austria4860.8−1.20.901.14
      Brazil4850.815.64.800.64
      Greece4350.70.01.040.81
      Norway4010.67.02.071.02
      Iran2990.532.119.300.88
      Abbreviations: % = percentage of world output; AAPG = annual average percentage growth; Ratio = output in 2011-2015/output in 2001-2005; R.C. = relative commitment to radiation therapy research compared to all cancer.
      Most countries had a smaller percentage presence in radiation therapy research than they did in oncology overall (Table 1). The Netherlands, Canada, and Belgium are notable exceptions. China, despite having a sustained and large increase in its volume of radiation therapy research over the 15-year period, still had a smaller percentage presence in radiation therapy research than in all cancer. A similar pattern was observed for Brazil.
      Figure 2 shows that the volume of radiation therapy research output is positively correlated with each country's GDP, with the spots for most of the selected high- and upper-middle-income countries close to the least-squares regression line. Notable exceptions are The Netherlands, which published more than 3 times as much as the regression line predicts, and Taiwan, Canada, and the United States, whose output was double or almost double the amount predicted. On the other hand, upper-middle-income countries, such as China and Brazil, published less than half the predicted amounts. Russia published less than one-tenth of the amount predicted from the regression line. All these differences are highly statistically significant (P < .001% on the Poisson distribution with 1 degree of freedom).
      Figure thumbnail gr2
      Fig. 2Outputs of radiation therapy research articles as a function of national wealth (log-log scales). Diagonal line is best power-law least-squares correlation line; dashed lines show values twice and half those of the correlation line. Abbreviations: AT = Austria; AU = Australia; BE = Belgium; BR = Brazil; CA = Canada; CH = Switzerland; CN = China; CZ = Czech Republic; DE = Germany; DK = Denmark; ES = Spain; FI = Finland; FR = France; GR = Greece; IE = Ireland; IL = Israel; IN = India; IR = Iran; IT = Italy; JP = Japan; KR = Republic of Korea; NL = The Netherlands; NO = Norway; PL = Poland; RS = Russia; SE = Sweden; SG = Singapore; TR = Turkey; TW = Taiwan; UK = United Kingdom; US = United States.

       Research domains

      Figure 3 shows the distribution of research articles during the study period in 9 of the research domains. Clinical research domains predominate, with studies focusing on (1) preparation and delivery of radiation therapy (PRED, 24%), (2) use of radiation therapy as part of combined modality management (COMB, 17%), and (3) evaluation of different dose fractionation schedules and sequencing of radiotherapy (FRAC, 15%). These 3 domains accounted for nearly half of all published radiation therapy research.
      Figure thumbnail gr3
      Fig. 3Distribution of radiation therapy articles by research domain, 2001-2015, and annual average percentage growth rates for the articles in the research domain. Abbreviations: ASSU = quality assurance; BIOL = (radio)biology; COMB = multimodality studies involving radiation therapy; FRAC = dose fractionation and sequencing studies; HESR = health services research; PALL = palliative care; PHYS = physics; PRED = preparation and delivery of radiation therapy; QUAL = quality of life.
      Basic science represented a further third of research outputs, with radiobiological (BIOL) investigation accounting for 19% and physics (PHYS) for 13% of total outputs. Health services research (HESR) was very limited, accounting for only 2% of the total. Palliative care (PALL) and quality of life studies (QUAL) in radiation therapy also made only small contributions to the literature, of 5% and 4%, respectively.
      Figure 3 shows that between 2001-2005 and 2011-2015, radiobiology research has had the most modest increase in volume of research output (AAPG = 4.8) compared with clinical domains such as preparation and delivery (AAPG = 9.0) and dose fractionation studies (AAPG = 9.3). Health services research has had sustained increases over the time period, with an AAPG of 10.8.
      Table 2 shows the relative commitment of different countries to the 9 research domains. Whereas the United States, France, and Austria are well represented in all the domains, this is not so for other countries. For example, China's radiation therapy research is focused on basic science, particularly on radiobiology. Canada has a focus on physics, palliative care, and health services research. The Netherlands, Belgium, Denmark, and Iran have a strong commitment to physics research, with The Netherlands also noticeable for its commitment to quality of life–related research and Iran to quality assurance. Australia, like Canada, and to a lesser extent the United Kingdom, is notable for its strong commitment to health services research; Norway excels in quality of life and palliative care research.
      Table 2Relative commitment of leading 25 countries to different domains of radiation therapy research, 2001-15 (A color version of this table is available at www.redjournal.org.)
      Abbreviations: ASSU = quality assurance; BIOL = (Radio)biology; COMB = multimodality studies involving radiation therapy; FRAC = dose fractionation and sequencing studies; HESR = health services research; PALL = palliative care; PHYS = physics; PRED = preparation and delivery of radiation therapy; QUAL = quality of life.
      Notes: Cells with values > twice world average tinted bright green; those > 1.40 × world average tinted pale green; those < 0.71 × world average tinted yellow; those < 0.51 × world average tinted pink.

       Clinical trial outputs

      Of the 62,550 radiation therapy articles, only 3926 (6.3%) involved clinical trials. Of these, 34 described preparatory work, and 687 were secondary sources (meta-analyses, systematic reviews, and other trials without specification of stage). The remaining 3205 articles (5.1%) described trials in the 4 established phases: 769 phase 1, 1065 phase 2, 1367 phase 3, and 4 described phase 4 trials. Table 3 demonstrates the variation in clinical trial outputs across the top 20 leading countries. The United States published the greatest number of outputs across phase 1 to phase 3 studies. Japan is notable for the high proportion of publications describing phase 1 studies relative to all other countries. With respect to phase 2 trial outputs, Japan again features strongly, as does Italy and China. Phase 3 trial outputs are dominated by the United Kingdom, Germany, and The Netherlands after the United States. An analysis of the ratio of phase 3 to phase 2 trial outputs from each of the 25 countries showed significant international variation. The United States, for instance, had a relatively low ratio of phase 3 relative to phase 2 outputs. This was also notable for countries such as Japan (0.50), Italy (0.58), Republic of Korea (0.60), and Spain (0.46). Conversely, countries such as the United Kingdom (3.45), Netherlands (4.72), Sweden (4.14), and Poland (5.02) published more than double the number of trial outputs of phase 3 studies compared with phase 2 studies in the study period. India, a middle-income country, also had a strong commitment to phase 3 studies (2.83).
      Table 3Number of articles relating to phase 1, phase 2, and phase 3 radiation therapy trials published by the 20 leading countries between 2001 and 2015
      CountryPhase 1Phase 2Phase 3% of total phase 1 outputs
      Percentage contribution of each individual country to total world phase 1, phase 2, and phase 3 research outputs.
      (n = 769)
      % of total phase 2 outputs
      Percentage contribution of each individual country to total world phase 1, phase 2, and phase 3 research outputs.
      (n = 1065)
      % of total phase 3 outputs
      Percentage contribution of each individual country to total world phase 1, phase 2, and phase 3 research outputs.
      (n = 1367)
      Ratio of phase 3 to phase 2 trial outputs
      Ratio of articles describing phase 3 studies and phase 2 studies for each individual country.
      World77010661367
      United States329.5349.6287.642.832.8210.82
      Japan108.198.148.9149.23.60.50
      Germany48.247.3106.96.34.47.82.26
      China31.661.875.64.15.85.51.22
      United Kingdom26.236.8126.93.43.49.33.45
      Canada28.935.376.83.73.35.62.18
      France236182.335.761.35
      Italy38.782.24857.73.50.58
      Netherlands22.819.391.131.86.74.72
      Republic of Korea12.446.227.91.64.320.60
      Australia13.722.6581.82.14.22.57
      Taiwan35.110.40.40.50.82.04
      Spain1137.3171.43.51.20.46
      India2.59.426.60.30.91.92.83
      Sweden3.68.836.40.50.82.74.14
      Belgium1018.817.41.31.81.30.93
      Turkey06.110.100.60.71.66
      Switzerland12.713.416.71.71.31.21.25
      Denmark4.114.3250.51.31.81.75
      Poland2.34.623.10.30.41.75.02
      Number of articles expressed as fractional country counts.
      Percentage contribution of each individual country to total world phase 1, phase 2, and phase 3 research outputs.
      Ratio of articles describing phase 3 studies and phase 2 studies for each individual country.

       Citation counts and research impact

      Table 4 shows the 25 leading countries with their mean ACI values (arithmetic and geometric) and their mean WS values. The latter is based on the numbers of their articles with enough citations to put them in the top 1% (≥93 citations), top 2% (≥68 citations), and top 5% (≥44 citations), as described in the Methods and Materials section. The countries are ranked according to the mean ranking across the 3 scales.
      Table 4Five-year citation performance of 25 leading countries in radiation therapy research, 2001-2011
      CountryCitableWS (ranking)Arithmetic (ranking)Geometric (ranking)
      Denmark301199 (1)19.8 (1)11.8 (1)
      Netherlands1604147 (3)18.7 (2)11.4 (2)
      United States13,572154 (2)17.2 (3)9.8 (3)
      Switzerland512131 (4)16.7 (5)9.5 (5)
      Belgium563117 (5)16.8 (4)9.6 (4)
      Canada1922101 (7)14.7 (6)8.4 (6)
      United Kingdom2045100 (8=)14.2 (7)7.8 (7)
      Austria35593 (10)13.4 (9=)7.2 (10)
      World39,657100 (8=)13.7 (8)7.3 (9)
      Germany330779 (12)13.4 (9=)7.5 (8)
      Australia80586 (11)12.5 (11)6.8 (13)
      Sweden57174 (13)11.9 (13)7 (11)
      France1804103 (6)12.1 (12)5.6 (17)
      Norway26749 (17)11.1 (14)6.9 (12)
      Italy169057 (14=)10.6 (15)5.3 (18)
      China106245 (18)10.5 (16=)6.1 (15)
      Republic of Korea113530 (21)10.5 (16=)6.3 (14)
      Spain51357 (14=)9.1 (19)4.6 (21)
      Greece32640 (19=)8.6 (21)4.8 (20)
      Taiwan5389 (24)9.4 (18)6 (16)
      Japan324327 (22)8.8 (20)5.2 (19)
      Brazil26540 (19=)7.6 (22)4 (22)
      Poland30250 (16)7.1 (23)3.1 (24)
      India50813 (23)5.4 (24)3.3 (23)
      Turkey4733 (25)4.2 (25)2.3 (25)
      Iran1160 (26)3.6 (26)2.2 (26)
      Abbreviation: WS = WorldScale.
      Citable = numbers of articles in these years; WS = WS mean value at top 1%, 2%, and 5% of citations; Arithmetic = arithmetic mean of actual citation impact values; Geometric = geometric mean of actual citation impact values. Countries are ranked by mean ranking on these 3 indicators.
      We find that Denmark, The Netherlands, the United States, Switzerland, and Belgium are the only countries with consistently superior performance at all 3 WS percentiles (ie, values > 100). High research output countries such as China, Japan, and Germany rank in the lower half of the table. Conversely, Denmark, Switzerland, and Belgium seem to produce research with greater impact, despite the low volume of their research relative to that of the other countries.
      Table 5 shows the citation impact of different radiation therapy research domains according to the mean WS values. It shows that whereas review articles, and research focusing on combined modality treatment (eg, radiation and drug therapies), are highly cited and potentially will have a greater impact on practices of care, research into quality assurance and health services research may have less impact, with a small fraction of articles in the top centiles of cited radiation therapy articles.
      Table 5Presence of radiation therapy articles in the 10 research domains in the top citation centiles, and overall ranking on basis of mean WorldScale (WS) values
      DomainCitableTop 5%Top 2%Top 1%WS 5%WS 2%WS 1%Mean
      REVS273325610459189191216199
      COMB6435452227135142177210176
      FRAC565034415488123137156138
      PRED894460120794135116105119
      PALL175896361711010397103
      BIOL79754701597111910089103
      PHYS47082528832108946890
      QUAL1611842911105906888
      HESR7272610372694161
      ASSU26607118854343039
      Abbreviations as in Table 2.
      Numbers of publications with 137 citations (top 1%), 86 citations (top 2%), 50 citations (top 5%), in 5 years after publication. WS = WS value (ratio of percentages of a country's publications in the top x% (1%, 2%, and 5%) relative to percentages of all worldwide publications in the top x% multiplied by 100). Mean is average of WS values.

       Open access articles

      Figure 4 presents a breakdown of the changing proportion of radiation therapy research articles available through the WoS that are closed (ie, behind a pay wall), gold open access (all freely available from the publisher), or green open access (available from the author's archive). The proportion of open access articles has continued to increase since 2001. Green open access articles are the best cited: the mean ACI for 2010-2012 articles was 19.7 citations in 5 years, compared with 15.7 citations for gold open access and 12.3 citations for closed articles.
      Figure thumbnail gr4
      Fig. 4Changing proportion of closed and open access articles over time, 2001-2015.

      Discussion

      This analysis of the global research landscape is the first to characterize the output of radiation therapy research globally and to identify trends in research priorities and contributions by individual countries. We found a doubling in overall research output over a 15-year period, consistent with trends observed in other disciplines (
      • Belter C.W.
      Bibliometric indicators: Opportunities and limits.
      ). The globalization of research has been a major contributor to this increase in output. Although the United States was responsible for more than one-third of the radiation therapy research output between 2001 and 2015, there was only a 55% increase in research output in the United States during that period, compared with more than 2000% in China.
      Significant increases in research output were also seen in several other middle-income countries, including India, Brazil, Turkey, and Iran, although their overall contribution to total worldwide radiation therapy research remained small. In this regard, it is important to acknowledge the issues related to access to radiation therapy in the majority of low- and middle-income countries (LMICs) (
      • Atun R.
      • Jaffray D.A.
      • Barton M.B.
      • et al.
      Expanding global access to radiotherapy.
      ), which undoubtedly will have important implications on their ability to influence the research portfolio.
      Furthermore, when these countries' research output is compared with their GDP, they still lag some way behind the major high-income countries. This is likely to be influenced by differences in cancer research funding between LMICs and high-income countries. We know that only 2.7% of total global cancer research investment is directly relevant to LMICs (
      • Sullivan R.
      • Purushotham A.
      Avoiding the zero sum game in global cancer policy: Beyond 2011 UN high level summit.
      ). Of that investment, the majority of cancer research funding is directed to studies that focus on cancer biology and drug development, rather than radiation therapy–related research (
      • Aggarwal A.
      • Lewison G.
      • Idir S.
      • et al.
      The state of lung cancer research: A global analysis.
      ,
      • Sullivan R.
      • Ginsburg O.
      • Aggarwal A.
      Research.
      ).
      It is important that LMICs are supported and encouraged to participate in research because of the need to continue to develop cost-effective treatment pathways, which can also meet key goals such as equity and efficiency within the constraints of their health system (
      • Rodin D.
      • Aggarwal A.
      • Lievens Y.
      • et al.
      Balancing equity and advancement: The role of health technology assessment in radiotherapy resource allocation.
      ). The wider research community will also benefit from a more globally inclusive research base, given inherent differences in cancer epidemiology and biology related to risk factor exposure. In the European Union for instance, limitations in the size (by population) and resources of individual countries is compensated by strong international collaboration, with the European Organization for Research and Treatment of Cancer (EORTC) acting to coordinate radiation therapy research in the region.
      It may be argued that with improved survival outcomes and more people living with cancer (
      • Jemal A.
      • Ward E.M.
      • Johnson C.J.
      • et al.
      Annual report to the nation on the status of cancer, 1975-2014, featuring survival.
      ), there should be greater attention in research to the domains of quality of life, health services, and palliative care. However, these areas represent only 4%, 2%, and 5% of total radiation therapy research outputs, respectively, over the study period. This is in stark contrast to other clinical research domains, such as preparation and delivery (24%) and dose fractionation studies (15%). Quality of life research seems particularly low given the dominance of clinical studies involving patients and the importance of this as a clinical endpoint.
      This discrepancy between the health needs of the population and current research priorities was also found in other cancer studies. In a bibliometric analysis, Sanson-Fisher et al (
      • Sanson-Fisher R.
      • Bailey L.J.
      • Aranda S.
      • et al.
      Quality of life research: Is there a difference in output between the major cancer types?.
      ) found that there were 4 times as many publications on chemotherapy in 2005 as on quality of life research, despite steady increases in the latter over the preceding 20 years.
      Health services research encompasses a broad multidisciplinary area addressing issues related to access, equity, and the value of health care. This branch of research helps to define new research priorities. It also aids effective implementation and sustainability of new innovative processes of care given the health system constraints—financial, political, or geographic—of a particular country. However, despite its importance, this area of research still lags far behind more established clinical and basic science research domains. It may not be considered to have the same value or relevance as other domains, as shown by its relatively low citation impact. Furthermore, this is an interdisciplinary subject area and requires collaboration between radiation oncologists and social scientists, such as health economists and epidemiologists.
      However, this view may finally be changing because of growing fiscal constraints that affect both high-income countries and LMICs and an increasing focus on value-based frameworks within medicine (
      • Hahn C.
      • Kavanagh B.
      • Bhatnagar A.
      • et al.
      Choosing wisely: The American Society for Radiation Oncology's top 5 list.
      ,
      • Chen R.C.
      Comparative effectiveness research in oncology: The promise, challenges, and opportunities.
      ). One example is the Health Economics in Radiation Oncology project under the auspices of the European Society of Radiation Oncology, which seeks to address the shortfall in applied research in this area (
      • Lievens Y.
      • Grau C.
      Health economics in radiation oncology: Introducing the ESTRO HERO project.
      ).
      The low commitment of several countries to palliative care research shown in Table 3 may be the result of the relative lack of senior academic appointments in this domain. This may limit opportunities for cross-sectoral research collaboration and for the attraction of research funding and the creation of research infrastructure (
      • Larkin P.J.
      • Murtagh F.
      • Richardson H.
      • et al.
      Collaboration: Securing a future for palliative care research.
      ). Norway demonstrated the strongest commitment to palliative care research, relative to other priorities. This is likely to reflect the country's strong support for palliative care at all levels of the public health care system (
      • Kaasa S.
      • Jordhoy M.S.
      • Haugen D.F.
      Palliative care in Norway: A national public health model.
      ).
      The predominance of clinical research outputs in our study is also a likely reflection of the particular time period in which these analyses were undertaken, in which significant technical improvements have been made (intensity modulated radiation therapy, particle therapy, motion management) that have sought to reduce the morbidity associated with treatment. In addition, large clinical trials in this era, such as those assessing multimodality therapy (
      • Bonner J.A.
      • Harari P.M.
      • Giralt J.
      • et al.
      Radiotherapy plus cetuximab for squamous-cell carcinoma of the head and neck.
      ) and different dose fractionation schedules (
      • Haviland J.S.
      • Owen J.R.
      • Dewar J.A.
      • et al.
      The UK Standardisation of Breast Radiotherapy (START) trials of radiotherapy hypofractionation for treatment of early breast cancer: 10-year follow-up results of two randomised controlled trials.
      ), have been a natural consequence of the preclinical research performed in the 1980s and 1990s, which had a much greater biological focus, with studies addressing dose per fraction, hypoxia, and drug–radiation interactions.
      Review articles were the most highly cited research domain, which suggests that evidence syntheses in the form of systematic reviews and meta-analyses have the potential to greatly influence practice, which is desirable. Research into multimodality therapy involving radiation therapy is also highly cited, most likely because of the large investment made in such studies (especially those involving pharmaceuticals) and the importance that these outputs have across the entire cancer spectrum from basic science to medical, surgical, and radiation oncology.
      Clinical trial publications accounted for 5% of total research output during the study period. The low overall proportion of publications related to clinical trials also points to a more worrying trend within radiation therapy research. This concerns the level of evidence required to integrate new processes of care and technologies into treatment, which remains reliant on small-scale observational studies and, more recently, modeling studies (
      • Lievens Y.
      • Borras J.M.
      • Grau C.
      Cost calculation: A necessary step towards widespread adoption of advanced radiotherapy technology.
      ,
      • van Loon J.
      • Grutters J.
      • Macbeth F.
      Evaluation of novel radiotherapy technologies: What evidence is needed to assess their clinical and cost effectiveness, and how should we get it?.
      ). Although there are constraints to conducting phase 3 trials in radiation therapy, increasing concerns about the value of new innovations means that investment in trials is required (public or private) to ensure new modalities are evaluated with rigorous methods so as to enable cost-effectiveness analyses to support their widespread implementation (
      • Zietman A.
      • Ibbott G.
      A clinical approach to technology assessment: How do we and how should we choose the right treatment?.
      ). Of even greater concern is that once available in the market following US Food and Drug Administration and European Union approval, few phase 4 studies are subsequently undertaken, even within centers that are early adopters of a new technology (
      • Lievens Y.
      Access to innovative radiotherapy: How to make it happen from an economic perspective?.
      ). Pragmatic research designs, such as multicenter observational cohort studies or nationally coordinated coverage with evidence development schemes, are alternative approaches that have been considered for evaluating the effectiveness of treatment in the real-world setting (
      • van Loon J.
      • Grutters J.
      • Macbeth F.
      Evaluation of novel radiotherapy technologies: What evidence is needed to assess their clinical and cost effectiveness, and how should we get it?.
      ,
      • Burock S.
      • Meunier F.
      • Lacombe D.
      How can innovative forms of clinical research contribute to deliver affordable cancer care in an evolving health care environment?.
      ).
      A number of interesting observations are offered in Table 3, which looks more closely at phase 1 to 3 trial outputs from individual countries. European countries such as the United Kingdom, The Netherlands, Sweden, and Poland produce a much higher ratio of phase 3–related trial outputs relative to phase 2. India also demonstrated a greater commitment to phase 3 relative to phase 2 trials, compared with several high-income countries, where the reverse trend was observed. For example, the United States, Japan, and Italy had a significantly higher proportion of phase 2 study outputs relative to phase 3. It is not clear why such differences are apparent, but they may relate to economic and cultural factors. For example, whether the necessary infrastructure or funding is available to conduct radiation therapy research trials may be one factor influencing these figures. Countries may vary in the level of evidence required by health care reimbursement organizations before they will routinely fund new technologies or practices of care. In addition, organizations such as the US Food and Drug Administration require demonstration of safety, rather than efficacy, within phase 3 studies, before approving new technologies, which has a downstream effect on the types of research evidence likely to be generated before clinical adoption.
      Although the United States was the largest contributor to randomized controlled trial publications, the proportion of its total radiation therapy research output devoted to randomized controlled trials was significantly less than that in other countries, such as India. This may reflect the recent trend of pharmaceutical and medical technology companies to conduct trials in countries where the personnel costs are lower and where the large pool of potential research participants can accelerate recruitment (
      • Glickman S.W.
      • McHutchison J.G.
      • Peterson E.D.
      • et al.
      Ethical and scientific implications of the globalization of clinical research.
      ). There may be many benefits that accrue to these countries from clinical trials research, such as the opportunity for international collaboration, investments in healthcare infrastructure, and the redirection of research priorities toward locally relevant and feasible interventions. However, concerns have also been raised about whether there is adequate transparency and oversight of human subjects in these countries, which may have weak regulatory systems and limited experience in research (
      • Glickman S.W.
      • McHutchison J.G.
      • Peterson E.D.
      • et al.
      Ethical and scientific implications of the globalization of clinical research.
      ).
      In this study we used 3 different citation scores based on actual citation counts to rank countries on the quality and importance of their published outputs. Although China, Japan, France, and Germany were high-output countries in terms of the number of research publications, their citation performance was significantly lower than that of several other countries that had lower research outputs (eg, Denmark). This may be partly because their articles tended to be published in low-impact journals or related to the language in which they were published. In that regard, a study of the impact of publication language on citation frequency in the scientific dental literature showed that articles published in English had a 6-7 times higher chance of being cited than articles published in German or French (
      • Poomkottayil D.
      • Bornstein M.
      • Sendi P.
      Lost in translation: The impact of publication language on citation frequency in the scientific dental literature.
      ).
      Furthermore, research articles from middle-income countries are poorly cited. It is unclear whether this relates to the perceived quality or level of interest in research from these countries or the low impact factor of the journals these studies are published in. This needs to improve to encourage middle-income countries to become more involved in radiation therapy research and influence practices of care. If not, practices of care may risk becoming regionally entrenched or influenced by a few select countries in North America and Europe, which could result in distinct knowledge gaps in the empirical literature.
      International collaboration is one mechanism by which this situation can be improved and prevent unnecessary duplication of research, to ensure the best available evidence is used to drive radiation therapy practice. One could consider a regional approach to identifying gaps in the evidence base and undertake relevant clinical and nonclinical studies, for example, through pre-existing regional alliances such as the Regional Office for the East Mediterranean and the Pan American Health Organization. In addition, the International Atomic Energy Association continues to support and coordinate multinational clinical radiation therapy trials (www-naweb.iaea.org/nahu/ARBR/crp.html) that have impacted on clinical practice (
      • Rosenblatt E.
      • Jones G.
      • Sur R.K.
      • et al.
      Adding external beam to intra-luminal brachytherapy improves palliation in obstructive squamous cell oesophageal cancer: A prospective multi-centre randomized trial of the International Atomic Energy Agency.
      ).
      Our analysis of open access articles demonstrates that they are steadily increasing over time, representing nearly 40% of articles currently published in the WoS either as gold or green open access. Although the overall citation impact of more recent open access articles (2010-2012) is higher than that of closed articles, this varies depending on country of origin and the type of radiation therapy research undertaken.
      The present study must be considered in the context of its strengths and limitations. The analysis has been undertaken on an individual country basis, and findings with respect to country outputs are potentially skewed depending on the size (by population) and resources of individual countries. However, this reflects the reality of their research strengths and weaknesses. Outputs relative to each individual country's GDP and research impact have also been presented. In addition, European countries that conduct a number of collaborative multinational studies may seem to produce comparatively less trial outputs when using fractional counts compared with the United States, for instance. A regionally based analysis may be one mechanism for addressing this in the future. We have used citation frequency as a proxy indicator for quality of research and dissemination of scientific findings. However, a true evaluation of the scientific quality of publications cannot be achieved without an independent and dedicated assessment of their merit. Furthermore, citation frequency cannot determine whether a publication changes practice and improves population health (
      • Belter C.W.
      Bibliometric indicators: Opportunities and limits.
      ).
      We have not provided a detailed analysis of the factors that have led to the observed trends and can only hypothesize potential reasons at this stage. The quantity of research outputs may be affected by publication bias, with failure of up to 20% to 30% of trials to report their results (
      • Jones C.W.
      • Handler L.
      • Crowell K.E.
      • et al.
      Non-publication of large randomized clinical trials: Cross sectional analysis.
      ). Equally excellent research may not be published or published elsewhere. This will have an impact on country-level integer and fractional counts, as well as on potential underrepresentation of clinical research outputs.
      We have selected publications available in the WoS for analysis, and it is therefore likely that some research output in national language journals has not been included, which could affect our results for country-level outputs. In addition, as with any bibliometric evaluation, it is not possible to guarantee inclusion of all relevant articles. However, attempts to minimize this have been sought by undertaking several iterations to develop the precision of our search filters to ensure inclusion of articles that have a relevant title word or Medical Subject Headings terms specific for clinical and basic science articles in radiation therapy. Although the WoS has selection criteria for the inclusion of journals based on repute and citation, it is unknown what proportion of low-quality “predatory” open access articles is included and the impact this has had on the estimations of total radiation therapy research output (
      • Shen C.
      • Björk B.C.
      ‘Predatory’ open access: A longitudinal study of article volumes and market characteristics.
      ). Finally, although our coding scheme for research types was made as explicit as possible, it is possible that some publications were miscategorized or that not all publications could be categorized according to the selected domains.

      Conclusions

      To conclude, our findings provide a detailed analysis of trends in radiation therapy research since 2001. Although there has been a doubling of radiation therapy research outputs over the study period, significant variation exists in the research output of individual countries, with evidence that radiation therapy research output is falling behind that of other cancer-related research domains. Although LMICs such as India, Iran, China, and Brazil continue to increase their radiation therapy research output, this still lags behind what is expected given their economic strength. Greater support is required to develop the necessary infrastructure to support high-quality research in LMICs that will contribute to the development of the specialty overall but allow the essential upscaling of radiation therapy resources in these countries.
      When considering the radiation therapy research types, there is evidence of individual countries' being committed to particular domains that reflect national cultures and economies. A major concern remains the very low proportion of trial-related publications within radiation therapy. This is an area that requires greater investment if we are to try and establish the relevant evidence base to promote the implementation of the most cost-effective, high-value care. In its absence, there will remain a lack of transparency as to the comparative benefits of innovations relative to existing treatments, especially given that the bulk of research focuses on clinical domains. To this end, greater focus on quality of life studies is required: despite the clinical predominance of most research output, few articles considered these endpoints. In addition, it is unknown what impact the slowing down of basic science research outputs over the study period means in the long term, with respect to identifying new pathways for improving patient outcomes. Greater emphasis on health services research would provide robust evidence on translating clinical evidence into practice. Finally, although citation impacts do not necessarily reflect influences on practices of care, they provide some understanding of the degree of interest or quality of the article. Given differences across countries, this may suggest that research outputs from particular countries are considered to be of higher quality and potentially have a greater impact on influencing practices of care.

      Acknowledgments

      This work was supported through Research Council UK's R4HC- MENA Programme REF: ES/P010962/1.

      Supplementary Data

      References

        • Eckhouse S.
        • Lewison G.
        • Sullivan R.
        Trends in the global funding and activity of cancer research.
        Mol Oncol. 2008; 2: 20-32
        • Delaney G.P.
        • Barton M.B.
        Evidence-based estimates of the demand for radiotherapy.
        Clin Oncol (R Coll Radiol). 2015; 27: 70-76
        • Shack L.
        • Lu S.
        • Weeks L.A.
        • et al.
        Determining the need and utilization of radiotherapy in cancers of the breast, cervix, lung, prostate and rectum: A population level study.
        Radiother Oncol. 2017; 122: 152-158
        • Borras J.M.
        • Lievens Y.
        • Dunscombe P.
        • et al.
        The optimal utilization proportion of external beam radiotherapy in European countries: An ESTRO-HERO analysis.
        Radiother Oncol. 2015; 116: 38-44
        • Aggarwal A.
        • Lewison G.
        • Idir S.
        • et al.
        The state of lung cancer research: A global analysis.
        J Thorac Oncol. 2016; 11: 1040-1050
        • Wallner P.E.
        • Steinberg M.L.
        • McBride W.H.
        • et al.
        A fork in the road: Choosing the path of relevance.
        Int J Radiat Oncol Biol Phys. 2015; 92: 214-216
        • Zietman A.
        The future of radiation oncology: The evolution, diversification, and survival of the specialty.
        Semin Radiat Oncol. 2008; 18: 207-213
        • Lawler M.
        • French D.
        • Henderson R.
        • et al.
        Shooting for the moon or flying too near the sun? Crossing the value Rubicon in precision cancer care.
        Public Health Genomics. 2016; 19: 132-136
        • Derrick G.E.
        • Pavone V.
        Democratising research evaluation: Achieving greater public engagement with bibliometrics-informed peer review.
        Sci Public Policy. 2013; 40: 563-575
        • Hood W.W.
        • Wilson C.S.
        The literature of bibliometrics, scientometrics, and informetrics.
        Scientometrics. 2001; 52: 291
        • Lewison G.
        Definition of cancer research: Journals, titles, abstracts or keywords?.
        DESIDOC J Libr Inform Technol. 2011; 31: 333-339
        • Lewison G.
        The definition and calibration of biomedical subfields.
        Scientometrics. 1999; 46: 529-537
        • Waltman L.
        • van Eck N.J.
        Field-normalized citation impact indicators and the choice of an appropriate counting method.
        J Informetr. 2015; 9: 872-894
        • Huffman M.D.
        • Baldridge A.
        • Bloomfield G.S.
        • et al.
        Global cardiovascular research output, citations, and collaborations: A time-trend, bibliometric analysis (1999-2008).
        PloS One. 2013; 8: e83440
        • Thelwall M.
        Interpreting correlations between citation counts and other indicators.
        Scientometrics. 2016; 108: 337-347
        • Belter C.W.
        Bibliometric indicators: Opportunities and limits.
        J Med Libr Assoc. 2015; 103: 219-221
        • Atun R.
        • Jaffray D.A.
        • Barton M.B.
        • et al.
        Expanding global access to radiotherapy.
        Lancet Oncol. 2015; 16: 1153-1186
        • Sullivan R.
        • Purushotham A.
        Avoiding the zero sum game in global cancer policy: Beyond 2011 UN high level summit.
        Eur J Cancer. 2011; 47: 2375-2380
        • Sullivan R.
        • Ginsburg O.
        • Aggarwal A.
        Research.
        in: Jemal A. The cancer atlas. 2nd ed. American Cancer Society, Atlanta, GA2015
        • Rodin D.
        • Aggarwal A.
        • Lievens Y.
        • et al.
        Balancing equity and advancement: The role of health technology assessment in radiotherapy resource allocation.
        Clin Oncol (R Coll Radiol). 2017; 29: 93-98
        • Jemal A.
        • Ward E.M.
        • Johnson C.J.
        • et al.
        Annual report to the nation on the status of cancer, 1975-2014, featuring survival.
        J Natl Cancer Inst. 2017; 109: 1-22
        • Sanson-Fisher R.
        • Bailey L.J.
        • Aranda S.
        • et al.
        Quality of life research: Is there a difference in output between the major cancer types?.
        Eur J Cancer Care (Engl). 2010; 19: 714-720
        • Hahn C.
        • Kavanagh B.
        • Bhatnagar A.
        • et al.
        Choosing wisely: The American Society for Radiation Oncology's top 5 list.
        Pract Radiat Oncol. 2014; 4: 349-355
        • Chen R.C.
        Comparative effectiveness research in oncology: The promise, challenges, and opportunities.
        Semin Radiat Oncol. 2014; 24: 1-4
        • Lievens Y.
        • Grau C.
        Health economics in radiation oncology: Introducing the ESTRO HERO project.
        Radiother Oncol. 2012; 103: 109-112
        • Larkin P.J.
        • Murtagh F.
        • Richardson H.
        • et al.
        Collaboration: Securing a future for palliative care research.
        Palliat Med. 2016; 30: 709-710
        • Kaasa S.
        • Jordhoy M.S.
        • Haugen D.F.
        Palliative care in Norway: A national public health model.
        J Pain Symptom Manage. 2007; 33: 599-604
        • Bonner J.A.
        • Harari P.M.
        • Giralt J.
        • et al.
        Radiotherapy plus cetuximab for squamous-cell carcinoma of the head and neck.
        N Engl J Med. 2006; 354: 567-578
        • Haviland J.S.
        • Owen J.R.
        • Dewar J.A.
        • et al.
        The UK Standardisation of Breast Radiotherapy (START) trials of radiotherapy hypofractionation for treatment of early breast cancer: 10-year follow-up results of two randomised controlled trials.
        Lancet Oncol. 2013; 14: 1086-1094
        • Lievens Y.
        • Borras J.M.
        • Grau C.
        Cost calculation: A necessary step towards widespread adoption of advanced radiotherapy technology.
        Acta Oncol. 2015; 54: 1275-1281
        • van Loon J.
        • Grutters J.
        • Macbeth F.
        Evaluation of novel radiotherapy technologies: What evidence is needed to assess their clinical and cost effectiveness, and how should we get it?.
        Lancet Oncol. 2012; 13: e169-e177
        • Zietman A.
        • Ibbott G.
        A clinical approach to technology assessment: How do we and how should we choose the right treatment?.
        Semin Radiat Oncol. 2012; 22: 11-17
        • Lievens Y.
        Access to innovative radiotherapy: How to make it happen from an economic perspective?.
        Acta Oncol. 2017; 56: 1353-1358
        • Burock S.
        • Meunier F.
        • Lacombe D.
        How can innovative forms of clinical research contribute to deliver affordable cancer care in an evolving health care environment?.
        Eur J Cancer. 2013; 49: 2777-2783
        • Glickman S.W.
        • McHutchison J.G.
        • Peterson E.D.
        • et al.
        Ethical and scientific implications of the globalization of clinical research.
        N Engl J Med. 2009; 360: 816-823
        • Poomkottayil D.
        • Bornstein M.
        • Sendi P.
        Lost in translation: The impact of publication language on citation frequency in the scientific dental literature.
        Swiss Med Wkly. 2011; 141: w13148
        • Rosenblatt E.
        • Jones G.
        • Sur R.K.
        • et al.
        Adding external beam to intra-luminal brachytherapy improves palliation in obstructive squamous cell oesophageal cancer: A prospective multi-centre randomized trial of the International Atomic Energy Agency.
        Radiother Oncol. 2010; 97: 488-494
        • Jones C.W.
        • Handler L.
        • Crowell K.E.
        • et al.
        Non-publication of large randomized clinical trials: Cross sectional analysis.
        BMJ. 2013; 347: f6104
        • Shen C.
        • Björk B.C.
        ‘Predatory’ open access: A longitudinal study of article volumes and market characteristics.
        BMC Med. 2015; 13: 230

      Comments

      Commenting Guidelines

      To submit a comment for a journal article, please use the space above and note the following:

      • We will review submitted comments as soon as possible, striving for within two business days.
      • This forum is intended for constructive dialogue. Comments that are commercial or promotional in nature, pertain to specific medical cases, are not relevant to the article for which they have been submitted, or are otherwise inappropriate will not be posted.
      • We require that commenters identify themselves with names and affiliations.
      • Comments must be in compliance with our Terms & Conditions.
      • Comments are not peer-reviewed.