Points of View Interpretation of academic research results for cost-effectiveness evaluation

Printable PDF

Hodai Okada, Senior Researcher, Pharmaceutical Industry Policy Institute

SUMMARY

Starting from a survey on cost-effectiveness evaluation published in the British Medical Journal (BMJ) last year, we investigated factors that influence the interpretation of research results conducted using known information derived from databases and other sources. The results of a systematic review of academic papers on cost-effectiveness analysis in Japan suggested that three factors may influence the results: the purpose of conducting the study, the decision to publish the paper, and differences in the results of systematic analysis. Cost-effectiveness analysis is a research area that has seen an increase in papers in recent years, and as an industry, we need to continue our efforts to ensure that the results of the studies conducted are correctly interpreted by the public.

1. Introduction

Cost-effectiveness assessment has been attracting attention in recent years as research to optimize healthcare costs, and its results are beginning to be used in reimbursement decisions and drug price adjustments in several countries. In Japan, a cost-effectiveness evaluation system has been in operation since 2019, and the results of cost-effectiveness evaluations are used to adjust drug ^prices1). 1) Since cost-effectiveness evaluations are conducted using information that has been accumulated and reported by the time the study is conducted, it is known that uncertainty may arise in the results depending on the choice of information source and the accuracy of the numerical values of the parameters used in the analysis. In Japan's cost-effectiveness evaluation system, as in the case of applications for drug approval, the process involves not only evaluation of materials submitted by companies, but also reanalysis by a specialized organization based on the results submitted by the companies. The number of academic papers on cost-effectiveness evaluation

The number of academic papers on cost-effectiveness evaluation is increasing, and systematic reviews, such as those conducted in clinical research, have begun to integrate the results of cost-effectiveness evaluations conducted to date and systematically evaluate the results of cost-effectiveness evaluations of vaccination and decision support system use ^{2) 3)}. Last June, the BMJ, one of the leading medical journals, reported a survey on industry sponsorship and evaluation results in cost-effectiveness evaluation studies in the healthcare ^field4). The study integrated the results of cost-effectiveness analyses of academic papers reported between 1976 and 2021, and found that the estimated incremental cost-effectiveness ratios (ICERs) of industry-sponsored studies were about 33% lower than those of unsupported studies, and that the ICERs of unsupported studies were about 20% lower than those of industry-sponsored studies. The estimated ICER (incremental cost-effectiveness ratio) for company-supported research is approximately 33% lower than the ICER for unsupported research. The following three factors may be responsible for these results.

1 and 2 affect the presence or absence of published results and do not affect the interpretation of the results of individual assessments; 3 affects the actual content of the analysis and therefore the interpretation of the results of individual assessments; and 4 affects the results of the analysis and therefore the interpretation of the results of individual assessments. The pharmaceutical industry needs to continue its activities to ensure that the public correctly interprets the results of research conducted at great expense and time. As information to lead to correct interpretation of the results of cost-effectiveness evaluations published as academic papers, we will organize issues focusing on understanding the current status of information disclosure of cost-effectiveness analysis as academic papers in Japan. In this paper, we will discuss 1 and 2 above based on academic papers, and 3 based on reports on cost-effectiveness evaluation systems in Japan. By presenting this information, we hope to help advance the discussion on how to publish cost-effectiveness evaluations as academic papers in Japan. The three elements presented here are similar to issues that may arise in retrospective database studies that use information obtained in the past, and we would like to consider them not only as issues related to cost-effectiveness evaluation, but also as issues related to database studies in general.

Survey Methodology

2-1. survey of academic papers

In order to discuss "1. Purpose of conducting research" and "2. Decision-making of argument culture," we conducted a survey of academic papers. The survey covered academic papers published between 2015 and 2022 that were included in the CEA Registry or PubMed published by Tufts Medical Center, and which showed the results of cost-effectiveness analysis using pharmaceuticals (excluding vaccines) as the medical technology under study. Cost-effectiveness analysis is one of the analyses used in cost-effectiveness evaluation, and has been adopted as an analytical method in Japan's cost-effectiveness evaluation ^system5). From those papers, we extracted papers that presented the results of cost-effectiveness analyses conducted under the assumption of the Japanese healthcare system and using QALYs as the effectiveness index. By using PubMed in addition to the CEA Registry, which is commonly used for previously published review articles, we believe that a certain level of coverage was ensured. In the survey, we tabulated both the paper-by-paper and the analysis units included in the papers. In the survey of units of analysis, only analyses for the main results in the paper (excluding sensitivity analysis and scenario analysis) were included, and in cases where a single paper contained multiple main results (e.g., positions of multiple analyses, target populations, drugs, etc.), the correlation on the results was considered and up to the two analyses that had the greatest impact on the conclusions of the paper were The results were included in the tabulation.

2-2. survey of cost-effectiveness evaluation systems

In order to discuss "3. Differences in the results of systematic analyses," a survey of cost-effectiveness evaluation systems was conducted. In Japan, the cost-effectiveness evaluation system, which began operating in 2019, provides detailed reports of analysis results to the ^public6). Under the cost-effectiveness evaluation system, after a cost-effectiveness evaluation is conducted by a company that serves as a manufacturer or distributor, a review and reanalysis of the analysis conducted by the company is conducted by the National Institute of Health Sciences or a university as a public analysis. This paper investigates trends in the results for the target population for which cost-effectiveness analyses were conducted, both by the manufacturer and by the public analysis, based on reports published from the start of the system to September 2023.

Results

3-1. background information on the survey of academic papers

Eighty-four papers were selected for this study (Figure 1). The number of papers in which cost-effectiveness analyses were conducted on pharmaceutical products in Japan has remained at about 10 per year since 2017 (Figure 2). The classification of pharmaceuticals under study (ATC classification) is shown in Table 1. Since this paper will focus on information classified by studies with and without company involvement, a tabulation classified by the presence or absence of company involvement is also shown for reference. The most common categories were L-classified antineoplastics and immunomodulators, with 22 of the 30 applicable papers being studies on antineoplastics and 7 on immunosuppressive drugs. In the following J category, systemic anti-infectives, 11 of the 14 studies were on hepatitis C drugs, and the remaining 3 studies were on influenza drugs. In the next category, Category A, Gastrointestinal and Metabolic Effects, 4 of the 12 articles were on drugs for type 2 diabetes and drugs for gastrointestinal symptoms associated with antineoplastic agents, respectively.

A summary of the cost-effectiveness analyses is shown in Table 2. The number of analyses extracted from each paper was 111. Aggregation was performed on an analysis-by-analysis basis and on a paper-by-paper basis for settings where the conditions were equivalent for analyses within the same paper. Corporate involvement was defined as papers in which the funding item in the paper indicated that the paper received a grant from a company or papers in which the authors included employees of a pharmaceutical company. Research involving companies accounted for 54% of the total, of which 67% were foreign companies and 33% were domestic companies, indicating that foreign pharmaceutical companies have made a significant academic contribution to cost-effectiveness analysis in Japan. The position of the analysis was defined as "public health care position" for analyses conducted within the scope of the public health insurance system. All cases in which other costs such as long-term care were included were classified as "social standpoint" for tabulation. Eighty-one percent of the total studies analyzed from the public health care standpoint, with relatively few analyses from the social standpoint. There was some variation by research structure with respect to the percentage of studies using real world data (RWD) databases, such as receipt information, as the source of information for the parameters used in the model.

3-2. purpose of conducting research

One factor that may have the greatest impact on the interpretation of the results of the aforementioned survey on industry sponsorship and evaluation results is the purpose of conducting the study. Because of the cost and time required to conduct a cost-effectiveness evaluation study, it is difficult to conduct a study for every drug product on the market that maintains the quality needed to be submitted as a scientific paper. The background of research can be broadly classified into two categories: research aimed at disseminating information on the cost-effectiveness of newly launched drugs and research aimed at solving social issues where the pros and cons of cost-effectiveness are debated after the drug is marketed. The former type of research often involves pharmaceutical companies in order to provide the market with information on newly launched drugs, while the latter is often led by academic institutions, and this difference in the role of the implementing institution with respect to the research objective may have influenced the difference in ICERs in systematic reviews The following is an example of a cost-effectiveness analysis that assumes the Japanese health care system.

We conducted a study to confirm the above hypothesis based on a review of papers that included the results of cost-effectiveness analyses assuming the Japanese healthcare system. Since it is difficult to infer the exact purpose of the study from the descriptions in the papers alone, we use the time from the date of approval of the drug's indication to the publication of the paper as an alternative indicator. Figure 3 shows the distribution of research systems when classified by 4.5 years. As hypothesized, research conducted on new drugs soon after approval involved a large proportion of pharmaceutical companies, while research conducted long after approval involved a relatively small proportion of pharmaceutical companies. From this result, it can be inferred that a certain relationship exists between the purpose of the study and the system under which the study was conducted.

Next, we confirmed the relationship between the results of the cost-effectiveness analysis of each study and the period from the date of approval of the drug's target indication to the publication of the article. We identified papers that showed that the drug under study was extremely superior to existing drugs (Dominant: the drug under evaluation cost less and was equally or more effective compared to the comparator drug) and papers that showed extremely inferior results (ICER 10 million yen or more), and tabulated the publication dates of each of these papers, It was found that studies concluded to be Dominant had a median time from approval to publication of 2.9 years, and were often published earlier after approval, compared to a median time of about 5 years for the rest of the literature (Table 3). Considering the impact on the novelty of research and the probability of acceptance as an academic paper, it is not difficult to imagine that when considering a drug for research, a drug that has been publicly known to have excellent cost-effectiveness for a long period of time after approval is unlikely to be the subject of research, and the association between the results of cost-effectiveness analysis and the period from the date of approval is also a reasonable The results of the cost-effectiveness analysis and the time from the date of approval are also considered to be reasonable results. The above results suggest that industry and academia play different roles in academic research on cost-effectiveness, and that there may be a correlation between their research objectives and the results of cost-effectiveness evaluations. In the current situation where it is impractical to conduct evaluations for all drugs, it is possible that such correlations that could affect interpretation may arise from factors other than those caused by the research system. The hypotheses presented in this section do not affect the interpretation of the results of individual evaluations, but they do affect the trend of results when multiple articles are integrated for analysis, and thus require caution in interpretation.

3-3. decision to publish

A similar matter to the objective of conducting research is the decision to publish a paper after the research has been conducted. Particularly in pharmaceutical companies, cost-effectiveness evaluations are often conducted as internal documents before and after the launch of a drug, but in many cases they are internal reference materials, and the evaluation perspective and research quality are different from those of academic research. While there are some pharmaceuticals for which such information is kept as in-house information and not published, there are also pharmaceuticals for which the results of cost-effectiveness evaluations are published as academic papers, and the industry may need to consider the merits of the difference in treatment regarding the disclosure of such information. In order to investigate the involvement of outside researchers in the 45 papers involving companies among the papers surveyed in this study, Figure 4 shows the characteristics of the authors of the papers. Researchers from academic institutions were included as authors in 93% of the cost-effectiveness analyses that were academic papers. It can be inferred that analyses conducted solely by employees of pharmaceutical companies are less likely to be published, and that there is a strong possibility that there is some relationship between the participation of researchers from academic institutions and the decision to publish a paper.

Cost-effectiveness evaluations are often categorized as similar to database studies because they generally use existing information such as databases. The Medical Affairs Subcommittee of the Drug Evaluation Committee of the Pharmaceutical Manufacturers Association of Japan (PMAJ) has indicated a direction to avoid publication bias in database studies, and has proposed the registration of studies in jRCT and UMIN before the studies are ^conducted7) as a countermeasure. Consideration for publication after the study is conducted is not desirable, as in the case of database studies, and can lead to bias in the results of published studies and affect the trend of results when multiple articles are integrated for analysis, as well as the purpose of conducting the study as indicated in 3-2. Cost-effectiveness evaluations are often conducted in-house in general, and it will be necessary for the industry to consider whether it is desirable to have the same policy of information disclosure through publication as these database studies even when the novelty of the study is unclear.

3-4. differences in systematic analysis results

Up to the previous section, we have investigated cost-effectiveness evaluations published as academic papers. In order to investigate differences in the results of systematic analysis by research organizations, it is necessary to have information on analyses conducted by multiple organizations on the same research subject, but since such cases are rare in academic papers, 3-4 will be based on a case study of the Japanese cost-effectiveness evaluation system. In the cost-effectiveness evaluation system, after a company as a manufacturer or distributor conducts a cost-effectiveness evaluation, a review and reanalysis of the analysis conducted by the company are conducted by the National Institute of Public Health and Medical Sciences or a university as a public analysis. This paper compares the results of the analysis by the manufacturer and the reanalysis by the public analysis.Since the institutionalization of the system in 2019, reports of cost-effectiveness analyses in 53 target populations for 20 products have been published for pharmaceuticals. Initially, in the 36 target populations for which the results of cost-effectiveness analyses (excluding cost minimization analyses) from both institutions are publicly available among them, Figure 5 shows the distribution of which institution's point estimates of the analysis results showed better cost-effectiveness. Because of the nature of cost-effectiveness analysis, it is natural that differences in the results will occur, so we checked here not for differences in the results, but for the existence of bias in the trends of the differences that did occur. If there is no systematic difference between the institutions conducting the studies, it is assumed that the difference between the results of the two analyses would have a symmetrical distribution centered on 0. Reasons for these differences in results are also reported in the report and are broadly summarized in Table 4. The most common differences were related to the setting of QOL values, which were attributed to the different choice of sources of information for extrapolated parameters. In other matters, most of the differences were also in how the parameters of the model were set. Activities to reconcile these differences are also being conducted by the Pharmaceutical Manufacturers Association of Japan (PMAJ), and a report summarizing past findings has been ^prepared8). This result may be a systematic difference that arises from the special circumstances of the system's structure, in which a review and reanalysis by public analysis is conducted after the company that serves as the manufacturer and distributor conducts a cost-effectiveness evaluation, and the results of the analysis are used to evaluate adjustments in drug prices, and from these few cases, academic research It is difficult to conclude from these few cases that similar differences have occurred in academic research. However, we cannot deny the possibility that such events also occur in the results of studies published as academic papers. Given the limited number of published results in academic papers of cost-effectiveness analyses of pharmaceuticals conducted under the assumption of the Japanese healthcare system, it can be assumed that these differences will narrow as experience accumulates. In addition, the Medical Affairs Subcommittee document mentioned in the previous section also indicates the direction of ensuring reliability in database research, and it is assumed that this will be resolved along with the movement toward standardization of these ^processes7).

In addition, the Central Social Insurance Medical Council's Expert Committee on Cost-Effectiveness Evaluation, which is currently in place, is discussing whether or not the results of analyses in the cost-effectiveness system can be published as ^articles9). There are only a few cases in which the results of cost-effectiveness evaluations of drugs covered by this system have become academic papers in English. Since cost-effectiveness evaluation itself is based on information from MEDLINE and other literature, a mechanism to return the results obtained from this information to the academic community should also be considered.

Conclusion and Discussion

We summarized the current status of academic papers on cost-effectiveness evaluation and examples of cost-effectiveness evaluation systems in Japan, and discussed three factors that we believe influence the interpretation of the results of systematic reviews of academic papers on cost-effectiveness analysis in particular. Although the number of cases in this study was limited, it was suggested that all of the initially assumed factors of "purpose of conducting the study," "decision to publish," and "differences in systematic analysis results" may have influenced the results. In particular, since conducting and publishing cost-effectiveness evaluation studies requires a reasonable amount of money and time, there are restrictions on the publication of study results, and it is desirable that the results of the paper be interpreted with an understanding of the background of these current conditions and the purpose of conducting the study. In parallel, the pharmaceutical industry needs to provide the results of the conducted research in a form that is easier to interpret. It has been confirmed that the items indicated in this paper have already been discussed in various committees of the Pharmaceutical Manufacturers Association of Japan (PMAJ), and it is expected that more transparency in research will be achieved.

While there are limitations in conducting and publishing cost-effectiveness evaluation studies, it is important for both the pharmaceutical industry and academic institutions to play different roles and accumulate information in order to optimize the cost of medical care in Japan. Clinical trials conducted by the pharmaceutical industry are gradually increasing opportunities to measure health-related quality of life indicators, which also contribute to providing information for conducting model analyses. It is important to create an environment in which information can be interoperable when conducting research that uses existing information, and appropriate publication of the results of both clinical trials and cost-effectiveness analyses will lead to comprehensive development as a research area.

Database studies that use existing data, including studies of cost-effectiveness analysis that use previously reported figures for research, are highly predictive of trends in the results obtained even before the study is conducted, and it is difficult to completely eliminate these factors at the time of the decision to conduct the study or not. Under such constraints, the pharmaceutical industry is examining ways to increase the transparency of research from various perspectives, and it is hoped that certain guidelines will be created in this new area to ensure the reliability of the results of research conducted.

Return to News List

TOP