Newsletter Sign-Up

Keep up to date with NPC news and events by signing up for our e-newsletter E.V.I.dently, and our CER Daily Newsfeed.

Subscribe POLICY

NPC Comments on Revised PCORI Methodology Standards

March 28, 2016

Dr. Joe Selby
Executive Director
Patient-Centered Outcomes Research Institute
1828 L St., NW, Suite 900
Washington, DC 20036 

Dr. Robin Newhouse
Chair PCORI Methodology Committee
Attn: comments@pcori.org

Dear Methodology Committee Members:

We welcome the opportunity to comment on the new and revised Patient-Centered Outcomes Research Institute (PCORI) Methodology Standards published for comment on January 25, 2016. We believe that the methodological standards provide a fundamental base for evidence to improve and transform medical practice. These standards are an important element to guide researchers in the conduct of high-quality research, for decision-makers to distinguish credible and valid research, and for patients and other stakeholders to identify research that is trustworthy and useful.

The National Pharmaceutical Council (NPC), a health policy research organization dedicated to the advancement of good evidence and science and to fostering an environment in the United States that supports medical innovation, firmly believes that research should be conducted in a rigorous and transparent manner using common and agreed-upon methods. Founded in 1953, and supported by the nation’s major research-based pharmaceutical companies, NPC and the NPC member companies include a wealth of scientific and research expertise. We are pleased to offer comments regarding the latest revision and future opportunities for PCORI’s leadership on research that is useful for patient and provider decision-making.

Overall Comments:

Actionable and Measurable Standards:

The current version of the PCORI Methodology Standards identifies a minimum set of practices for conducting comparative effectiveness research (CER). However, it can be difficult if some standards are not “actionable” or “measurable.” For example, IR-5 specifies that researchers “provide sufficient information in reports to allow for assessments of the study’s internal and external validity,” but the criteria for “sufficient” remains in the eye of the researcher and the reader. In contrast, MD-2, which addresses statistical methods for dealing with missing data, outlines which methods are considered valid and which methods are discouraged. Actionable and measurable standards provide a target for decision-makers to judge the usability of the evidence produced and for researchers to aim.

Recommendation: We recommend the Methodology Committee engage in a review of the current methods standards to outline the criteria required to meet each standard. For standards in which there are not clear requirements, examples of ways in which the standards would be fulfilled would benefit all stakeholders.

Standards Associated with Patient-Centeredness:

We applaud the Committee on clarifying how patients and stakeholders should be involved in the prioritization of research, conduct of research, and the dissemination of research findings. As outlined in the standards, a broad set of stakeholders, including the biopharmaceutical industry, contribute to a more robust research process. Purchasers, payers, and industry communities seek to ensure that relevant questions are addressed, research findings are usable, and results are translated and implemented in practice.1 This engagement is an important aspect toward improving the relevance of CER questions and ultimately the impact of CER on health care decision-making.2

Standards for Data Integrity and Rigorous Analysis:

We commend the Committee on the recognition of the growing body of research standards and the need to increase research plan and analytic transparency. These are important steps forward. However, we believe there are other opportunities for the PCORI Methodology Committee to play a pivotal role in the nation’s research enterprise.

We commend the broader recognition of standards such as STROBE (for observational research) and SRQR and COREQ (for qualitative research). As outlined in the Affordable Care Act, the methodology standards “shall build on existing work on methodological standards for defined categories of health interventions and for each of the major categories of comparative clinical effectiveness research methods.” Good practices identified by professional societies and consortia such as RECORD (for observational studies), the GRACE Checklist (for observational, the CER Collaborative (for observational studies, indirect treatment comparisons, and modeling studies), CENT (for N-of-1 trials), CONSORT Extension (for pragmatic trials), and others are not referenced in this version of standards.3, 4 ,5 ,6 ,7, 8 ,9 Because stakeholders are increasingly looking to PCORI as a leader in the establishment of standards for CER, broader inclusion and recognition of good research practices is needed. 2

Recommendation: We recommend that additional methods or standards developed by other groups be considered and at a minimum referenced to increase stakeholder awareness or adopted in the PCORI methods.

Over the past decade, standards for research have proliferated among different research disciplines (e.g., biostatistics, pharmacovigilance, econometrics). This proliferation can increase the reach of standards across the types of research. However, it can also have unintended consequences. For example, a recent study compared and contrasted nine existing sets of standards or guidelines (based on 23 elements) for conducting observational studies. While most guidelines standards agreed on what elements were important (e.g., the need for a study protocol), there was disagreement 52% of the time on how the standards should be acted upon and addressed.10 This disagreement can contribute to variation in study quality, and create discrepancies adopted into care decisions. There is a need to identify common and agreed-upon methods for research through consensus-based approaches. An ongoing process enables agreement where quicker consensus is feasible and an iterative process for new, novel, or controversial methods.

Recommendation: Stakeholders perceive PCORI to be a leader in establishing CER standards. Few other organizations have the access to research experts from a variety of research disciplines and communities or experience in facilitating multi-stakeholder processes for prioritization as PCORI. We commend the Committee on the advancement of the standards. We believe the Committee and PCORI can offer their leadership to advance a set of common and agreed-upon methods through a consensus-based process that will form the foundation for research that the public can use and trust.

Finally, we commend the Methodology Committee on the recognition of the importance of research transparency. Without insight into which outcomes were pre-specified and the research analysis process used, the validity of study results may be questioned. The efforts to register studies before study start at clinicaltrials.gov and other sites seek to increase research transparency.

Recommendation: Some sites such as clinicaltrials.gov were developed for clinical trials. Although used for observational studies as well, many modifications are needed for these sites to account for features relevant for observational studies (e.g., propensity methods to balance confounding, analytic definitions to identify exposures). PCORI has an opportunity to collaborate with other entities such as the National Institutes of Health to enhance existing research infrastructure which encourages transparent, reproducible, and credible research.

We offer these suggestions to improve the rigor, transparency, and conduct of research to inform patient and provider decision-making. We propose the development of more actionable and measurable standards, broader inclusion of all research methods, and a consensus-based process to identify a common and agreed-upon set of methods for the field of CER. The National Pharmaceutical Council looks forward to further dialogue and engagement with the PCORI Board, Methodology Committee, and staff.

Sincerely,

Jennifer S. Graff, PharmD
Vice President, Comparative Effectiveness Research
National Pharmaceutical Council

Robert W. Dubois, MD, PhD
Chief Science Officer
National Pharmaceutical Council

---------------------------------------------------------------------

  1. Concannon TW, Khodyakov D, Kotzias V, Fahey G, Graff J, Dubois RW. Employer, Insurer, and Industry Perspectives on Patient-Centered Comparative Effectiveness Research: Final Report. Santa Monica, CA: RAND Corporation, 2015. http://www.rand.org/pubs/research_reports/RR1242.html.
  2. 2016 Comparative Effectiveness Research and the Environment for Health Care Decision-Making. March 2016. Available at: http://www.npcnow.org/system/files/research/download/NPC_ChartPack_v04_highres.pdf.
  3. Benchimol EI, Smeeth L, Guttmann A, et al. RECORD Working Committee. The REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) Statement. PLoS Med. 2015;12(10):e1001885.
  4. Dreyer NA, Velentgas P, Westrich K, Dubois R. The GRACE Checklist for Rating the Quality of Observational Studies of Comparative Effectiveness: A Tale of Hope and Caution. J Manag Care Pharm;2014:20(3):301-8.
  5. Berger M, Martin B, Husereau D, et al. A questionnaire to assess the relevance and credibility of observational studies to inform healthcare decision making: an ISPOR-AMCP-NPC good practice task force report. Value Health. 2014; 17:143-156.
  6. Caro JJ, Eddy DM, Kan H, et al. A modeling study questionnaire to assess study relevance and credibility to inform health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force report. Value Health. 2014;17:174–82.
  7. Jansen J, Trikalinos TA, Cappelleri JP, et al. Indirect treatment comparison/network meta-analysis study questionnaire to assess study relevance and credibility to inform health care decision-making: an ISPOR-AMCP-NPC Good Practice Task Force report. Value Health 2014;17:157–173.
  8. Vohra S, Shamseer L, Sampson M, et al. CENT group. CONSORT extension for reporting N-of-1 trials (CENT) 2015 Statement. BMJ. 2015;350:h1738.
  9. Zwarenstein M, Treweek S, Gagnier JJ, et al. CONSORT group; Pragmatic Trials in Healthcare (Practihc) group. Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ. 2008;337:a2390.
  10. Morton C, Costlow R, Graff JS, Dbois RW. Standards and guidelines for observational studies: quality is in the eye of the beholder. J of Clin Epi. 2016;71;3-10.
Newsletter Volume

E.V.I.dently January 2016

MessageThe Importance of Evidence, Value, Access and Innovation in 2016In his annual health care outlook, NPC President Dan Leonard explains why...
Newsletter Volume

E.V.I.dently November 2015

MessageCan We Work Together to Ensure Patient Access to Treatments and Encourage Innovation?Has biopharmaceutical innovation brought value and made a...
Newsletter Volume

E.V.I.dently October 2015

MessageWhat’s the Fuss About Evidence From Clinical Experience? Here’s Why It MattersAt a health care forum in Washington, DC, earlier this month,...
Newsletter Volume

E.V.I.dently September 2015

MessageValue Is in the Eye of the BeholderIn recent months, a number of health organizations have developed “value assessment frameworks,” which are...
Press Release

NPC Highlights Stakeholder Views on Comparative Effectiveness Research and the Environment for Health Care Decision-Making

(Washington, DC, March 26, 2015)—A new survey of health care stakeholders reveals continued optimism that comparative effectiveness research (CER)...
Press Release

NPC Annual Stakeholder Survey: Comparative Effectiveness Research Is Important, But Impact on Health Care Decision-Making Is Still on the Horizon

(Washington, DC, May 19, 2014)—A new survey of health care stakeholders reveals continued optimism for the use of comparative effectiveness research...
Press Release

CER Collaborative Improving Health Outcomes Through New Comparative Evidence Tools

(Washington, DC, March 24, 2014)—Three peer-reviewed articles published in Value in Health offer important guidance to aid formulary decision-makers...
Press Release

National Pharmaceutical Council Statement on Appointment of PCORI Board Chair

Washington, DC, September 19, 2013—The National Pharmaceutical Council (NPC) today released the following statement regarding the Government...
Commentary and Testimony Page

NPC Comments on PCORI Proposal for Peer Review of Primary Research and Public Release of Research Findings

November 7, 2014Dr. Joe SelbyExecutive DirectorPatient-Centered Outcomes Research Institute1828 L St., NW, Suite 900Washington, DC 20036Dear Dr....
Commentary and Testimony Page

The State of CER and the Environment for Health Care Decision Making

This month, NPC released its third annual survey on stakeholders’ views on comparative effectiveness research (CER), “The State of Comparative...
Commentary and Testimony Page

Keeping Our Eye on the CER Ball

Until the cherry blossoms begin to bloom later next month, most of the water cooler conversations in Washington will focus on the federal budget....
Commentary and Testimony Page

Leveling the Playing Field? Could a Landmark Decision Alter the CER Communications Landscape?

A growing chorus of voices is publicly sharing data on the effectiveness of treatment options, but one viewpoint has long been shut out of key areas...
YouTube Videos

Use of Big and Real-World Data by Pharma: More than Data Warehousing

Real-world data—a subset of big data—is clinical data used for decision-making that did not come from a clinical trial. According to Aaron Galaznik,...
YouTube Videos

Engaging Patients in Research

National Pharmaceutical Council President Dan Leonard explains why it's important to include patients throughout the research process. He says that...
YouTube Videos

Engaging Patients in the Research Process

Dr. Dan Mullins, professor of pharmaceutical health services research at the University of Maryland School of Pharmacy, explains why it is important...
YouTube Videos

Patient Input on What Makes Comparative Effectiveness Research Useful

Myrl Weinberg, chief executive officer of the National Health Council, says that many patients are still concerned about how comparative...
Blog Post

Researchers Consider When Evidence Is Fit for Purpose at Concordium Meeting

Coming up at AcademyHealth’s inaugural Concordium 2015 meeting, look for Jennifer Graff, PharmD, NPC’s vice president for comparative effectiveness...
Blog Post

CER Tweets of the Week (May 18-22): CER Stakeholders Tweet Discussions From #ISPORPhila; Generalizing RCT Results to Wider Populations

This week, many comparative effectiveness research (CER) stakeholders attended the International Society for Pharmacoeconomics and Outcomes Research...
Blog Post

Filling Gaps in Evidence: A Tale of Two (Retrospective) Studies

In his latest column in the Journal of Comparative Effectiveness Research, NPC Chief Science Officer Robert W. Dubois, MD, PhD, poses critical...
Blog Post

#TBT: 5 Years Later—Comparative Effectiveness Research Survey & the ACA

As part of our “Throwback Thursday” blog series, we’re taking a look at a topic that’s currently in the news and tagging it with previous...
cersurvey16

2016 Comparative Effectiveness Research and the Environment for Health Care Decision-Making

NPC's sixth annual survey of stakeholder views on comparative effectiveness research (CER) and the...
cersurvey-15

2015 Comparative Effectiveness Research and the Environment for Health Care Decision-Making

NPC's fifth annual survey, "Comparative Effectiveness Research (CER) and the Environment for Health...
  •  
  • 1 of 5
  • >