FDA's Best Practices for Conducting and Reporting Pharmacoepidemiologic Safety Studies Using Electronic Healthcare Data Sets

Re: Docket No. FDA-2011-D-0057: Best Practices for Conducting and Reporting Pharmacoepidemiologic Safety Studies Using Electronic Healthcare Data Sets

Dear Sir/Madam:

The Biotechnology Industry Organization (BIO) thanks the Food and Drug Administration (FDA) for the opportunity to submit comments on the “Draft Guidance for Industry and FDA Staff on Best Practices for Conducting and Reporting Pharmacoepidemiologic Safety Studies Using Electronic Healthcare Data Sets”. BIO supports the Agency’s efforts to fully utilize new electronic health care data sources for post-approval pharmacoepidemiologic studies. These new data resources offer great promise to revolutionize the practice of pharmacovigilance with more timely and cost-effective methods for conducting post-market studies, but great care must be taken to minimize the potential for confounding and bias. We hope the Draft Guidance will provide additional transparency in scientific exchange between FDA in Sponsors when initiating appropriate pharmacoepidemiological studies in a regulatory context. With this goal in mind, we request clarification of several aspects of the Draft Guidance, particularly around the process for submitting and reviewing study protocols.

BIO represents more than 1,100 biotechnology companies, academic institutions, state biotechnology centers and related organizations across the United States and in more than 30 other nations. BIO members are involved in the research and development of innovative healthcare, agricultural, industrial and environmental biotechnology products, thereby expanding the boundaries of science to benefit humanity by providing better healthcare, enhanced agriculture, and a cleaner and safer environment.

BIO commends FDA for developing this comprehensive document to identify pharmacoepidemiologic safety study best practices, consistent with the PDUFA IV commitment. BIO also thanks the Agency for reviewing our previous comments on this topic. The scope of this Guidance is well-circumscribed and the scientific recommendations are consistent with best expected practices and state of the art in the field. The purpose, as explicitly stated in the introduction (lines 20-23 and 33), is to ultimately optimize FDA’s review of protocols and final reports that are submitted to the Agency for this type of study. FDA has wisely narrowed the focus of this Guidance to apply specifically to the use of electronic healthcare information, which differentiates it from already well-established but more general best practice guidance in the field (FDA 2005 guidance, International Society of Pharmcoepidemiology (ISPE) guidelines and STROBE (STrengthening the Reporting of OBservational studies in Epidemiology)).

From this perspective, there are several recommendations that BIO member companies can provide on the Guidance structure and content that are based on our experience with the process of designing and implementing pharmacoepidemiologic studies that have been reviewed by the Agency in the past. There are also several statements that are found to warrant additions or further clarifications, which are mentioned in this response.

GENERAL COMMENTS:

I. The Role of Pharmacoepidemiological Studies in Determining Causality, Association, and Magnitude of Effect

The emphasis throughout the document should be that the goal of these studies is to identify potential causal associations of drugs and estimate their magnitude. Therefore, when reporting any finding the extent to which it can be interpreted as a causal effect should be discussed. The document should also more clearly distinguish between causal effects and associations. For example, in database studies the validation of drug exposure is limited, and hence, attributing outcomes to drug exposure based on an association between prescription claims and medical claims can be misleading. Additional research efforts in this area are needed. In the Introduction of the Draft Guidance, we suggest a change from “to assess the risk attributed to a drug exposure” to “to assess the risk associated with a drug exposure” (line 25).

In addition, the document seems to circumvent the possibility of using these types of studies to provide evidence of a lack of an association between the drug and the event (which may have been seen spuriously during clinical trials). We request that the Agency reconsider this position.

II. Process and Timing for Protocol Review

To promote transparency in the conduct of pharmacoepidemiology safety studies, the Guidance states that “FDA encourages industry to inform FDA of all pharmacoepidemiologic safety studies; to submit plans and protocols for such studies before study initiation; and to submit comprehensive final reports with detailed methods and results to FDA in a timely manner.” (lines 43-45) While the Draft Guidance is clear on the expectations regarding the submission of protocols and observational safety studies, there is no information on the process by which FDA will review and comment on these study protocols. What are the specific timelines and expectations regarding feedback and approval? Who will provide feedback on the methodologic approaches of the study and how will discrepancies and disagreements be resolved?

Based on BIO member experience, FDA has reviewed final draft protocols and provided helpful feedback that requires incorporation into the final study protocol. However, the Draft Guidance appears to only refer to submission of final study protocols. Review of a final study protocol and provision of extensive regulatory feedback at the time of study implementation would be quite disruptive. BIO would support regulatory review at the final draft protocol stage to facilitate timely FDA review, with final protocol submission to the Agency upon its completion or at a mutually agreed upon time. If this 2-stage review is in actuality a general FDA practice, then it should be explicitly stated in the Guidance.

A clear, consistent process for protocol submission and review will minimize the potential for this process to significantly prolong the timeframe needed to complete studies and analyses, thereby delaying the availability of results that may affect future regulatory and development activities.

III. Communication with the Sponsor in Advance of an FDA-Initiated Study

In the spirit of transparency and scientific dialogue to advance the body of evidence around a particular product’s benefit/risk profile, we also request that FDA communicate with the Sponsor when developing the Agency’s own protocol for a pharmacoepidemiological study using large healthcare data sets in a regulatory context. We note that FDA has access to a number of large healthcare data sets, such as Medicare claims data or electronic health records databases that are occasionally used for independent FDA research on potential safety signals. Product Sponsors should have adequate insight into FDA’s process and rationale for selecting a particular study protocol so that there is opportunity to discuss the most appropriate methodology to further evaluate signals of serious risk around a given product. BIO member company experience suggests that interactions between FDA and Sponsors will encourage a selection of a study methodology that will result in the most valuable and medically relevant information for patients, physicians, and regulators.

BIO suggests that the Guidance should discuss a standard process or timeline for this FDA-Sponsor interaction. BIO also would like to discuss with FDA a process for Sponsors to access the database utilized in FDA-initiated pharmacoepidemiological studies so that conclusions can be replicated and validated.

IV. Scope of Recommended Protocols Submissions to FDA

Additionally, we request that FDA clarify the scope of the type of studies requiring FDA review. It is not immediately obvious whether “…all pharmacoepidemiologic safety studies…” (line 43) includes non-regulatory requested (internally initiated) studies or studies requested by other regulatory agencies. We note that many of these Sponsor-initiated studies are hypothesis generating in nature and are not intended for regulatory purposes or to provide conclusive scientific evidence.

BIO recommends that the scope of the recommendations should cover studies that are elements of regulatory commitment to the FDA only. We request that FDA clarify that these statements pertain to those studies that are agreed upon as part of a postmarketing study requirement or commitment (PMR/PMC) with the FDA or as part of a risk management plan. We also suggest that the Guidance state that the scope of the recommendation for protocol submission do not extend to studies that are descriptive in nature, hypothesis generating, assess a safety issue outside of the scope of a PMR/PMC, or are conducted as part of a comparative effectiveness assessment.

We also recommend greater clarity in the Guidance around the difference between hypothesis-strengthening and hypothesis-testing studies and that additional examples be provided.

V. Non-Disclosure of Study Protocols

BIO also notes that the Draft Guidance is silent on whether the study protocols submitted to FDA will be held confidential or publically disclosed. Certain study protocols may include information that is proprietary or confidential in nature and premature disclosure could undermine the competitive standing of the Sponsor. Therefore, BIO encourages FDA to revise the Draft Guidance to state that study protocols will not be publically disclosed.

Consistent with the FDA Amendments Act of 2007 (FDAAA) and Congressional intent, manufacturers are already required to register and submit results information for “applicable clinical trials” to the ClinicalTrials.gov database. An applicable clinical trial is defined by FDAAA to include controlled, clinical investigations, other than Phase 1 investigations, of a drug or device subject to FDA regulation. Observational and exploratory studies are excluded from this requirement, a significant protection of confidential proprietary information. However, many Sponsors choose to register these studies on a voluntary basis when proprietary information is not involved.

VI. Submission of Preliminary Feasibility Analyses

In Section IIIC on Study Approach Considerations, FDA encourages investigators to briefly describe any alternative study approaches and databases they considered before arriving at the proposed approach and to clarify why those proposed alternatives were neither feasible nor optimal in the context of answering the specific study question (lines 223-226). However, we note that contracted vendors such as Contract Research Organizations (CRO) or Academic Research Organizations (ARO) that are often responsible for the preparation of the study protocol may not have been involved in conducting the feasibility assessment. Many pharmacoepidemiology studies are outsourced by biotechnology and pharmaceutical companies, which involves relying on the selected CRO for the development of the study-specific protocol that is focused on the scientific question of interest (with scientific input and oversight from the company). The CROs independently implement the study and prepare final reporting of the results.

It is therefore of concern that this Guidance recommends inclusion of preliminary feasibility study details and rationale for decision making at the level of specific databases, methodology and approaches considered and ruled out for the study. In the situation of outsourced studies, as recommended, these details would have to become part of the study protocol.

We also suggest that these details may well be of a strategic and confidential nature, and at the point of final study protocol preparation are at best contextual and quite peripheral to the study. For example, a response from a CRO to a Sponsor’s Request for Information (RFI), which is often proprietary information belonging to the CRO provided under a Confidentiality Agreement, is often an important consideration and the underpinning for that manufacturer’s decision on vendors’ databases, study design, analytic plan, etc. Including specific detail on all data, methods, and other essential features considered in a final study protocol, as well as the rationale for ruling them in or out for the study, poses concerns given the sensitive nature of this information.

We request that FDA specify whether this information may be included instead in a briefing document distinct from the protocol. Under this alternative proposal, Sponsors could segregate out FDA’s recommendations in this regard from the narrow information specific to the study at issue, which is traditionally included in study protocols. The scope of considerations and rationale for decision making could be included in a briefing document accompanying a draft final protocol submitted for review and feedback by the Agency. The final protocol would then incorporate study-specific feedback and be submitted at completion to the Agency.

VII. The Importance of Conducting Preliminary Pilot Studies and Data Assessments

We ask FDA to consider a adding a separate category in Section V. Best Practices – Study Design to highlights the importance of conducting preliminary pilot studies and data assessments. These voluntary, Sponsor-initiated assessments can help Sponsors to better understand the study population(s) and sample size implications, evaluate the data source and its limitations, identify potential confounding factors in the study population, and define drug and outcome variables before finalizing the protocol.

VIII. Case Specifications and Outcome Validation:

Evaluation of index case code specifications and formal validation of outcomes are recommended, as is the incorporation of the information gleaned through these processes into the study protocol and analysis plan; if necessary, the modifications would need to be submitted as a final protocol amendment (lines 734-736, 182-186). Encouragement for pilot studies to evaluate case specification ahead of the finalization of a study protocol so that findings can inform this specification might be a worthy recommendation that would help minimize the need for amendments, as would be early conduct of feasibility or preliminary studies (lines 229-230).

For outcome validation, a plan to analyze only those study outcomes that were medically validated and adjudicated poses no significant difficulties. However, we request that FDA consider clarifying whether ascertainment or validation of other important variables should also be conducted (e.g., confounding variables). Consideration of other ways of using validated data or findings from sensitivity analyses, such as for context for the interpretation of findings, without requiring incorporation of the information into the analytic plan via protocol amendment, should also be provided.

BIO supports the validation of outcomes as a best practice, but we recognize that in some instances it may not be practical or feasible to validate certain outcomes. This should not always preclude the study from being conducted if the study can be informative despite this limitation. Indeed, the Draft Guidance also suggests that “For studies without outcome validation, the investigator should provide appropriate justification of the outcome definition used.” (lines 728-730).

IX. Quality Assurance (QA) / Quality Control (QC)

This Draft Guidance provides a great deal of guidance on QA/QC procedures. However, data holders rarely disclose completeness of data capture particularly when data reporting is voluntary (e.g., inpatient medical procedures). Therefore, the recommended QA/QC procedures are beyond data users’ control. Since the Draft Guidance addresses best practices in using electronic health data sets, we recommend FDA consider generating a separate guidance for the best practices of generating electronic health data, and move the sections IV-E and VI-G to the separate guidance.

It is also not clear whether the FDA is referring to the use of simulated data to assess the analytical performance of the statistical methods used or simple replication of analysis using different methods. We request that the Agency provide more specific guidance on the types of sensitivity analyses considered to be essential.

X. Statistical Analysis

Section VI on “Best Practices – Analysis” states that “In the study protocol, investigators should include a prespecified analysis plan that addresses the specific study objectives. The plan should specify primary and any secondary analyses. If investigators plan to perform preliminary analyses, they should prespecify the plan.” (lines 792-794) Given the detail that can go into the analytic plan, we suggest that it be standard practice for there to be a stand-alone analytic plan separate from the protocol, with only the top-level analytic details going into the protocol. A similar process is utilized in the context of drug development. Therefore, we suggest that the formal statistical analysis plan be separate from the protocol.

Additionally, we question if primary and secondary analyses have the same meaning in observational studies as they do in randomized controlled trials (i.e., whether this distinction is meaningful in observational studies) We suggest that the concept of primary and secondary analyses be eliminated in the context of observational research guidelines.

XI. Standardized Processes for Contracting with Certified CRO/ARO

While outside of the scope of this particular Guidance, industry, FDA, and CROs should evaluate opportunities to standardize the process for initiating studies conducted by contracted venders to enhance efficiencies. For example, CROs/AROs and other relevant researchers could create a voluntary, consolidated repository of CRO/ARO capabilities that contain information such as expertise, credentials, experience with specific data sources, and experience for ensuring quality assurance and quality control measures.

Alternatively, rather than having to make specific requests from vendors for every study commissioned, perhaps vendors should develop recommendations that would expedite the process of protocol development and enhance compliance with the Guidance recommendations. For example, vendor protocol templates could have study team expertise and credentials (lines 246-254) or a section on the vendor’s quality assurance and quality control processes that are requested (lines 412-429). Some of the requests regarding approval or denial of claims (lines 292-294) that reflect health plan-specific policies may also be made available by the vendors ahead of time.

CONCLUSION:

BIO appreciates this opportunity to comment on “Draft Guidance for Industry and FDA Staff on Best Practices for Conducting and Reporting Pharmacoepidemiologic Safety Studies Using Electronic Healthcare Data Sets”. We would be pleased to provide further input or clarification of our comments, as needed.

Sincerely,

Andrew J. Emmett
Managing Director, Science and Regulatory Affairs
Biotechnology Industry Organization (BIO)