Home

Risk Assessment of Observational Data

May 29, 2003

Dear Sir/Madam:

The following comments supplement those presented by the Biotechnology Industry Organization (BIO) at FDA's public meeting to discuss the Risk Management Concept Papers on April 9-11, 2003. BIO represents more than 1,000 biotechnology companies, academic institutions, state biotechnology centers and related organizations in all 50 U.S. states and 33 other nations. BIO members are involved in the research and development of health-care, agricultural, industrial and environmental biotechnology products. The Biotechnology Industry Organization (BIO) appreciates the opportunity to comment on Concept Paper #3: Risk Assessment of Observational Data: Good Pharmacovigilance Practices and Pharmacoepidemiologic Assessment.

General Comments

BIO agrees that observational studies can be useful for evaluating safety signals. However, additional discussion is needed regarding the designs and statistical control, as well as effect sizes that warrant regulatory actions. The potential for inconsistent implementation is significant and could lead to spurious differences in labeling of similar biologics.

The concepts in this paper, including pharmacovigilance plans, should be applied on a case-by-case basis. To this end, we recommend the FDA describe the approach for ensuring that requirements for observational data are applied consistently.

Small companies will require time to implement some of the provisions of the proposal given limited capabilities in pharmacoepidemiology, data mining, survey methodology and root cause analysis. We recommend the final document take this into account.

Specific Comments

What is pharmacovigilance?

We believe the concept paper appears to confound risk assessment and risk intervention.

Proposed Actions (lines 22-28)

  • Lines 22-28: Replace the words "prevention" and "minimizing" by "characterization".

What is a pharmacovigilance plan?

Sections "C. What is a pharmacovigilance plan" and "D. How does a pharmacovigilance plan differ from a risk management plan" are conceptually discordant concerning the motivation of pharmacovigilance plans. The former indicates that pharmacovigilance plans are targeted to ongoing evaluation of identified safety signals. The latter suggests a broader scope that includes identification of new signals (lines 414-417 is consistent with the use of pharmacovigilance plans on a case-by-case basis). While it is clear that enhanced pharmacovigilance activities may uncover new signals, we recommend a non-specific pharmacovigilance plan not be implemented merely for the identification of new signals.

Proposed Actions (lines 55-60 with reference to lines 69-70)

  • Provide a clear operational definition of the term signal
  • Define pharmacovigilance plan as enhanced pharmacovigilance
  • Stipulate that the advisability of a pharmacovigilance plan will be determined on a case-by-case basis (reconcile lines 56, 69 and 414-417)
  • Indicate that the specifics of the plan will be negotiated by the sponsor and the FDA (including the Office of Biostatistics and Epidemiology)
  • Make it clear that pharmacovigilance plans are targeted to identify safety signals for important events, as opposed to any and every signal, and note that the resource intensive pharmacovigilance plan is not a first-line approach to signal identification
  • Provide clear criteria to help ensure that pharmacovigilance plan requirements are applied consistently
  • Stipulate that all pharmacovigilance plans should provide the rationale and basis for their implementation
  • Stipulate that all pharmacovigilance plans should describe the factors that establish the event(s)-of-interest as a signal(s)

Characteristics of a good case report (lines 79-117) & Privacy and human subject protection (lines 237-246)

BIO agrees with FDA's view of a good case report. However, we believe there are limits, including confidentiality, to obtaining complete information in some instances. Obtaining high-quality reports requires considerable effort on the reporter's part. The fundamental stumbling block to high-quality reports is a lack of reporter-incentive. There is no easy solution to this problem.

Confidentiality issues limit sponsors' ability to obtain information on individual case safety reports. Health Care Professional education may be able to mitigate this because some reporters are unfamiliar with the special nature of surveillance activities. Sponsors' efforts alone will not change HCPs' perception of their risk related to mishandling of confidentiality issues. Moreover, confidentiality regulations may be a convenient reason, in some cases, for withholding patient information.

Incomplete information on adverse events will remain a reality, despite FDA's and Industry's best efforts. Therefore, we believe additional discussion is needed to determine how incomplete information should be used in signaling.

Proposed Actions (lines 81-104)

  • Include a paragraph that delineates how current confidentiality regulations do, and, do not limit reporters' ability to provide the information needed to complete a good case report
  • Undertake an FDA information campaign to educate HCPs concerning the role of confidentiality in safety surveillance

How are case series developed?

BIO agrees that there should be a structured approach to developing case series. This will involve casting a wide net to identify potential cases for possible inclusion in the series. We recommend the FDA provide guidance regarding the principles for including potential cases in the final series. For example, inclusion in the final case-series might depend on the nature of the event, the validity of the diagnosis, and the strength of the evidence for causality among other things.

Proposed Actions (lines 119-130)

  • provide a conceptual framework for counting (constructing a case-series) to serve as the basis for regulatory decision-making
  • Line 126: the phrase "and appropriate counting" should be added to line 126 to read "… would be developed to provide consistent characterization and appropriate counting of the adverse events …"
  • Line 128: the word "potentially" should be added to line 128 to read "… datamining techniques may be applied to the database to identify potentially relevant cases."

When and why are pharmacoepidemiologic studies recommended (lines 139-184)

BIO agrees that observational studies can be useful for evaluating safety signals. BIO believes that the main utility of Pharmacoepidemiologic studies in the biotech industry is in the evaluation of potential signals and the further characterization of signals that have already been identified. Although potential signals may arise from pharmacoepidemiologic studies, most will derive from other surveillance activities such as spontaneous and stimulated reporting systems.

While pharmacoepidemiologic studies represent a significant surveillance refinement, in many cases they provide a relatively crude tool for establishing causality. The concept paper notes references concerning the conduct and interpretation of such studies. Nonetheless, we believe specific FDA guidance on the role and advisability of the various study designs in signal evaluation would promote standardization within industry and the agency. We recommend the discussion also delineate the level-of-evidence for signal confirmation provided by each design and its corresponding measures-of-effect. We also recommend closing with a statement of the conceptual link between the observed level-of-evidence and potential regulatory actions.

There is debate concerning statistical adjustment for confounding and interpretation of observed-effect-size in signal research. Experience suggests a tendency to ignore confounding and equate effect sizes from randomized controlled trials with those from observational studies. We believe it is important for the FDA to address these issues in the context of signal confirmation.

Automated databases offer some important advantages. However, they are difficult to use for reliable inference and have definite limitations that are not widely appreciated.

Proposed Actions (lines 139-184)

  • Make a clear statement that observational studies do not constitute a first-line approach to signal identification
  • Provide guidance indicating the FDA's view of how optimal study design varies with the signaling context
  • Provide guidance concerning how the observed level-of-evidence (design used, magnitude of observed effect, etc.) can affect regulatory action
  • Provide general guidance concerning handling of confounding and interpretation of effect size in signaling research
  • Provide guidance concerning the concept of FDA-alerting thresholds for ongoing observational studies
  • Line 167-187: provide a classification system for automated databases
  • Line 167-184: provide a discussion of the relative merits and limitations of the different classes of automated databases for purposes of evaluating signals.
  • Line 182-184: provide guidance regarding the validation (outcomes and confounders) of data derived from automated databases, as well as situations where validation may not be necessary
  • Line 182-184: address how confidentiality may or may not affect the use and validation of automated databases

Registries

We believe the definition of a registry used in the concept paper is broad and does little to reduce confusion about registries. Registries that are not highly focused are of little value for evaluating signals. Since the FDA has considerable experience with registries, we recommend the agency review its experience and illustrate how registry-variants have succeeded and failed in evaluating signals.

Proposed Actions (lines 186-209)

  • Provide a classification system for the range of study designs that the FDA considers registries and indicate which of these the agency feels can be used for signal detection as opposed to signal confirmation
  • Provide FDA guidance concerning how the requirement to participate in a registry to gain access to a newly approved drug may affect

Surveys

Lines 211-235: Surveys probably offer the least utility for evaluating signals. We recommend the FDA review its experience with surveys and demonstrate how these surveys have succeeded and failed in evaluating signals.

Privacy and human subject protections ensured?

Lines 237-246: We recommend the FDA provide additional guidance regarding the impact of confidentiality initiatives on the current proposal. In particular, there is a need to indicate when proposed activities may not qualify as surveillance.

Calculating incidence rates and reporting rates (lines 261-296)

Reporting Rates (lines 261-286): BIO recognizes the pros and cons of reporting rates and their comparison to background rates. A consistent approach to interpretation of reporting rates subject to underreporting would be helpful. We recommend the agency engage partners to produce guidelines concerning the use of reporting rates in regulatory decision-making.

Product comparisons based on reporting rates require strong assumptions that are often questionable. We recommend that regulatory decisions based on comparisons of product reporting rates not be made without consideration of assumptions and a clear understanding of the surveillance model used for each compound, as well as evaluation of the procedures used to estimate treatment exposure. We believe such comparisons should be avoided when products are used for different indications.

Background rates (lines 288-296): BIO agrees that background rates are useful. In general, indication-specific background rates are not available. Additional discussion is needed to understand the agency's use of background rates and the use of controls for observational data.

Proposed Actions (lines 261-296)

  • Line 264-268: the FDA should clarify how to count new cases and patient exposure for delayed occurrence after exposure to a drug
  • Discuss in considerably more operational detail the use of reporting rates in signal detection, signal confirmation and regulatory decision-making
  • Provide more specific guidance, conditioned on the surveillance model, regarding accounting for under-reporting and how it should influence decision-making
  • Delineate criteria for the adequacy of exposure estimation
  • Discuss the need for age and gender-specific background rates
  • Provide guidance concerning the requirement for a control group (concurrent, historical from an automated database, etc.) for signal detection/confirmation, when indication-specific and or general population-based background rates are available

Individual causality assessments

In practice, individual causality assessments are generally unreliable. As such, guidelines are needed to determine which events to include in constructing case-series and calculating reporting rates; e.g., should events that are judged to be "unlikely" or "not related" be included in the calculation of reporting rates?

We believe it is noteworthy that the concept paper ignores events that are deemed "not related" to product. While infrequent, there is no doubt that this category of adverse events exists and can be identified.

Assessing the degree of causality based on the quality of information supporting a causal association is a very useful concept. We believe it would be helpful to discuss the relative weighting of the different types of information for evaluating the strength of causal evidence.

We recommend the concept paper also address the implications of different thresholds for causality in expedited reporting vs. signaling.

Proposed Action (lines 324-372)

  • Provide guidance concerning the manner in which construction of case-series (counting cases in the numerator) should be conditioned on the causality assessment of the individual cases
  • Address the differences in individual causality assessment for expedited reporting (for example in the MedWatch program) vs. signal detection and confirmation
  • Allow for the possibility that a relationship to suspect drug can be excluded in a small number of cases (crucial because a small number of cases can make a big difference when evaluating signals for certain important events)

How would safety signals be reported to the FDA (lines 374-394)

Line 376 stipulates that the FDA would expect sponsors to submit all available safety information when a signal is detected. This line suggests that it is the responsibility of the sponsor to identify signals. Yet, as mentioned previously, the concept paper does not provide a clear operational definition of the signal. Even with a clear definition of "signal" there will be room for reasonable people to disagree on the moment at which time a signal occurs. We believe this is a particularly important point to resolve before the document becomes final. In addition, we recommend the FDA address the timeline for notification of a signal, as well as the potential impact of agency-sponsor disagreement on the presence and time of signal identification.

Line 378 refers to the sponsor's need to provide an assessment of the risk/benefit profile of the product. However, neither risk nor benefit is defined, operationally or otherwise, in the document. It is not clear whether this proposal goes beyond current best practices for benefit/risk assessment.

Proposed Action (line 374-394)

  • The FDA should provide details concerning the proposed reporting framework and consequences of disagreements concerning the presence and timing of a signal before implementation
  • As previously mentioned the FDA should provide an operational definition of signal that can be implemented consistently and can be reliably measured, before considering moving further with a new signal reporting framework
  • The FDA should define a preferred framework for quantifying risk vs. benefit before requiring a more formal product risk/benefit characterization

How can safety signals be monitored through enhanced pharmacovigilance (lines 396-455)

Line 447-450: We recommend the FDA delineate its procedure for making the decision to take an issue to the Drug Safety and Risk Management Advisory Committee.

FDA Questions

  1. 1. How can the quality of spontaneous reports be improved?

    Answer: FDA education of HCPs concerning confidentiality in drug surveillance. Maintenance, by sponsors, of lists of key-events that require special investigation. Sponsor guidelines for consistent work-up of key events. Direct Sponsor-Physician investigation of key events. Mandatory reporting of adverse events by Health Care Professionals will probably not work in the USA.

  2. 2. What are the possible advantages or disadvantages of applying datamining techniques to spontaneous report databases for purposes of identifying signals?

    Answer: No automated technique is available that can overcome the low information content (per-case quantity, quality, reliability, accuracy) of the typical spontaneous reports database. Disadvantages include false-positives and considerable additional manual work and resource consumption for an unsubstantiated public health benefit. The main advantage is increasing the number of potential signals. However, this may come at a price; i.e., diversion of resources from more productive surveillance activities.

  3. 3. What are the possible advantages and disadvantages of performing causality assessments at the individual case level?

    Answer: Individual causality assessments are generally unreliable. However, they can be very useful in a limited number of cases. Individual causality assessments are most likely to add information and change decisions when a small number of high-quality events with well established pathophysiologies and etiologies can affect a regulatory decision. For the most part, they add little to the tacit assumption that a drug induced etiology is possible. At times they can be downright confusing to team members who are not safety professionals, because the causality threshold for event counting may differ with the surveillance activity; e.g., expedited reporting vs. signal confirmation.

  4. 4. Under what circumstances would a registry be useful as a surveillance tool and when would it cease to be useful?

    Answer: The greatest utility for signal evaluation is in the setting of a clearly and relatively narrowly defined safety question (or limited number of safety questions) and where the design permits the reliable measurement and comparison (to control) of reliable and important outcomes and cofounders. They cease to be useful for signal confirmation when they allow nothing more than a posteriori fishing expeditions (though they may have some utility in signal detection, certainly more than some attempts at data mining). At some point the distinction between a registry and data mining begins to blur.

  5. 5. Under what circumstances would active surveillance strategies prove useful to identify as yet unreported events?

    Answer: While the concept paper describes a potential focus for active surveillance, it does not define active surveillance. We recommend the paper describe the different types of active surveillance. In addition, we recommend it delineate the basis for different types of surveillance and explain the agency's basis for requiring it.

We appreciate the opportunity to provide our comments and look forward to reviewing additional guidance.

Sincerely,

Gillian R. Woollett, MA, DPhil
Vice President Science and Regulatory Affairs