On April 3, 2023, the U.S. Food and Drug Administration (FDA) released recommendations for marketing submissions regarding pre-determined change control plans for artificial intelligence/machine learning (AI/ML)-enabled device software features in draft guidance. published. The draft guidance establishes principles and marketing submission recommendations for a subset of AI that the agency calls “machine learning-enabled device software features” (ML-DSFs). Automatically by software or manually by human input.
As explained in the 2022 FDA Year in Review, the FDA’s 2019 proposed regulatory framework for modification of artificial intelligence/machine learning-based software as medical devices introduces the concept of a prescribed change control plan (PCCP). it was done. PCCP identifies expected changes to AI-based software as medical devices (SaMDs) based on the algorithm’s intended use, retraining, and model update strategy. Section 3308 of the Consolidated Appropriations Act of 2023 codified this concept by enacting section 515C of the Federal Food, Drug, and Cosmetic Act (FDCA). The new section authorizes the FDA to approve her PCCP submitted for premarket approval. de novo A Classification Order or 510(k) describes planned changes that may be made without affecting the safety or efficacy of the device (i.e., PCCP allowed). Section 515C also gives the FDA the authority to require the PCCP to include labels for safe and effective use of the device as the ML-DSF changes. Under the April 3 draft guidance, successful PCCP submissions will allow device manufacturers to improve ML-DSF performance for specific patient populations or enhance device safety without additional marketing submissions. You may be able to pre-approve future device changes, such as
The FDA’s approach to the PCCP in its draft guidance is based on public comment and feedback from a 2019 discussion paper, software risk management concepts from the International Medical Device Regulators Forum (IMDRF), and Digital Health Software Pre-Certification (Pre-Cert). ) pilot program. It also incorporates the principles outlined in the White House AI Bill of Rights blueprint and reflects the FDA’s learnings from reviews of various AI/ML SaMD products. The draft guidance presents a holistic approach to help manufacturers and FDA navigate the regulatory complexities posed by AL/ML SaMD, and the rapidly evolving generative AI with medical device capabilities and capabilities. It sets the stage for the FDA to deal with the new wave of tools.
detail
data set
In the draft guidance, FDA distinguishes between the terms “training data,” “tuning data,” and “testing data.”
- training data Data used to build an ML model, including weights, connections, and component definitions. These should be representative of the proposed intended use group.
- tuning data (also called validation data) is the data used to evaluate a small number of trained ML-DSFs to explore. for example, different architectures or parameters. The FDA chose the term “tuning” rather than “validation” because the term is used in the ML community. no This is consistent with the FDA’s definition of “validation” in the Device Quality System Regulation (i.e.“confirmation by examination and provision of objective evidence that the specified requirements for the particular intended use can be consistently met”).
- test data Data used to establish reasonable assurances of safety or efficacy (performance). This data should be independent of the training and tuning data and should come from multiple sites different from the training and tuning data.
Licensed PCCP
Obtaining approval by including a PCCP as part of the marketing application eliminates the need for manufacturers to submit a premarket approval supplement. de novo Submissions or new 510(k)s for certain planned or anticipated device modifications that would normally require such submissions under applicable regulations. For example, changing the quantitative measure of the ML-DSF performance specification (for exampleimproved analytical and clinical performance resulting from retraining on new or broader datasets), changes related to device input to ML-DSF (for examplenew models or versions of the data acquisition system), or limited changes to the use and performance of the device (for examplefor use within specific subpopulations, such as retraining on larger datasets) may be suitable for PCCP.
Changes that deviate from the PCCP or are not implemented in accordance with the methods and specifications set forth in the PCCP change protocol require a new marketing submission. In addition, changes that change the original intended use and indications of use will require a new marketing application. The FDA includes a flow chart for evaluating whether modifications to devices comply with an approved PCCP. A new marketing application will be required to change the approved PCCP itself. For 510(k) submissions, the determination of substantial equivalence will be made against versions of the device that were cleared or approved prior to any changes being made under the PCCP.
A separate section within your marketing submission, the PCCP must include:
- A range of FDA-approved specifications for the properties and performance of the proposed change (i.e., Change description)
- Based on identified test methods, data, and statistical analysis, relevant verification and validation tests and acceptance criteria (i.e., change protocol)
- Documenting an assessment of the benefits and risks of implementing the proposed PCCP (i.e., Impact evaluation).
FDA notes that the PCCP must be fully detailed in the public part of the filing (for example510(k) Summary, de novo Decision Summary, Safety and Efficacy Premarket Approval Summary or Order of Approval) to support transparency. PCCPs must also comply with the quality system regulations of 21 CFR Part 820, specifically Design Controls (21 CFR § 820.30), Nonconforming Products (21 CFR § 820.90), and Corrective and Preventive Actions (21 CFR § 820.100) .
As described below, the draft guidance includes additional information on what applicants should include in their descriptions of amendments, amendment protocols, and impact assessment sections.
Change description
FDA expects a PCCP clarification of the changes to identify changes to the ML-DSF that manufacturers intend to implement. FDA recommends that the PCCP include only a specific, limited number of changes that can be verified and verified, and that such changes are described in sufficient detail to enable understanding. This recommendation is consistent with the NIST AI Risk Management Framework, which emphasizes explainable and interpretable AI. The change description must also clearly state:
- If the manufacturer intends to automatically implement the proposed changes (i.e.changes are implemented automatically by software, as in a self-managed ML process) or manually (i.e., changes require human input, action, review, and/or decision-making before implementation. This is a technique known as reinforcement learning from human feedback.)
- Either the proposed change will be uniformly implemented in all devices on the market, or it will be implemented differently in different products, e.g. due to the unique characteristics of a particular clinical site or individual patient. be done
modified protocol
The Change Protocol section of the PCCP should describe the methodology to be followed when developing, validating, and implementing the changes described in the Change Description section. FDA has listed four major components that must be included in the change protocol.
- data management practices (for examplehow input and reference data are collected, annotated, organized, stored, retained, managed, and used for modification)
- Practice retraining (for examplepurpose of the retraining process, description of ML models and components subject to change, practices to follow and retraining triggers)
- Performance Evaluation Protocol (for exampleperformance evaluation triggers, application of isolated test data for testing, calculated performance metrics, statistical analysis plans used to test the performance goals of each change, and performance evaluation failures are recorded and changed will not be confirmed)
- Update procedure (for exampleverifying that the modified device validation and validation plan is the same as that performed on the previous version of the device, explaining how software updates are implemented, A description of how users are affected, and a description of how changes are communicated to users).
FDA will address each of the four major components for each of the changes listed in the change description, and that the manufacturer will be responsible for any other claims that the Proposed Method is used elsewhere in the marketing submission. We recommend including a description of how it differs from or is similar to the method.
Impact evaluation
The impact assessment section of the PCCP provides guidance to manufacturers on the benefits and risks of implementing PCCP in ML-DSFs, and mitigation of those risks, using the manufacturer’s existing quality system as a framework for conducting impact assessments. It is intended to document the evaluation of FDA expects impact assessments to:
- Compare the device version with each change implemented to the device version without the change implemented.
- Discuss the benefits and risks (including risks of social harm) of each amendment
- Discuss how the activities proposed within the modification protocol continue to provide reasonable assurance of device safety and efficacy
- Discuss how the implementation of one change affects the implementation of another change
- Discuss the overall impact of implementing all changes.
In addition to discussing the specific impact of changes to the ML-DSF itself, FDA encourages impact assessments to discuss how each change included in the PCCP affects the overall functionality of the device. I am expecting (for example, other device software features and device hardware). FDA notes that for some devices, it may be less burdensome to include the impact assessment content in the Modified Protocols section of the PCCP, rather than as a separate section.
The draft guidance includes example scenarios for ML-DSF using PCCCP.
Takeaway
This long-awaited draft guidance provides a useful framework for manufacturers and developers to address the expected changes in AL/ML SaMD. However, manufacturers should evaluate the impact of these recommendations on existing quality system processes such as design controls, data management practices, risk management plans, adverse event reporting, labeling, etc., and potential implementations. There are also the above challenges. The draft guidance also adds to an expanding set of perspectives on the ethical and sound design, development, and deployment of AI/ML technologies in healthcare from industry coalitions, the World Health Organization, the European Union, and the United Kingdom. These evolving frameworks and regulatory approaches suggest an impending paradigm shift from a monitoring process targeted at AI/ML-enabled tools to an approach describing ML models that learn from data in real-time or near real-time. increase.
Device manufacturers incorporating ML-DSFs into their medical devices should consider submitting comments on the draft guidance. FDA encourages the public to submit comments on the draft guidance (Docket No. FDA-2022-D-2628) within 90 days of its publication. Federal Gazetteuntil 3 July 2023.
