In April of 2023, the FDA published the draft guidance, “Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence/Machine Learning (AI/ML)-Enabled Device Software Functions.” In this preliminary guidance, the FDA presented a forward-thinking strategy rooted in scientific principles to guarantee the safe, efficient, and swift adaptation, enhancement, and refinement of AI/ML-powered devices.
Although all of this all sounds intimidating, our goal is to break down the most significant parts of the Predetermined Change Control Plan (PCCP) for manufacturers to better understand this complex guidance.
What is a Predetermined Change Control Plan (PCCP)?
.png)
The Predetermined Change Control Plan (PCCP), is an innovative approach to address ML-related risk in premarket submissions for medical devices. This plan empowers manufacturers to refine and tune AI/ML algorithms through frequent deployments within the constraints of safety and effectiveness without the need to make an additional premarket submission. What does this mean in practical terms? It translates to a significant reduction in time and resources required by manufacturers and regulators in preparing and reviewing premarket submissions for software changes.
The utilization of PCCP is meant to reduce regulatory burden by providing the means for real-time improvements in devices - post-market. The flexibility to adapt and enhance without excessive bureaucratic hurdles is pivotal in enabling advancements in medical device technology.
The FDA has designated the acronym ML-DSF (machine learning-enabled device software function) to describe this specific AI use in the medical industry.
The focus on ML-DSF and the adoption of approaches like the PCCP represents a pivotal shift in the regulatory paradigm, fostering innovation and efficiency while maintaining the high standards of safety and efficacy in medical device development.
In order to understand the PCCP, first you need to understand why it is necessary in the AI/ML space.
To begin, artificial intelligence (AI), and more specifically, the AI subset of machine learning (ML), is a new and powerful tool companies around the world are beginning to integrate into their medical device systems in order to be more efficient and productive. The FDA recognizes the importance of AI/ML in the medical industry and is trying to create a process to allow companies to develop safe and effective medical devices that use ML models trained by ML algorithms.
Since, by definition, ML-DSF’s are constantly changing and improving themselves, the FDA has created the PCCP guidance to allow ML-DSF manufacturers to follow these guidelines instead of constantly creating new submissions for medical device modifications that would otherwise require a premarket approval supplement, such as a Special 510(k), De Novo Submission, or PMA supplement. In the FDA’s words, “This draft guidance proposes a least burdensome approach to support iterative improvement through modifications to an ML-DSF while continuing to provide a reasonable assurance of device safety and effectiveness.”
How to Create a PCCP
The PCCP draft guidance includes three main sections:
- Description of Modification: Itemization of the proposed device modifications, along with specific justifications for the ML-DSF modifications.
- Modification Protocol: Describes the means, methods, and manner to develop, validate, and implement these changes.
- Impact Assessments: Entails documenting the evaluation of benefits and risks associated with implementing a PCCP for an ML-DSF, along with strategies to mitigate those risks.
With the proper knowledge, team, and IT systems, manufacturers are able to create a compliant PCCP for their ML-DSF. Below, we will discuss each section of the PCCP.
PCCP Description of Modifications
Purpose of a Description of Modifications
This section is a description of each planned modification to a ML-DSF in the PCCP as either a stand-alone submission or in concert with a “new” premarket submission. For modifications to an approved PCCP, FDA recommends that a PCCP include “only a limited number of modifications that are specific, and that can be verified and validated.”
How to Complete Description of Modifications
Modifications to the PCCP that are difficult to verify and validate due to circumstances such as overly complex modifications, lack of specificity or clarity, or insufficient data or testing methods may impact the approval of the PCCP. By limiting the number of modifications, the FDA can concentrate their review efforts on a defined set of changes. This focused approach enables thorough scrutiny and assessment of each modification's impact on the device's performance, safety, and effectiveness.
The Description of Modifications should itemize the proposed device modifications, along with specific justifications for the ML-DSF modifications. To facilitate comprehension, labeling changes associated with these modifications should be detailed separately in a Modification Protocol. The details should be sufficient for understanding the nature of the planned changes and should link each modification to a specific performance evaluation activity.
It is essential to indicate if the modifications are automatic (implemented by software) or manual (requiring human intervention). This distinction becomes crucial in the FDA's assessment, especially concerning the substantial equivalence or safety and effectiveness of the modifications. Additionally, it should specify whether modifications will be uniform across all devices or differ based on unique clinical site conditions or individual patient characteristics.
Modifications suitable for a PCCP are those aimed at maintaining or enhancing device safety or effectiveness. Acceptable modifications may include quantitative performance enhancements, expansions in device inputs, or limited changes for specific subpopulations. All modifications should align with the device's intended use and maintain its indications for use.
The FDA will assess PCCPs considering the specific device's benefit-risk profile, proposed modifications, and its experience applying these policies across different device types.
PCCP Modification Protocol
Purpose of Modification Protocol
The purpose of the modification protocol is to establish a comprehensive set of procedures, delineating the step-by-step process for implementing the aforementioned protocol modifications while ensuring continued device safety and effectiveness. This includes the documentation describing the methods that will be followed when developing, validating, and implementing modifications delineated in the Description of Modifications section of the PCCP. The Modification Protocol also includes V&V activities that shall be performed (including pre-defined acceptance criteria).
How to Complete Modification Protocol
While the Description of Modifications outlines planned changes, the Modification Protocol describes the means, methods, and manner to develop, validate, and implement these changes. The goals include identifying methods, acceptance criteria, and data used for modifications. This ensures generated information aligns with regulatory requirements, mitigating identified risks, and being efficient for both manufacturers and FDA review.
These goals include:
- Identify the methods and data used to develop, validate, and implement all proposed modifications
- Identify the test methods, data, statistical analyses, and specified acceptance criteria for all proposed modifications
- Ensure that the information that would otherwise be generated and submitted to the Agency (i.e., if the modifications were implemented on a device that did not have an authorized PCCP) will be generated by the manufacturer for each modification and maintained consistent with recordkeeping requirements and in accordance with the manufacturer’s QMS.
- Ensure that the risks that have been identified in the Impact Assessment as related to modifications detailed in the Description of Modifications (including the update process and communication to users) will be mitigated
- Be least burdensome for the manufacturer to develop and for FDA to review. This includes being traceable and specific to the modifications detailed in the Description of Modifications section and sufficiently comprehensive to support specific modifications.
PCCP Impact Assessment
Purpose of the PCCP Impact Assessment
The Impact assessment entails documenting the evaluation of benefits and risks associated with implementing a PCCP for an ML-DSF, along with strategies to mitigate those risks. The manufacturer conducts this assessment within the existing quality system (design control) framework.
The Impact Assessment, included in the marketing submission, is structured to:
- Compare modified versus unmodified device versions
- Discuss benefits and risks of each modification, including social harm risks
- Ensure that activities in the Modification Protocol maintain device safety and effectiveness
- Evaluate inter-dependencies among modifications and the collective impact of all modifications (and other device modifications or not)
How to Complete the PCCP Impact Assessment
Manufacturers should document how individual modifications affect not only the ML-DSF but also the device's overall functionality, including other software functions and hardware. Additionally, for multiple function device products, manufacturers should consider including relevant information following FDA guidance to assess impacts effectively. It is also necessary to account for the device's critical quality attribute to ensure the device is performing at the same, or a better metric with the ML-DSF modification.
What do Medical Device Companies Need to Know About PCCP Submissions?
The PCCP process involves a periodic regulatory review of the manufacturer’s plan for maintaining the safety and effectiveness of a device. These include:
- A first time PCCP approval
- For deviations from an approved PCCP
- When ML-DSF modifications or protocol changes significantly impact the device’s safety and effectiveness
The FDA says, “Premarket authorization for an ML-DSF with a PCCP must be established through the pathway, De Novo pathway, or PMA pathway, as appropriate, as a PCCP must be reviewed and established as part of a marketing authorization for a device prior to a manufacturer implementing any modifications under that PCCP.”
A big mistake medical device companies make with regards to a new PCCP submission is thinking all they need to do is create a new process. A new process is necessary, however, in order to create a compliant PCCP, companies also need the correct IT system that will allow the new process to be efficiently and effectively tracked and executed.
How Ketryx Can Help Manufacturers & Developers with their PCCP
Ketryx helps companies operationally comply and release software at the speed allowed by their PCCP by enabling developers to utilize modern development tools and practices while equipping quality teams with automated documentation and testing. Ketryx also automatically updates traceability and authorizes companies to use it as controls for change management to ensure they are following their PCCP. Our team empowers manufacturers to use the tools that are used by software professionals today to develop and update software.
FDA Predetermined Change Control Plan (PCCP) Template
Ketryx has built an in-depth PCCP template for manufacturers and companies looking to produce an FDA-compliant PCCP. This template explains the step by step instructions and procedures for each of the three components of the PCCP (Description of Modifications, Modification Protocol, Impact Assessment) along with an example product. It is designed for companies to easily replace text with their company specific information.
FDA Predetermined Change Control Plan (PCCP) Interview with PCCP Experts, Erez Kaminski and Lee Chickering
As a benefit to our readers, we have included an edited transcript of an interview with two top engineers at Ketryx about the FDA’s Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence/Machine Learning (AI/ML)-Enabled Device Software Functions
Interview Transcript:
So what is a predetermined change control plan?
Traditionally people who develop validated systems would not often change that validated system. Why? Because it's already validated to its intended use, which means that it has been rigorously tested and developed to do the thing you say it does. Now, if it already does that thing, why do you need to change it, right? Like if a pacemaker works, why do you need a new pacemaker? It works. But what happens is over time you realize that there's many different reasons for why you want to change, whether that's new materials or new software comes in issues that are discovered in the market like software bugs or failures, or maybe cybersecurity requirements.
For the last 10-20 years, the FDA and other regulators all over the world have tried to figure out how they can enable people to change their devices with less regulatory overhead in order for them to focus on creating safe and reliable devices and not kind of regulatory compliance. Regulatory compliance should be something that's kind of the result of supporting high quality products. That's not something that you do in order to do it. All around the world, there have been a lot of thoughts of how you do it and kind of most recently, the FDA has uncovered that its path forward will be the Predetermined Change Control Plans.
A Predetermined Change Control Plan is a way to modify a validated system that's under regulatory compliance based on a pre-approved plan that describes the change. It says as I submit my device to the regulator, like a pacemaker or a car or an airplane, I say here are the things that I suspect I will need to change in the future, here are the ways I'm going to change it, and here are the ways I'm going to measure if that change is indeed effective, safe, reliable and provides better performance than my previous version. Because again, if it's worse, why would you change? Right? Like you always want to change for the better. This is now going to be a new regulatory regime starting with AI and machine learning based medical devices. From FDA's public statements, it's probably going to extend to many other domains. Until now, many things required you to resubmit to the FDA or to file what's called a letter to file inside your own documentation.
Until now, there has been this diagram that says, oh, if you change this, then you need to resubmit it to the FDA. However, that's really burdensome. It's really taking time from teams that do this work. Instead of that, what the FDA said is even for things that are kind of outside of the scope of this and are more than the small changes that allow you to do it internally, you can produce a PCCP as part of your submission to the FDA for the first 510(k). This also explains how you plan to change your artificial intelligence or machine learning system in the future. The FDA then approves that whole packet. They say they view your capabilities as sufficient to perform that change and you will be legally allowed to do that change without further submission to the agency.
To make that very practical, let's talk about machine learning. So machine learning is a way to create statistical models that can predict or classify information input into output and replace some functions the humans did until now. With those types of models, there are different metrics or scores that measure what's called a critical quality attribute. What they're trying to say is this machine learning or AI system has a critical quality. For example, it's level of accuracy or the level of error, which is a number. And that critical quality attribute, that metric as it's called in the machine learning space, is what I should measure and reaffirm once I update the software to be sure that I'm at the same level or better than I want it to be. That's what I write in the Predetermined Change Control Plan. What am I going to change the machine learning classifier, how am I going to change it, and how I'm going to change it by updating data and retraining it through my existing pipeline. You say, I'm going to use this metric, which is for example, the accuracy or the error or the F1 score. Then I'm going to see if that is better or worse than what I promised the FDA and my product does until now. And if it's better, then I'm going to release that to the public without additional approval from the agency because they already told me if it's above a specific score, then it is acceptable to the FDA.
The weird thing here is there are a lot of metrics. There's probably hundreds of metrics for machine learning models. So you need to decide with your subject matter experts what's the right quality of the model of the software system that you want to measure in order to make sure it's performing well. I think that it's pretty easy to understand that. This will of course usher a whole new era in medical device design and software design in AI because it would allow medical device software teams the kind of almost highest regulated form of commerce. This will allow teams to really utilize modern machine learning practices and update their models in almost real time or at least very routinely which should likely increase the quality and equity actually of their model because they get new data. Teams now have more data than they did when they released it because it's in use. They have more ideally diverse data and they are able to retrain the model and make it better over time. What do you do? How do you change it? How do you measure if that changes well? And then kind of how do you release it? That's kind of what the FDA is asking here.
What about future, and unforeseen modifications that will be covered under the PCCP?
The plan is you need to know and understand your device enough to know what you will change in the future. So if you're like, I'm going to change something totally new, it could be that you need to submit a new submission and a new change control plan. Once you go outside the scope of your PCCP, that PCCP no longer applies. That's what they're asking, right? They're like, if you know what you're doing and you know how the device acts and is going to change, then we're happy to approve you to do whatever you want. But if you don't know, then it's not appropriate.
Very similar to the PCCP, not all modifications for a pharmaceutical drug may receive FDA approval, right? And so for this PCCP, only those that the modifications outlined, you have to prove that you are providing substantial equivalence or safety and effectiveness criteria in order for it to be authorized. The FDA can deny some of your modifications outlined in your PCCP if you don't prove that.
What mistakes do companies make when they're doing the PCCP?
So I think one mistake is they think that they only need a new process.They think that they only need somewhere written a procedure that says, you know, you do ABC and D, you produce a PCCP. You use that PCCP in ABC way. But what they're missing is in order to do PCCP for software or machine learning software, you actually need to also have the right IT systems for that. You can't keep using the IT systems that only allow you to change once a year or once every other year when you're planning to change every week or every month or every quarter. You need to reimagine how you work in order to support this. It's not just that you need a new document and then a bunch of people to run around and fix it. You need a new way to do IT systems that would allow you to do it. It's about giving people the right tools to do it because a new transition and the speed of things in the cycle time of production requires new, modern tooling.
How does Ketryx solve the problems that the companies have?
Ketryx can write your PCCP and help you understand how to integrate that into your quality system. A quality system is the set of procedures that you use to follow and we give you the IT systems that allow you to perform this in reality and not just on paper.
It's important to get ahead of the curve and establish your procedures, establish how you're going to do this early in order to win. People are already looking at the draft version and not waiting for the final guidance, because if you wait for the final guidance, you're already too late and you're behind the curve. I think one of the important parts of this is it's going to open up a lot of possibilities. You can move a hundred times faster, but what you're missing is a way to do that. It's not just about having a procedure that explains how to do it with people. You need a system to do that because it involves continuous integration, continuous deployment, which is a whole topic.
Ketryx automatically updates traceability and allows you to use it as controls for change management to ensure you're following your PCCP as part of changes. Most importantly, it allows you to use the tools that are used by software professionals today to develop and update software.
.jpg)
