If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Center for Digital Health - Data & Analytics, Mayo Clinic, Rochester, MNDepartment of Primary Health Care Sciences, University of Oxford, Nuffield, Oxford, England
Artificial intelligence and clinical software solutions are augmenting patient care, optimizing clinical workflows, and altering health care organization operations. The regulation and oversight mechanisms under development for this wave of digital innovation will set the precedent for health care and play a key role in shaping the future of digital health. It is critical for the physician voice to be active in both policy making and product development to ensure that clinical outcomes and the needs of patients are prominently represented. With digital technology rapidly changing health care, it can be difficult for practicing physicians to participate in the future while simultaneously tending to their clinical duties. We aim to ease this cognitive burden by providing physicians a primer for current regulation of software and artificial intelligence by the Food and Drug Administration. Regulation and oversight will greatly benefit from physicians primed to add their knowledge and patient-first values to the future of digital technology.
Whether we realize it or not, we all interact with artificial intelligence (AI) daily, including facial recognition software on smart phones, personalized advertisements, and email spam filters. Research leveraging AI algorithms is also prominent in health care, with 1 review finding that the number of AI-based publications has increased from 596 in 2010 to 12,422 in 2019.
However, despite the increased academic interest in AI, the growth in the field of AI-enabled medical devices available for commercial use has not paralleled this broader trend. According to the public listing by the Food and Drug Administration (FDA), there are currently 530 devices that have AI or machine learning (ML) capabilities
One reason for the gap between research and clinical deployment is barrier to entry: regulatory requirements in health care far exceed those in other industries.
Although regulatory requirements are often viewed as barriers, they can also be interpreted as a framework to ensure that only safe, effective AI is used in the health care industry. Although the FDA has published various articles on their stance on AI- or ML-enabled devices,
only certain functionalities are regulated and, thus, have explicit standards and expectations for design, development, and deployment by health care institutions. Interpretation of FDA regulation, including new final guidance on the Clinical Decision Support (CDS) software, offers an opportunity for physician participation to influence AI- or ML-enabled clinical care that optimizes patient benefit.
As key stakeholders, physicians can help guide the oversight of AI health care technology to ensure its quality and success. The physician voice will help set expectations for health care AI in a manner that delivers its promise while still prioritizing clinical outcomes and the best interest of patients. We hope to encourage physician participation in the future of AI innovation and governance by providing an overview of the current regulatory landscape for health care AI and recommendations for navigating this space.
Defining AI or ML and Software As a Medical Device (SaMD)
Health care stakeholders are yet to reach a consensus on the definition of AI and ML when it is applied to health care. The American Medical Association, for example, has opted to avoid using the term “artificial intelligence” entirely for purposes of clinical care and rather use the term “augmented intelligence” to emphasize the sustained role of humans and physicians.
For the purpose of this discussion, we have used the definitions provided in a recently proposed FDA regulatory framework, which has defined AI as “the science and engineering of making intelligent machines, especially intelligent computer programs.”
Proposed regulatory framework for modifications to artificial intelligence/machine learning (AI/ML)-based software as a medical device (SaMD). US Food & Drug Administration. https://www.fda.gov/media/122535/download. Accessed December 28, 2022.
Proposed regulatory framework for modifications to artificial intelligence/machine learning (AI/ML)-based software as a medical device (SaMD). US Food & Drug Administration. https://www.fda.gov/media/122535/download. Accessed December 28, 2022.
The FDA has combined these terms and refers to them generally as “AI/ML,” referring to such products as containing “AI-/ML-enabled” functionalities.
When AI- or ML-enabled software meets the statutory definition of a “medical device,” it may be subject to FDA regulation. A medical device subject to FDA regulation under the Food, Drug, & Cosmetic Act includes “an instrument, apparatus, implement, machine, contrivance, implant, in vitro reagent, or other similar or related article, including a component part, or accessory which is…intended for use in the diagnosis of disease or other conditions, or in the cure, mitigation, treatment, or prevention of disease.”
Software as a Medical Device is software used for 1 or more medical purposes without being a part of hardware. Most AI- or ML-enabled software can function on its own as SaMD as opposed to software that functions as an integral part of medical device hardware (ie, software in a medical device).
Proposed regulatory framework for modifications to artificial intelligence/machine learning (AI/ML)-based software as a medical device (SaMD). US Food & Drug Administration. https://www.fda.gov/media/122535/download. Accessed December 28, 2022.
Artificial intelligence- or ML-enabled SaMD products have their own standalone, intended use, whereas AI- or ML-enabled software in a medical devices generally does not.
Scope of Fda Regulation for AI or ML
Determination of whether the AI or ML function meets the definition of a medical device and is regulated by the FDA starts with establishment of the intended use, which is the scope of how the software is used based on the subjective and objective intents of the developer’s claims, including labeling, materials, promotion, and advertising.
One of the primary mechanisms for describing the intended use to regulators is indication(s) for use, which is a short statement that includes information such as what the AI or ML does, who will use it, who it will be used on, where it will be used, and what limitations, if any, should apply to its use.
This is the foundation for planning regulatory strategies and will ultimately govern the claims allowed for marketing and promoting the use of AI or ML.
Indications for use help innovators or manufacturers establish whether the intended software function qualifies as an FDA-regulated medical device. For example, an otherwise regulated software function intended only for educational use is not regulated by the FDA because it does not meet the definition of a medical device. Conversely, software functions that meet the definition of a medical device but are intended only for research fall under a different set of regulations intended for the protection of human subjects (eg, 21 Code of Federal Regulation 812 for investigational device exemptions).
Generally, the FDA’s regulation of medical devices is risk based, meaning that the higher likelihood that an AI or ML product may cause harm to the user (patient, caregiver, or health care professionals [HCPs]), the higher the likelihood that FDA oversight will be necessary.
For example, the FDA has a category of AI or ML that fits into the category of “enforcement discretion,” in which the FDA opts to not currently enforce compliance with the Food, Drug, & Cosmetic Act because of minimal risk of the intended use. The 21st Century Cures Act and the FDA’s interpretation of this congressional language help AI or ML manufacturers determine whether the risks associated with use of the product require FDA oversight. In almost every case, regulatory analysis involves interpretation of clinical risk, which necessitates the physician perspective. Understanding how physicians contribute to this regulatory analysis starts with understanding the regulations and software categories exempted from FDA oversight.
21st Century Cures Act Software Exemptions
To help determine appropriate regulatory policies for a software function, the 21st Century Cures Act and FDA guidance documents have clarified that certain software functions do not meet the definition of a medical device and are, therefore, not subject to FDA regulation. Table 1 provides a general summary of certain exempted software functions; however, the guidance documents themselves should be consulted before relying on any exemptions because determination is often based on the context of use.
Table 1Exempt Software Functions Under the 21st Century Cures Act
Software function
Summary
FDA examples
Administrative support of a health care facility
Software functions intended for general health care operations
Billing Scheduling Population health
General wellness
Low-risk software functions intended only for general health or activity, without reference to a specific disease or condition wellness
Weight management Physical fitness Mental acuity
Transferring, storing, converting formats, displaying data and results
Software functions intended only to transfer, store, convert, or display data without controlling or altering the functions of a connected medical device
Networks IT infrastructure
Serving as an electronic patient record
Software functions intended to act as the equivalent of a paper medical record for use by health care professionals and not intended to analyze or interpret data as a medical device
Electronic health records Health IT
FDA, Food and Drug Administration; IT, Information Technology.
The 21st Century Cures Act has also defined the CDS software, which is excluded from the definition of a medical device (ie, “nondevice”) and, therefore, not subject to FDA regulation.
The FDA has defined the CDS software as “a variety of tools including, but not limited to, computerized alerts and reminders for providers and patients, clinical guidelines, condition-specific order sets, focused patient data reports and summaries, documentation templates, diagnostic support, and contextually relevant reference information.”
The act has defined 4 criteria, all of which must be satisfied for a software functionality to be considered “nondevice” CDS. Table 2 provides a summary of the 4 criteria defined by the act.
Table 2Non-Device Clinical Decision Support Software Criteria in Section 520(o)(1)(E) of the Food, Drug, & Cosmetic Act
21st Century Cures Act Criteria
Criteria 1
Criteria 2
Criteria 3
Criteria 4
Criteria Language
Not intended to acquire, process, or analyze a medical image or signal from an in vitro diagnostic device or a pattern or signal from an acquisition system
Intended for the purpose of displaying, analyzing, or printing medical information about a patient or other medical information
Intended for the purpose of supporting or providing recommendations to HCPs about prevention, diagnosis, or treatment of a disease or condition
Intended to enable HCPs to independently review the recommendations so as not to rely primarily on the recommendations to make a clinical diagnosis or treatment for an individual patient
Key interpretations from FDA final guidance
Repeat measurements may be considered a “pattern” and, therefore, fail to meet criterion 1
“Medical information” is normally communicated and understood by HCPs in the clinical context
Software outputs that are specific to a disease or condition or that are used in a time-critical clinical context may fail to meet criterion 3 because of the creation of automation bias
The FDA recommends “labeling” to enable independent review, including a summary of the logic, the data used, and validation results
FDA, Food and Drug Administration; HCP, health care professional.
In September 2022, the FDA issued final guidance clarifying the agency’s interpretation of the 4 criteria required for CDS under the 21st Century Cures Act. The guidance includes the following: (1) clarification on definitions and terminology, (2) consideration for satisfying the criteria, and (3) numerous examples. In order for software to be considered nondevice CDS, the end user must be an HCP. The FDA also provided several interpretations to help ensure that HCPs have the necessary information and time to make independent determinations, including guidance to consider whether a targeted recommendation or risk score will cause an HCP to become overly reliant on results (“automation bias”), guidance to consider whether there is sufficient time for the HCP to critically assess the results, and recommendations on information provided with the results that enable the HCP to independently review the basis for recommendations provided by the software. The CDS guidance also clarified the FDA’s interpretation of criteria 1, which states that software functions intended to acquire, process, or analyze medical images or signals from an in vitro diagnostic device or patterns from a signal acquisition system do not qualify as nondevice CDS. The FDA’s final guidance added that repeat sampling or measurements from a signal acquisition system, including electrocardiography, computed tomography, DNA samples, glucose measurements, or blood or tissue samples, could qualify as a pattern and, therefore, not meet the nondevice CDS criteria.
The category known as “enforcement discretion,” in which the FDA does not intend to enforce its regulations over certain software functions, is not acknowledged in the FDA’s final CDS guidance document; however, this category may still apply to other software functions, as identified in another FDA guidance document titled “Policy for Device Software Functions and Mobile Medical Applications: Guidance for Industry and Food and Drug Administration Staff.”
The scope of software functionality within the FDA’s “enforcement discretion” category includes products that “help patients self-manage their disease or conditions without providing specific treatment or treatment suggestions” or those that “automate simple tasks for healthcare professions” that were taught in medical school or routinely used in clinical practice (eg, calculation of body mass index).
In the process of defining a product’s functionality and categorizing its regulatory status, it is advisable to consider the end users’ needs in the real-world clinical use scenario as a primary influencing factor for decision making. In addition to physicians playing a critical role in the assessment of the risk of an intended software function, physicians also represent real-world scenarios for which the software functions will impact. Physician involvement at the earliest stages of defining the needs that the software will address is a critical factor for choosing the optimal regulatory strategy for the intended use.
FDA Review and Regulation
The FDA has classified medical devices software based on risk. Class I comprises lowest-risk software, with minimal impact on patient safety; class II comprises moderate-risk software and includes many CDS software functions; and class III comprises highest-risk software, such as life-sustaining software functions.
As the FDA class increases with risk, the expected controls also increase. Most class I devices are exempted from FDA notification before market release. Products that are nondevice or in the “enforcement discretion” category also do not require FDA notification before market release. Devices subject to the FDA notification requirement (generally, classes II and III) must prepare the necessary evidence to be either “cleared” through the 510(k) process or “approved” through the premarket approval (PMA) process. For devices subject to the 510(k) process, the typical pathway for class II devices, submission to the FDA allows them to determine whether the device is substantially equivalent to an already FDA-cleared medical device. Of note, most AI- or ML-enabled SaMD products must be “cleared” via 510(k) submission. The PMA process, on the other hand, which is typically required for class III devices, involves a more robust review. This not only requires a longer and more expensive pathway to FDA approval but also provides certain benefits such as pre-emption from state tort law.
Defining the regulatory status (nondevice, enforcement discretion, or device or oversight) and class for an AI- or ML-enabled device is critical to determining the pathway(s) to the market, including the anticipated rigor of the FDA process that the device must undergo before commercialization.
Another potential route through FDA review is the de novo “authorization” pathway. This pathway is applicable to devices that do not align with an already-cleared FDA-predicate device; however, class II regulatory controls are sufficient to ensure safety and effectiveness of the novel product. When new and innovative devices have no substantial equivalent, they are not eligible for the less arduous 510(k) pathway. However, rather than automatically subjecting all new devices to a full class III PMA review, the de novo authorization process allows the submitter to simultaneously seek FDA authorization and justify a class II level of risk. Generally, on successful FDA authorization under the de novo pathway, the device is designated as class II and subject to the 510(k) process going forward, including the ability for others to claim substantial equivalence. The FDA has authorized 18 AI- or ML-enabled devices through the de novo submission process, generally resulting in the creation of new product codes and applicable regulations for novel technologies.
A flowchart summarizing FDA pathways is presented in Figure 1.
Figure 1Steps for determining the extent of Food and Drug Administration jurisdiction and class designation. FDA, Food and Drug Administration; PMA, premarket approval.
In addition to determining the appropriate FDA review pathway, device class also impacts continued FDA regulatory controls throughout the lifetime of the product.
The following are the FDA’s regulatory controls, with increasing requirements: general controls (all devices), special controls (class II), and premarket authorization (class III). For example, all medical devices are subject to “general controls,”
In contrast, “special controls” dictate that such products contain specific labeling content that is not required for products that are simply under “general controls.”
The FDA provides guidance on how to determine device classification, which includes searching publicly available databases.
In addition to indications for use, FDA product codes and affiliated regulations help define the pathway to market and relevant FDA controls. To help determine regulatory obligations, the FDA also accommodates questions and provides feedback using the voluntary Q-Submission Program.
The use of this process to interact with the FDA can help guide innovators to conclusions on a variety of critical considerations, such as indications for use, labeling, and clinical study protocols before overcommitting resources. At later stages, the presubmission process under the Q-Submission Program helps AI developers refine the characteristics of the device and the proposed clinical evidence necessary to bring their product to market. Additionally, the FDA provides a process for requesting feedback regarding whether the software qualifies as a medical device, the anticipated risk class (class I, II, or III), and the regulatory pathway, such as the 510(k) or PMA review pathways, through 513(g) submission.
Many AI- or ML-enabled software products require a documented quality management system (QMS) before commercial release. The FDA’s quality system regulation (21 Code of Federal Regulation 820) is often wrongly assumed to be only a “compliance checklist” necessary to complete before the FDA clears or approves a product. Rather, FDA regulations are intended to ensure that there are processes, trained personnel, and oversight in place to ensure that the medical device is predictably safe throughout its development and deployment lifecycles.
An FDA-compliant QMS comprises a system of policies, procedures, and documentation that align with good manufacturing practices and company operations to enable safe and effective medical devices throughout their lifetime.
The QMS design controls consist of design planning; design inputs that establish user needs and risk controls; design outputs, such as user materials; verification to ensure that the product works as planned; validation to ensure that the product works in its intended setting; and processes for transferring the software into the clinical environment. Food and Drug Administration investigators also look for compliance with applicable document and record requirements; effective management and resource allocation; and required administrative controls, such as an audit program, handling of complaints, corrective actions, and events that require reporting to the FDA.
Physicians can play a role at every stage (Figure 2).
Figure 2A quality management system is required before commercial release and consist of product realization, monitoring and improvement, as well as management and resource allocation. Physicians can contribute at any stage in this cycle. QMS, quality management system.
Physicians play a key role in multiple areas of a successful QMS. Food and Drug Administration regulations and guidance documents specific to software help ML or AI manufacturers build the necessary infrastructure to prepare for FDA submissions and inspections.
However, there is some flexibility in how compliance activities are conducted, and the physician voice can help target efforts to areas with the greatest impact on practice of medicine and patient care.
Physicians play a key role in risk management during the design input stage. The standards and guidance related to risk management are critically important to the QMS. Physician involvement in the formal framework for the assessment and mitigation of the risk of harm is both expected by the FDA and in the best interest of patients ultimately impacted by AI or ML. Operationally, it is preferred to have a physician sign off on the final risk management report, summarizing the risk mitigation activities and residual risks of the software. After deployment, physicians also play a critical role in the analysis of the risks of AI or ML causing harm and in the determination of a corrective action plan to mitigate and prevent future harm.
Physician involvement is also important in development and maintenance of messaging associated with deployed software during the design output stage. The FDA’s jurisdiction includes enforcement over the “labeling” of medical devices, which broadly encompasses any representation or claim accompanying the device. Physician involvement ensures that messaging aligns with the intended scope of the software. Generally, promotion of a medical device outside of the approved label is called “off-label-promotion” and may subject the device to enforcement actions by the FDA. Likewise, if a product was thought to be exempted from FDA regulation but the labeling indicates otherwise, it may result in regulatory action by the FDA or other agencies. For example, the Federal Trade Commission has taken action against 2 “melanoma detection” mobile applications for false and unsubstantiated claims.
“Melanoma Detection” App Sellers Barred from Making Deceptive Health Claims: FTC Charged Mole Detective Sellers with False Advertising Earlier this Year. Federal Trade Commission. Press Release August 13, 2015. https://www.ftc.gov/news-events/news/press-releases/2015/08/melanoma-detection-app-sellers-barred-making-deceptive-health-claims#:∼:text=%E2%80%9CMelanoma%20Detection%E2%80%9D%20App,Earlier%20this%20Year. Accessed September 28, 2022.
It is considered a matter of safety to ensure that claims are aligned with the requisite level of validation necessary to achieve FDA clearance or approval. Physicians play an important role in the generation of informational material and interpretation of how software will be used clinically.
The Future of FDA for SaMD
Although draft and final guidance documents are traditionally the FDA’s method for informing the industry of device requirements, the agency’s Digital Health Center of Excellence is engaging collaboratively to produce a regulatory framework for AL- or ML-enabled devices. This center of excellence acknowledges the unique positioning of innovative digital health products and seeks transparency in the developmental stage of generation of useful tools and frameworks for regulation of such products. The FDA has identified several priorities in the Artificial Intelligence/Machine Learning-Based Software as a Medical Device Action Plan, which was published in January 2021,
including the creation of a tailored regulation framework for this technology, consensus building for good ML practices, transparency to end users with patient-centered focus, and methodology for rigorous evaluation of algorithm bias and robustness. These FDA priorities require continued cross-functional collaboration among the FDA, industry, patient advocacy groups, international organizations, and physicians.
Conclusion
Physicians are critical stakeholders in driving the future of ML or AI. Many of the most interesting and impactful AI or ML applications in health care are regulated by the FDA because of their risk profile. Empowering physician innovators to both implement and influence FDA regulations will build a stronger foundation for physician-aligned and patient-centered AI or ML, which will enable the future of medical care.
Potential Competing Interests
The authors report no competing interests.
References
Benjamens S.
Dhunnoo P.
Mesko B.
The state of artificial intelligence-based FDA-approved medical devices and algorithms: an online database.
Proposed regulatory framework for modifications to artificial intelligence/machine learning (AI/ML)-based software as a medical device (SaMD). US Food & Drug Administration. https://www.fda.gov/media/122535/download. Accessed December 28, 2022.
“Melanoma Detection” App Sellers Barred from Making Deceptive Health Claims: FTC Charged Mole Detective Sellers with False Advertising Earlier this Year. Federal Trade Commission. Press Release August 13, 2015. https://www.ftc.gov/news-events/news/press-releases/2015/08/melanoma-detection-app-sellers-barred-making-deceptive-health-claims#:∼:text=%E2%80%9CMelanoma%20Detection%E2%80%9D%20App,Earlier%20this%20Year. Accessed September 28, 2022.
00004-4&id=gr1.jpg){kind=link}
00004-4&id=gr2.jpg){kind=link}