Most health systems assume federal regulators are watching how they use AI. They aren't. The FDA regulates the companies that build AI-enabled devices; it does not inspect a hospital's sepsis model, ambient scribe or prior-authorization bot. That leaves the real work of AI governance in healthcare with the healthcare organization itself, under HIPAA, nondiscrimination rules, state statutes and accreditor expectations that have all moved in 2026. This guide is for the CMIO, CIO or compliance director who needs to know what the agency covers, what it doesn't, which compliance requirements actually apply to providers, and what evidence an auditor or surveyor will ask to see. If you're among the healthcare organizations deploying AI at scale faster than you can document it, this is where the AI governance gap shows up first.
How does the FDA regulate AI in healthcare?
The FDA regulates AI and machine learning software when it meets the legal definition of a medical device: software intended to diagnose, treat, mitigate or prevent disease. That includes software as a medical device, such as imaging AI that flags a suspected stroke, and AI built into hardware. Before any of these products can be sold, the manufacturer goes through FDA premarket review, usually a 510(k) clearance or De Novo authorization, and the agency keeps a public list of authorized products that now runs well past a thousand entries, most of them in radiology.
The agency has also adapted its tools for the FDA AI era. Its final guidance on a predetermined change control plan for AI-enabled device software, in its current August 2025 version, lets a manufacturer get authorization in advance for how a model will be retrained and updated, so each planned change doesn't need a new submission. A broader draft guidance on managing AI-enabled devices across their lifecycle, issued in January 2025, is still not final.
Notice who all of this binds: the manufacturer. Those rules shape what you can buy, and the documentation you can ask for. They don't tell a hospital how to run its AI program.
The FDA regulates AI device makers, not your use of AI
There is no federal inspection of a hospital's AI use, no registration for clinicians who rely on AI tools and no rule requiring a hospital AI committee. Once a cleared AI-enabled medical device is installed, how your clinicians use it, which patients it runs on and how you monitor it are governed by your own policies, your medical staff bylaws, state law and your accreditor, not by the agency.
The exception is when a health system becomes a manufacturer. If your data science team builds diagnostic AI systems and markets them to other hospitals, or significantly modifies a cleared device beyond its labeling, you may take on manufacturer obligations. Internal AI development for your own patients occupies a gray zone that most systems manage with conservative policy rather than a regulatory filing; get regulatory counsel involved before any model leaves your walls.
This is the core of the regulatory framework CMIOs need to internalize. Device clearance answers the question "is this product safe and effective as designed?" Your governance process has to answer "is it safe and effective here, for our patients, in our workflows, today?"

Which clinical AI tools fall outside FDA oversight?
More than most people expect. Under the 21st Century Cures Act, clinical decision support software is not a device if it meets four criteria, including that clinicians can independently review the basis for its recommendations. The FDA clinical decision support guidance, updated in January 2026, loosened the agency's earlier reading: it now applies enforcement discretion to some software that offers a single recommendation, while still treating tools that analyze images or signals, or that drive time-critical decisions, as devices.
Large categories of AI applications sit outside device regulation entirely. Ambient documentation and other AI medical scribes, generative AI that drafts patient messages, revenue-cycle and prior-authorization automation, scheduling and capacity tools, and most operational AI agents are not medical devices. Neither are many predictive models that put AI in clinical workflows through the EHR, such as deterioration or readmission scores, depending on how they're presented to clinicians.
That is the biggest gap in healthcare AI. The tools with the most direct effect on daily care, and the most exposure to protected health information, are often the ones no regulator has reviewed. Oversight of AI like this falls entirely to your governance process.
What to require from FDA-cleared AI vendors: change-control plans, labeling and performance data
For AI devices that are regulated, use the clearance documentation as procurement leverage. Ask each vendor for the decision summary, the device labeling and indications for use, and the validation data behind them: which patient populations, sites and equipment the model was tested on. If the model is updated over time, ask whether it has an authorized predetermined change control plan and what changes it allows without a new submission.
Then ask what clearance doesn't cover. How does AI performance vary by age, sex, race and ethnicity, and across scanner models? What monitoring does the vendor provide after go-live, and will they share drift alerts? What happens to your data, and is any of it used for further AI development? Put the answers in the contract: performance commitments, notification of model changes, audit rights and a business associate agreement.
An FDA-cleared AI solution can still fail in your environment if your patient population, imaging protocols or documentation habits differ from the validation set. Treat clearance as the minimum bar for regulated products, not as a substitute for local validation.

The rules that do apply to providers: HIPAA, ONC decision-support transparency, Section 1557
HIPAA compliance applies to every AI system that touches PHI, whether or not FDA regulates it. That means a risk analysis covering the vendor's environment, a business associate agreement, minimum-necessary data flows, and access controls on prompts, outputs and logs. HIPAA and FDA requirements overlap only at the edges; most of the work sits on the HIPAA side. HHS's proposed Security Rule update remains unfinalized, with the latest regulatory agenda pointing to 2027, but the existing rule already covers AI.
ONC's HTI-1 rule requires certified EHR developers to disclose source attributes for predictive decision support interventions: what data a model uses, how it was validated and how it should be used. Those disclosures are required for now, but ONC's HTI-5 final rule, released in October 2026, removes those AI "model card" requirements effective December 28, 2026. After that, the transparency your EHR vendor provides becomes something you'll have to request yourself.
Section 1557 of the Affordable Care Act adds a nondiscrimination duty. Under 45 CFR 92.210, covered entities must make reasonable efforts to identify patient care decision support tools that use race, color, national origin, sex, age or disability as inputs, and mitigate the risk of discrimination, with compliance required since May 1, 2025. That covers algorithms and AI models alike, FDA-cleared or not.
What do state AI laws require from healthcare organizations?
State legislatures fill much of the space federal rules leave open. California requires disclaimers on AI-generated patient communications that a clinician hasn't reviewed, and requires health plans to leave final medical-necessity decisions in utilization review to a licensed physician or qualified professional. Texas requires practitioners to review AI-generated records and, under its broader AI law in effect since January 2026, to disclose AI use in care. Illinois bars AI from delivering therapy.
Colorado repealed its original AI Act before it took effect and replaced it with a narrower law, effective January 1, 2027, that largely exempts HIPAA-covered entities outside employment and financial-assistance decisions. There is no comprehensive federal statute on AI in health care. Instead, a December 2025 executive order directed the Justice Department to challenge state rules it considers burdensome, so expect litigation, not quiet.
The practical takeaway: maintain a state-by-state register for every state where you deliver care or telehealth. Regulatory compliance in AI is now a moving, multistate obligation, and a single national policy rarely satisfies all of them. Systems with European patients or partners should also track the EU AI Act, which pushed its rules for AI in medical devices to August 2028.
The Joint Commission and CHAI guidance: seven elements
In 2025, the Joint Commission and the Coalition for Health AI published guidance on responsible AI use built on seven elements: AI policies and governance structures; patient privacy and transparency; data security and data use protections; ongoing quality monitoring; voluntary, blinded reporting of AI safety-related events; risk and bias assessment; and education and training. It is the closest thing the industry has to an accreditor's checklist.
In 2026 it became more concrete. CHAI released governance playbooks in May, and the Joint Commission then launched a voluntary certification for the responsible use of AI, assessing governance, risk and bias reduction, data management, monitoring of safety and effectiveness, and education and training. It isn't required for accreditation, but it signals what surveyors will increasingly look for.
For most health systems, the seven elements map neatly onto structures you already run: a governance committee, privacy and security reviews, quality monitoring and safety-event reporting. As AI adoption grows, the work is making AI explicit in each one, and making sure AI governance committees have clinical, legal, IT and patient-safety voices at the table.

The evidence pack an auditor or accreditor will ask for
When a surveyor, auditor or regulator asks about AI, they will want documents, not intentions. Build a standing evidence pack, the backbone of healthcare AI compliance: an inventory of all AI in use, clinical and operational, with its owner, purpose, data sources, FDA status and risk tier; the policy and committee charter governing AI use; and minutes showing approvals, conditions and rejections.
For each high-risk tool, add the pre-deployment validation results, the Section 1557 bias review, the HIPAA risk analysis and BAA, vendor documentation (clearance summary, labeling, change-control plan), clinician training records and patient-notice language where state law requires it. Then add monitoring: performance reports, drift reviews and any AI safety events with their follow-up.
Responsible AI governance is mostly record-keeping discipline; it's how you govern AI systems in practice. Every AI tool should be traceable from request to retirement, and AI governance requires a named human owner for each one. Our AI governance framework for healthcare walks through the committee structure, and our healthcare AI and cyber governance framework covers the security controls that sit beside it.
Where does the NIST AI RMF fit in a hospital AI governance program?
NIST’s AI Risk Management Framework is voluntary, but it gives you the operating model the rules above assume. Its four functions, Govern, Map, Measure and Manage, translate directly: govern with policy and roles, map each AI use case and its risks, measure performance and bias, and manage AI risks through controls and monitoring. NIST's AI RMF page also hosts a profile for generative models, which addresses hallucination, data leakage and misuse.
The value is consistency. Using one framework for AI at every stage of the AI lifecycle, from vendor selection to retirement, lets you apply the same questions to a cleared imaging algorithm, an ambient scribe, an internal readmission model and enterprise AI tools like chat assistants. Organizations that want an external certification can pair it with ISO/IEC 42001, the international standard for an AI management system.
Our NIST framework explainer covers the functions in depth. If you need outside help with the privacy side, our guide on how to choose a HIPAA consultant explains what to look for.

Monitoring AI after go-live: drift, bias and safety events
Validation at go-live is a snapshot. Clinical AI systems drift as patient mix, documentation habits, coding and equipment change, and a model that worked last year can quietly degrade. Set monitoring for every high-risk tool: performance against outcomes, alert and override rates, subgroup performance and data-quality checks on inputs. Review it on a schedule, and give the committee authority to pause healthcare AI systems that underperform.
Bias monitoring belongs in the same dashboard. Section 1557 makes it a legal duty for decision support tools that use protected characteristics, and the Joint Commission guidance makes it an accreditation expectation. Look at outcomes, not just inputs: a model can avoid race as a variable and still perform worse for some groups.
Finally, treat AI failures as safety events. Add an AI category to your event-reporting system, train staff to use it, and feed reports back to the vendor and the committee. That is how AI systems remain safe after the vendor's validation study ends, and it's the evidence that shows surveyors your program works.
Frequently asked questions
Does the FDA regulate AI used in hospitals?
The FDA regulates AI products that meet the definition of a medical device, through premarket review of the manufacturer. It does not inspect how hospitals use AI; that falls to your own policies, state law and your accreditor.
Which healthcare AI tools are not regulated by the FDA?
Ambient scribes, generative AI that drafts patient messages, revenue-cycle and prior-authorization automation, scheduling tools and many EHR risk scores generally sit outside device regulation, depending on how they are presented to clinicians.
What rules apply to healthcare providers using AI?
HIPAA applies to any AI that touches PHI, Section 1557 requires reasonable efforts to identify and reduce discrimination risk in patient care decision support tools, and state laws in California, Texas, Illinois and Colorado add their own duties.
What does the Joint Commission expect for AI governance?
Its 2025 guidance with CHAI sets seven elements: governance structures, patient privacy and transparency, data security, ongoing quality monitoring, AI safety-event reporting, risk and bias assessment, and education and training. A voluntary certification followed in 2026.
What evidence should a hospital keep for AI audits?
An inventory of all AI in use with owners and risk tiers, the AI policy and committee charter, approval minutes, and for each high-risk tool its validation results, bias review, HIPAA risk analysis, BAA, vendor documentation, training records and monitoring reports.
Ready to make your AI governance audit-ready?
VisioneerIT focuses on AI governance for healthcare. Our AI governance and compliance team helps health systems inventory AI, stand up governance committees, and build the evidence pack auditors and the Joint Commission expect. Our Chief AI Officer advisory gives you senior AI leadership without a full-time hire, and our cybersecurity consulting group secures the data flows behind every model. We work across healthcare systems of every size. Talk to our team about a governance gap assessment.
Key Things to Remember
- The FDA regulates AI device makers through premarket review; it doesn't inspect how hospitals use AI.
- Many high-impact tools, including ambient scribes and generative patient messaging, fall outside FDA oversight entirely.
- Use clearance documentation (labeling, validation data, change-control plans) as procurement leverage, then validate locally.
- HIPAA and Section 1557 apply to providers now; ONC's HTI-5 final rule drops HTI-1's AI model-card requirements from December 28, 2026.
- State rules in California, Texas and Illinois are in effect, and Colorado's narrower law starts January 1, 2027.
- The Joint Commission and CHAI's seven elements, and the 2026 voluntary certification, define what surveyors will look for.
- Keep a standing evidence pack and monitor every high-risk tool for drift, bias and safety events.

