Short answer: AI on medical data is a pipeline with seven questions, and most failures come from skipping one. Where does the data come from (a FHIR record, documents, devices, the patient's own words)? How does the model reach it (a context API that assembles what is relevant, a Model Context Protocol server, retrieval over the record)? Who decided it may (the patient's consent, scoped by purpose and time, logged)? What protects the people in it (de-identification for anything that leaves the care context, synthetic data for development, pseudonymisation for research)? Can every output be traced (provenance on inputs and outputs)? How do we know it works (evaluation against clinical ground truth, not vibes)? Which rules apply (GDPR Article 9 and 22, the MDR if the output informs clinical decisions, the EU AI Act, HIPAA)? This page is the map; each question has its own page.
The pipeline
The pipeline for AI on medical data has seven stages, each with a question: source (where the data comes from), access (how the model reaches it), control (who decided it may), privacy (what protects the people in it), traceability (whether outputs can be traced), evaluation (how success is measured) and rules (which laws apply).
| Stage | Question | What good looks like | Page |
|---|---|---|---|
| Source | Where does the data come from? | A FHIR record with codes (LOINC, SNOMED CT, ATC), documents with structure extracted, device data with provenance; not screenshots and PDFs alone | FHIR Observation, FHIR DiagnosticReport |
| Access | How does the model reach it? | A context API or MCP server that returns a bounded, provenance-labelled slice of the record; retrieval with structured filters, not a dump | RAG on medical records, Healthcare API for AI, MCP server |
| Control | Who decided it may? | Patient consent per purpose and duration; scopes enforced server-side; an access log the patient reads; revocation that works | Patient consent platform, Who viewed my record? |
| Privacy | What protects the people? | Identified data stays in the care context; de-identified or synthetic data for development, analytics and model training; pseudonymised data for research under permits | De-identification, Synthetic patient data |
| Traceability | Can outputs be traced? | Provenance on every input (who recorded it, when, from where) and on every AI output (model, version, inputs used), stored with the record | Medical data provenance |
| Evaluation | How do we know it works? | Test sets with clinical ground truth, faithfulness to sources, error analysis by subgroup, monitoring in production, human review paths | Evaluating clinical LLMs |
| Rules | Which laws apply? | GDPR Articles 9 and 22, DPIA; MDR if the output informs diagnosis or treatment; the AI Act's high-risk regime for devices and for listed uses; HIPAA in the US | The EU AI Act and health AI, SaMD classes, GDPR Article 9 |
Three kinds of AI, three risk profiles
AI on medical data comes in three kinds with three risk profiles: assistive and administrative AI, such as plain-language summaries and drafted notes, where the risk is privacy and uncaught errors; decision support, such as flagging interactions or triaging, where the risk is clinical and the MDR applies; and population and research AI, where the risks are re-identification and bias.
Assistive and administrative: summarising a record for the patient in plain language, drafting a visit note from a transcript for the clinician to edit, answering "where is my last blood test" over a consented record. Risk sits in privacy and in errors a human may not catch; the MDR does not apply as long as no clinical decision is informed, but the AI Act's transparency duties and the GDPR do.
Decision support: flagging an interaction, suggesting a differential, triaging. Risk is clinical; the MDR applies (usually class IIa or higher), the AI Act's high-risk regime follows, and IEC 62304 and clinical evaluation are required. See CDS Hooks explained for how such systems plug into an EHR.
Population and research: finding patterns across many records, pharmacovigilance signals, cohort discovery. Risk is re-identification and bias; the data must be anonymised or pseudonymised under permits, aggregated with minimum cell sizes, and the methods evaluated for confounding. See Secondary use of health data.
The failure modes
The common failure modes of AI on medical data are stuffing an entire record into a prompt, prompt injection through documents, silent inference that writes a model's guess into the record as if a clinician had, training on identified data without a legal basis, evaluating only on the vendor's demo set, and debugging logs full of patient data.
- Context stuffing: pasting an entire record into a prompt. Costly, leaks data, and the model attends to the wrong parts. A bounded, ranked context with provenance works better and is auditable.
- Prompt injection through documents: a scanned letter that says "ignore previous instructions". Treat all record content as data; never let it set policy. See RAG on medical records.
- Silent inference: a model inferring a diagnosis from behaviour and writing it into the record as if a clinician had. Outputs are labelled as AI-generated with provenance and never become a Condition without a human assertion. See FHIR Condition explained.
- Training on identified data without a basis: fine-tuning on patient records needs a legal basis beyond "we had the data"; de-identified or synthetic data usually serves.
- Evaluation on the vendor's demo set: evaluate on your population, your languages, your document types, and keep evaluating after deployment.
- Logs full of PHI: prompts and completions logged for debugging become a second, unprotected record. Pseudonymise or do not log content.
Standards and building blocks
The standards and building blocks for AI on medical data are FHIR for the record and for structured AI outputs, the Model Context Protocol for governed access to tools and data, SMART on FHIR for authorisation, HL7's AI/ML data lifecycle work for traceability, ISO/IEC 42001 for AI management, and the AI Act's harmonised standards.
FHIR for the record and for the AI's structured outputs (Observation with a derivedFrom, DocumentReference for generated summaries, Provenance with the model as agent); the Model Context Protocol for giving assistants governed access to tools and data; SMART on FHIR for authorisation; the HL7 AI/ML Data Lifecycle and provenance work for traceability; ISO/IEC 42001 for an AI management system; the AI Act's harmonised standards as they are published.
In Anpheros
For AI on medical data, Anpheros provides the FHIR R4 record with codes and provenance, a context API that returns a provenance-labelled slice of the record within a token budget, an MCP server through which assistants read a person's own record with that person's consent, and consent and access logs the person controls. Anpheros does not run diagnostic AI.
Anpheros provides the record (FHIR R4, coded, with provenance), a context API that returns a provenance-labelled slice within a token budget, an MCP server so assistants such as Claude read a person's own record with that person's consent, and consent and access logs the person controls. It does not run diagnostic AI and does not write AI inferences into the record as clinical facts. The developer site has the details: healthcare API for AI, MCP server, LLMs on healthcare data.
Related pages
- RAG on medical records: how retrieval over a FHIR record works
- The EU AI Act and health AI
- Healthcare API for AI: provenance-labelled patient context
- De-identification of health data
- AI and your medical record (for patients)
- Secondary use of health data
- Anpheros MCP server (developer site)
- LLMs on healthcare data (developer site)
Sources
- Model Context Protocol specification
- HL7 FHIR R4: Provenance
- Regulation (EU) 2024/1689 (AI Act), EUR-Lex
- ISO/IEC 42001:2023, AI management system
- WHO: Ethics and governance of artificial intelligence for health, guidance on large multi-modal models (2024)
- HL7 SMART App Launch
Frequently asked questions
Can I fine-tune a model on patient records?
Only with a legal basis for that purpose (consent, or a research basis under Article 9(2)(j) with safeguards), a DPIA and usually de-identification; identified records cannot be used for training because they happened to be available. Synthetic and de-identified data serve most development needs.
Is an AI summary of a record a medical device?
Not if it restates what the record contains for the patient or clinician without interpreting it for diagnosis or treatment. The moment it suggests what the findings mean clinically, it is decision support under the MDR.
What is MCP and why does it matter here?
The Model Context Protocol is an open standard for connecting AI assistants to tools and data sources through servers with defined capabilities. For health data it lets an assistant read a person's record through a governed server, with the person's consent and logging, instead of pasting data into a chat.
How do I evaluate a clinical LLM feature?
With a held-out set of real cases from your population labelled by clinicians, metrics for correctness and faithfulness to the sources, analysis by subgroup and language, review of failure cases, and monitoring after launch. Vendor benchmarks are a starting point, not evidence.
Try it free in the sandbox
Your first FHIR call in 5 minutes. An HL7 FHIR R4 API with the patient's consent built in: try it in the sandbox, then connect your own app or clinic system.
Free · 30 synthetic patients · no card
Anpheros builds a medical record that the patient controls and can share with any clinic, on an HL7 FHIR R4 platform, based in the EU. These guides help you understand and organise your health; they do not replace your doctor.
About Anpheros →
AnpherosThe more you log, the clearer your record
Your health history in one place, under your control.
Free · Works in your browser · No card
