Most health tech teams reach the same point: the LLM feature works in a demo, product wants it in the next release, and someone asks whether real patient data can go through it. The honest answer is usually "not yet", because the model call is only one hop in a data path that also includes embeddings, a vector database, prompt logs, error tracking and an evaluation set someone built from production traffic.
This checklist is for CTOs, product leads and compliance owners who need to answer that question properly. It is general education about what HIPAA requires, not legal advice, and your privacy counsel should review the final design. It covers who is a business associate, what each contract must say, how much data to send, and what your security program has to show.
1. Is it PHI, and who becomes a business associate?
HIPAA applies to covered entities (health plans, clearinghouses and most providers) and to their business associates, meaning anyone who creates, receives, maintains or transmits protected health information on their behalf. If your product processes PHI for a hospital or clinic, you are almost certainly a business associate. Every vendor you pass that PHI to is then a subcontractor, and is a business associate in its own right.
Two points from HHS's cloud computing guidance catch teams out. A cloud provider that stores ePHI is a business associate even if the data is encrypted and it holds no key. And the "conduit" exception covers transmission-only services with transient access, not a service that stores or processes the data. A model API that processes your prompts is not a postal service.
What do I need to check before sending patient data to an LLM under HIPAA?
Confirm whether the data is PHI and whether each vendor that handles it is a business associate. HHS guidance says a cloud provider that stores ePHI is a business associate even if it cannot decrypt it, so you need a BAA with the model API, vector database, logging and monitoring vendors. Send only the minimum necessary, de-identify by Safe Harbor or Expert Determination where possible, get no-training and retention terms in writing, encrypt and log access, and update your Security Rule risk analysis to cover the new data flows.
2. A BAA with every vendor in the data path
Draw the full path a prompt takes, then list every system that touches PHI on the way in and on the way out. For a typical LLM feature that list is longer than the model provider.
- The model API, and the specific product tier or endpoint you call. Ask the vendor which services its BAA covers, because coverage often depends on the configuration.
- The embedding service and the vector database, since chunks of clinical text and their metadata live there.
- Logging, tracing and LLM observability tools that capture prompts and outputs.
- Error tracking and APM tools, which happily attach request bodies to stack traces.
- Evaluation and labeling platforms, if real encounters are used to build test sets.
- Backups, caches and queues, plus support tools where staff paste screenshots.
The Privacy Rule sets what a BAA must contain. Two terms matter most for LLM work: the business associate must make any subcontractors agree to the same restrictions, and at the end of the contract it must return or destroy the PHI it still holds, if feasible. A vendor that cannot say how it deletes your data, including from logs and backups, cannot really sign that term.
3. Minimum necessary, applied to prompts
The minimum necessary standard asks you to limit uses, disclosures and requests of PHI to what the purpose needs. There are exceptions, the main one being disclosures to a provider for treatment, but most product features (summaries for operations staff, coding support, patient messaging drafts) should be designed as if the standard applies.
- Decide which fields the feature needs, and build the prompt from those fields rather than pasting whole records.
- Strip identifiers the model does not need to do the task, such as names, MRNs and contact details.
- Scope retrieval to the patient and the user in front of the screen. Our guide to per-user access control in RAG covers how to enforce this at query time, not in the prompt.
- Write the rule down in your policies, because the standard expects documented policies, not good intentions.
4. De-identification: Safe Harbor or Expert Determination
Properly de-identified data is not PHI, which makes it the cleanest option for evaluation sets, prompt tuning and analytics. HIPAA recognises two methods.
- Safe Harbor: remove 18 listed identifiers of the patient and their relatives, employers and household members, including names, geography smaller than a state, all date elements except the year, contact details, record and account numbers, biometric identifiers such as voice prints, and full-face photographs. You must also have no actual knowledge that what remains could identify someone.
- Expert Determination: a qualified expert applies statistical methods, concludes that the risk of identification by an anticipated recipient is very small, and documents the methods and results.
Free text is where both methods get hard. Clinical notes and transcripts mention names, dates and places in ways that automated scrubbers miss, so test your de-identification pipeline on real samples before you rely on it, and treat its output as PHI until you have.
5. Retention and no-training terms, in writing
A BAA says how a vendor may use PHI. It does not always answer the questions that matter for an LLM vendor, so get these answered in the contract or its referenced terms.
- Will prompts, outputs or files be used to train or improve any model? The answer you want is no, by default and in writing.
- How long are prompts and outputs retained, including for abuse monitoring, and can retention be set to zero for your account?
- Where is the data processed and stored, and which subprocessors are involved?
- How are deletion requests handled, and is deletion confirmed?
6. Encryption, access controls and audit logs
The current Security Rule requires unique user identification and audit controls that record and examine activity in systems containing ePHI. Encryption is an "addressable" specification today, which does not mean optional: you implement it, or document why an equivalent measure is reasonable. For an LLM pipeline, encrypt in transit and at rest everywhere PHI sits, including the vector store and any prompt cache, give services their own credentials with the least access they need, and make sure audit logs show who triggered each model call on which record.
7. Update the risk analysis for the new data flows
A risk analysis covering all ePHI you create, receive, maintain or transmit is a required Security Rule specification, and OCR has run a Risk Analysis Initiative focused on enforcing it. Adding an LLM feature creates new places where ePHI lives, so the analysis needs a fresh pass before launch.
- Map the new flow end to end, including retries, fallbacks to a second model provider and batch jobs.
- List the new assets: embedding stores, caches, prompt logs, evaluation datasets.
- Add LLM-specific threats, such as prompt injection that pulls another patient's data into an answer. A prompt injection testing checklist is a good starting point.
- Rate each risk, record the safeguard, and assign an owner for anything left open.
8. Logging without storing PHI
Teams need traces to debug and evaluate LLM features, and full prompt logs are the fastest way to copy PHI into a tool that has no BAA. Log identifiers, token counts, latency, model version, retrieval document IDs and evaluation scores, and keep raw text out of general-purpose tools. If raw text must be kept for a period, keep it in a store covered by your BAA, with short retention and restricted access. Our guide on what to log from an LLM app without storing personal data goes through the fields. Decide this when you design the LLM infrastructure, because retrofitting redaction into a pipeline that already logs everything is much harder.
9. Patient-facing disclosures
HIPAA itself does not set a disclosure rule specific to LLMs, but state law may. California, for example, enacted AB 3030 in 2024: providers that use generative technology to write patient communications about clinical information must include a disclaimer and tell patients how to reach a human, unless a licensed provider has read and reviewed the message. Check the states you operate in, check that your notice of privacy practices still describes how you use data, and design the review step so it is real rather than a click-through.
What changed in 2025 and 2026
HHS published a proposed overhaul of the Security Rule in the Federal Register on January 6, 2025, and comments closed on March 7, 2025. It is still a proposal: no final rule has been published, the 2026 Unified Agenda lists it as a long-term action with a target of July 2027, and HHS says the current rule remains in effect meanwhile. Among other things, it would remove the required versus addressable distinction, require encryption of ePHI at rest and in transit and multi-factor authentication (each with limited exceptions), and require a technology asset inventory and a network map of how ePHI moves. If you are designing an LLM pipeline now, building to those proposals costs little and avoids rework if they become final.
If you cannot draw where a prompt goes, you cannot sign for where the PHI went.
Where Octalcode fits
Octalcode builds LLM applications, SI agents and RAG systems, and our MLOps and SI infrastructure work is where most of this checklist lives: data-path mapping, BAA-ready vendor choices, logging that keeps raw text out of the wrong tools, and evaluation without real identifiers. We sign an NDA before any client data and limit access to the project team.
We describe our work as HIPAA-grade, not as a certification, and your counsel and compliance team stay the final word. If you are scoping a feature for a provider or plan, our healthcare SI page explains how we approach it.