LLM privacy: data flows, duties and controls

LLM privacy work starts with a list of copies. One message to an LLM feature creates copies in the prompt, the retrieved context, the tool results, the completion, the trace, the evaluation set and the provider's abuse log. This guide lists each copy, the GDPR duty that applies to it, and the controls to put in place before launch.

Data flow from User prompt through Redaction and LLM provider to Completion, with copies in traces and logs, eval datasets and provider logs

Opsware designs and implements logging, retention, redaction and evaluation for LLM applications as part of its LLMOps work.

It follows one request from the user to the model provider and back, and it is written for privacy leads and the engineers they work with. Five companion articles cover the details:

Nothing here is legal advice, because how these rules apply to your feature depends on your purposes, your contracts and the countries you serve.

The data flow, stage by stage

Take a support assistant that uses retrieval. When a customer types a question, the app pulls the customer's account record and some help-center articles, then calls a hosted model. The model may call a tool to look up an order or open a ticket, and the app returns the answer while an observability SDK records the whole exchange. The table shows where personal data lands at each stage.

StagePersonal data presentWhere it is storedTypical lifespan
User promptAnything the user typed: names, account numbers, health details, other people's informationYour chat history, the request to the model, every log in betweenAs long as chat history is kept
System prompt and injected contextCustomer profile fields, CRM notes and conversation summaries that the app addsBuilt per request, then copied into traces and provider logsThe same as traces and provider logs
RAG documents and embeddingsSource documents such as tickets, contracts, emails and wiki pages; embeddings of their chunks; metadata such as author or customer IDVector database, object storage for source files, sometimes a vector store hosted by the providerUntil someone deletes or re-indexes
Tool inputs and outputsOrder records, account lookups, search results, anything an API returns to the modelThe request payload, traces, agent memoryThe same as traces; agent memory may have no limit
CompletionThe model's answer, which can repeat or infer personal data, sometimes wronglyChat history, traces, provider logs, cachesThe same as chat history and traces
Traces and logsFull prompts, completions, tool payloads, user IDs, session IDs, IP addresses, feedback commentsTracing backend, application logs, log aggregation, data warehouseOften the tool's default retention, which nobody chose
Evaluation datasetsProduction conversations copied in as test cases, with human labels and commentsEvaluation tool, spreadsheets, git repositoriesNo limit unless someone sets one
Fine-tuning data and weightsTraining examples taken from production, and anything the model memorized from themTraining files at the provider or in your storage; the trained modelAs long as the model is used
Provider-side copiesPrompts and completions kept for abuse monitoring; stored responses, files and threads; prompt cachesThe model provider and its sub-processorsSet by the provider's terms, often up to 30 days for abuse logs; stored files and threads stay until you delete them

One sentence a user types is copied into most of the stores in this table during a single request. Traces, evaluation sets and fine-tuning files usually live longest, often longer than chat history, because nobody set a retention rule for them.

If you keep a data map, give each of these stores its own entry, not one line for "AI assistant". The guide “Data mapping and discovery” explains how to structure those entries.

Which GDPR duties apply where

The GDPR has no AI chapter, so AI data privacy for an LLM feature means applying its ordinary rules to each copy. Check each stage against six topics: lawful basis, minimization, retention, requests, processors and transfers, and impact assessments.

Lawful basis and purpose

Answering a user's question with their own data usually rests on contract necessity or legitimate interest, the same basis as the underlying service. Secondary uses are harder:

  • copying production conversations into evaluation sets;
  • using conversations to fine-tune a model;
  • letting the provider train on them.

Each secondary use is a new purpose, which needs its own basis and must be compatible with what users were told under Articles 13 and 14 of the GDPR. The EDPB's Opinion 28/2024 on AI models treats development and deployment as separate phases, so if you rely on legitimate interest, each phase needs its own assessment.

Data minimization

Every field the app adds to a system prompt is disclosed to the model provider and copied into your traces. Minimization in an LLM app starts with the prompt: send the order status, not the whole customer record, and retrieve the three relevant chunks, not the whole ticket history. It also applies to logging. The article “PII redaction for LLM prompts and traces” explains how to redact data before it leaves your process.

Retention

Storage limitation under Article 5(1)(e) applies to traces and evaluation sets as it does to the CRM. Set a time limit for each store, and check that the provider's retention for the same content matches what your privacy notice says. The article “Zero data retention and DPAs at model providers” compares provider retention defaults and zero-retention options.

Access, deletion and correction requests

An access request covers chat history, traces, evaluation rows and anything else linked to the person. A deletion request covers the same stores, plus vector stores and, in some cases, models. Chatbots also raise correction requests, because a person may ask for a false statement the model makes about them to be corrected. The article “Access and deletion requests against LLM logs and vector stores” covers the LLM stores, and “DSAR automation: building the data subject request workflow” covers the general request workflow.

Processors, sub-processors and transfers

A hosted model provider that acts on your instructions is a processor and needs a data processing agreement (DPA) under Article 28. The guide “Data processing agreement (DPA): what it must contain” lists what that contract must contain. The observability vendor, vector database host, evaluation platform and gateway vendor are processors too, and each has its own sub-processor list, as the guide “Subprocessor lists and change notices” explains.

If any of these vendors process data outside the European Economic Area (EEA), you need a transfer tool under Chapter V of the GDPR, such as an adequacy decision, standard contractual clauses or another Article 46 safeguard. For certified US companies, the adequacy decision is the EU-US Data Privacy Framework, which the General Court upheld on September 3, 2025 (Latombe v Commission, T-553/23). The applicant appealed to the Court of Justice on October 31, 2025 (C-703/25 P), and the appeal was still pending in October 2026. Keep standard contractual clauses in your contracts so transfers stay covered if the Court of Justice annuls the framework.

A model provider can also act as a controller for part of the processing, which means it decides its own purposes for that part. Several providers keep the right to use content for abuse monitoring or, on some plans, for training. For example, section 2.3 of Mistral's DPA (effective July 27, 2026) lists these purposes for which Mistral acts as controller:

  • model training, unless the customer has opted out;
  • automated abuse monitoring on its APIs, which does not apply when zero data retention is on;
  • conversations attached to thumbs-up or thumbs-down feedback.

Name these provider purposes in your privacy notice and in your records of processing.

Impact assessments

Article 35 requires a data protection impact assessment (DPIA) where processing is likely to result in a high risk. Supervisory authorities screen for criteria such as large-scale processing, new technology and sensitive data. An LLM feature often meets several of them, because it processes customer conversations at scale and receives special-category data that users volunteer. Writing a DPIA also makes the team list every store in the table above, so a DPIA is useful even where the law does not require one.

Some LLM features are also high-risk AI systems under the EU AI Act, and some deployers of those systems must also carry out a fundamental rights impact assessment. The article “Where the EU AI Act and GDPR overlap” explains which deployers, from which date, and how the two assessments fit together.

Common AI privacy concerns in LLM features

Public discussion of AI and privacy concerns is mostly about model training, but privacy reviews of LLM features more often find these gaps:

  • Tracing records everything. Many LLM observability SDKs record full inputs and outputs unless told otherwise. The OpenTelemetry GenAI conventions make message content opt-in, because it "is likely to contain sensitive information including user/PII data". Vendor SDKs do not always follow that default, so check what yours records.
  • Evaluation sets copied from production. A developer copies fifty real conversations into a test suite to reproduce a bug. Two years later the conversations are still in a git repository with no retention rule, and nobody finds them during a deletion request.
  • Stateful provider features. Zero data retention (ZDR) does not cover endpoints that store data between requests. OpenAI's documentation lists Conversations, Assistants, Threads, Vector stores, Files, Fine-tuning, Evals and Batches among the endpoints that are not eligible (OpenAI, Your data, October 2026). A hosted vector store keeps your documents until you delete them.
  • Residency that covers only part of the flow. Providers treat "where inference runs" and "where data is stored" as separate settings. OpenAI's regional options in October 2026 show the split, with the Europe region covering storage and processing while several other regions cover storage only. Read the specific setting for each feature you use.
  • Agent memory. An agent that saves "memories" about users across sessions builds a profile store, which is often missing from the data map.
  • Feedback buttons. A thumbs-down with a comment can send the whole exchange to the provider under different terms. Anthropic's retention article says it keeps data from feedback submissions for five years (Anthropic Privacy Center, October 2026). Anthropic's default retention for API inputs and outputs is 30 days.

An LLM privacy checklist for launch

Use this checklist as a review before launch, and run it again when the feature adds a data source, a tool or a provider.

  1. Inventory every store. List chat history, traces, application logs, the vector database, the source document bucket, evaluation sets, fine-tuning files, agent memory, caches and objects stored at the provider. Give each store an owner, a location, a retention rule and a way to find one person's data in it.
  2. Record purposes and bases. Serving the user, quality evaluation, fine-tuning and provider abuse monitoring are separate lines in the record of processing activities (ROPA), and the privacy notice should match.
  3. Minimize prompts. Review what the system prompt and the retrieval layer add, and remove fields the model does not need. Filter retrieval by the user's permissions so one customer's documents never reach another customer's prompt.
  4. Redact or pseudonymize before logging. Decide whether traces need raw content at all. If they do, mask identifiers in the SDK before export, and keep any re-identification key in a separate vault with its own access control. Masked traces are still personal data while you hold the key, as “Pseudonymization vs anonymization under GDPR” explains.
  5. Set retention for each store. Give raw trace content a short limit and aggregated metrics a longer one. Review and prune evaluation sets on a schedule, and set expiry on hosted files and vector stores where the provider supports it.
  6. Choose provider settings. Confirm that training is off for your plan and decide whether you need zero data retention or modified abuse monitoring. Pick the regional endpoint, and check which features you use fall outside those settings.
  7. Sign the vendor contracts. Put a DPA in place with each model provider, observability vendor, vector database host and gateway vendor, review their sub-processor lists and subscribe to changes. Put a transfer tool in place for each flow outside the EEA.
  8. Make requests answerable. Tag every stored record with a stable user or customer ID so access and deletion run as queries rather than text searches. Write down how a deletion reaches embeddings, evaluation sets and backups.
  9. Assess. Run a DPIA where the screening criteria are met, and classify the feature under the AI Act: provider or deployer, any high-risk use, and which Article 50 transparency duties apply.
  10. Secure the stores. Traces and evaluation tools hold the most concentrated personal data in the stack, and they are often the least protected. Apply the access control, SSO, audit logging and encryption you use for a production database. The guide “GDPR Article 32 and security controls for personal data” covers the baseline.
  11. Test for LLM data leakage. Before launch, try to make the assistant reveal another user's data, the fields injected into the system prompt, or documents the user should not see. Repeat the test after significant changes to prompts or retrieval.

Questions that come up

Is a vector embedding personal data?

Treat it as personal data if it was made from text about an identifiable person and is stored with metadata or source text that links it to them. Research shows embeddings can be turned back into much of their source text, and Morris et al. (2023) recovered 92% of short inputs (32 tokens, about two dozen words) exactly in their test setup. Most RAG systems also store the chunk text next to the vector.

Does zero data retention mean we don't need a DPA?

No. The provider still processes personal data in memory to generate each response, and zero-retention terms keep exceptions for legal requirements and flagged abuse. You still need an Article 28 contract and, for processing outside the EEA, a transfer tool.

Do we need consent to use an LLM on customer data?

Not as a general rule. The lawful basis follows the purpose, and serving the customer usually rests on contract or legitimate interest. Consent or another specific condition may be needed for special-category data, for training on customer data, and in some ePrivacy situations.

Our provider says it never trains on API data. Is that enough?

That statement covers only one risk. You also need to know the retention period, who can read retained content, where it is processed, which sub-processors are involved, and what happens with stateful features. The article “Zero data retention and DPAs at model providers” answers those questions for each provider.

Does self-hosting the model solve the privacy questions?

It removes the model provider from the flow, but your own traces, vector stores and evaluation sets still hold personal data, and every GDPR duty still applies to them. The article “Self-hosted LLM stack for data residency” explains what each layer changes.

Has the GDPR changed for AI?

Not yet. The Commission's Digital Omnibus proposal of November 2025 would change the definition of personal data and add rules on legitimate interest for AI development. In October 2026 the Parliament's committees had not yet voted on it. The article “Where the EU AI Act and GDPR overlap” tracks the proposal's status.

Need this built?

If you want this implemented rather than read about it, tell us about your project. We will reply with questions or a rough plan.

Privacy engineering notes by email

When we publish a new guide on data mapping, DSR automation or AI privacy, we send it to this list. Nothing else.