"We'll self-host it" is a common answer to a residency question from a customer's procurement team. It is a good answer only if you know which layer the question was about. Self-hosting your tracing backend keeps traces in your region; it does nothing for the prompt you still send to a US-hosted model API.
This article breaks the LLM stack into layers, says what self-hosting each one changes for GDPR transfers and residency commitments, and what it costs to run. It is written from the privacy and compliance side: the question is where personal data goes and who can reach it, not which tool has the best dashboard. For the obligations that attach to each store, start with the AI privacy guide.
Residency, transfers and access are three different questions
Teams tend to blur these together, and customers' questionnaires do too.
- Residency is where data is stored and processed. It is usually a contractual commitment ("customer data stays in the EU"), sometimes a sector rule, rarely a GDPR requirement as such.
- Transfers are the GDPR concept. Chapter V restricts transfers of personal data to countries outside the EEA unless there is an adequacy decision, appropriate safeguards such as standard contractual clauses, or a derogation. A transfer can happen without any storage abroad: remote access by a support engineer in another country, or sending a prompt to an inference endpoint outside the EEA, both count.
- Access is who can technically or legally reach the data: the vendor's staff, its sub-processors, and the authorities of the countries where the vendor and its parent company are established. Data stored in an EU region of a US provider is resident in the EU, but the provider's US parent remains subject to US law.
Self-hosting addresses all three for the layer you host, because you control location, staff access and the legal entity. It addresses none of them for layers you still buy as a service.
The layers you can self-host
Model inference
Open-weight models (from Meta, Mistral, Alibaba's Qwen team, Google's Gemma family and others) can be run on your own GPUs or on rented GPUs in a region you choose, using serving engines such as vLLM (Apache-2.0). This is the only layer that removes the model provider from the data flow entirely: prompts, retrieved context and completions never leave infrastructure you control.
What it does not do: make the model as capable as the best hosted models for every task, remove licensing questions (open-weight licenses differ, and some restrict use), or remove the cloud provider if you rent GPUs. A GPU instance in a US hyperscaler's Frankfurt region keeps data in Germany, with the hyperscaler as your processor.
A middle path is a hosted model on a cloud platform's regional endpoint: Azure's EU Data Zone, Vertex AI's EU multi-region, a Bedrock EU geographic profile, or OpenAI's Europe region. These keep inference in the region under the cloud's DPA without running GPUs yourself. Details and caveats are in the vendor article.
Gateway or proxy
An LLM gateway sits between your application and model providers: routing, key management, rate limits, budgets, fallbacks, sometimes caching. Open-source options such as LiteLLM (MIT outside its enterprise directory) can be self-hosted.
For privacy, a self-hosted gateway matters for two reasons. Every prompt passes through it, so a hosted gateway is a processor with access to all of your traffic; self-hosting removes that processor. And it is the natural place to enforce policy: block requests to non-approved regions or providers, apply redaction before prompts leave your network, and strip fields from logs. It does not change what happens after the gateway forwards the request to a hosted model.
Tracing and observability
Traces hold the most complete copy of your LLM traffic: every prompt, retrieved chunk, tool payload and completion. Several LLM observability tools can be self-hosted (licenses below are as stated in each project's LICENSE file in October 2026; check again before you adopt one):
- Langfuse: tracing, prompt management, evaluations, datasets and a playground. Core code is MIT-licensed, with some features under a separate enterprise license. Self-hosting means running the web and worker containers plus PostgreSQL, ClickHouse, Redis or Valkey, and S3-compatible blob storage.
- Arize Phoenix: tracing, evaluation, prompt management, datasets and experiments, built on OpenTelemetry and OpenInference. Licensed under the Elastic License 2.0, which permits self-hosting but restricts offering it as a managed service.
- OpenLLMetry (Apache-2.0): instrumentation rather than a backend. It emits standard OpenTelemetry data that you can send to whatever backend you run.
The OpenTelemetry GenAI semantic conventions, still in Development status, define common attribute names for model calls and mark message content as opt-in because it is likely to contain personal data. Instrumenting against the conventions keeps you free to move between backends, including from a hosted one to a self-hosted one, without re-instrumenting.
Self-hosting the tracing backend is often the highest-value move for privacy relative to its cost. It removes a processor that would otherwise hold a full copy of your traffic, keeps that copy in your region, and lets you set retention and access exactly. It does not reduce what is in the traces; redaction before export still matters (see the prompts and traces article).
Prompt management
Prompt templates themselves rarely contain personal data, unless someone pastes a real customer example into one. Where prompt management lives matters less for residency than for change control. If your prompt tool also stores the test runs and playground sessions that use real data, treat it like the tracing backend.
Vector database and document store
The RAG index holds chunked source documents and their embeddings, often the most sensitive corpus in the stack (support tickets, contracts, HR documents). Self-hostable options include Postgres with pgvector and engines such as Qdrant (Apache-2.0). Self-hosting keeps the corpus in your region and under your access controls, and makes deletion a query you control.
Watch the embedding step. If you generate embeddings with a hosted embedding API, every chunk of every document is sent to that provider at indexing time and every query at search time. A self-hosted vector database with a hosted embedding model still sends the corpus abroad if the embedding endpoint is abroad. Use a regional endpoint or a self-hosted embedding model.
Evaluation
Eval datasets drawn from production are long-lived copies of real conversations, and LLM-as-judge evaluations send them to a model again. Running evals in a self-hosted tool keeps the datasets in your environment; the judge model call is another inference call and follows the same rules as the model layer.
What self-hosting does not fix
- Hosted model calls. As above. The provider's region and retention terms govern that copy.
- Hosted embeddings and judges. Easy to forget because they are "just" utility calls.
- Your cloud provider. Self-hosting on a hyperscaler moves the processor relationship to the hyperscaler; it does not remove it.
- Support and engineering access. If engineers outside the EEA can query the self-hosted trace store, that is remote access and a transfer to account for.
- Telemetry from the tools themselves. Some open-source tools send usage telemetry to their maintainers by default. It usually contains no traffic content, but check and disable it.
- Obligations. Self-hosting changes who processes the data, not whether you must honor access, deletion and retention rules. You now own all of it.
The operational cost
Every self-hosted layer is a production system with personal data in it. That means:
- Security baseline. SSO, role-based access, audit logging, encryption at rest and in transit, network isolation, vulnerability patching. A trace store with weak access control is a worse privacy outcome than a well-run hosted service. The security controls guide lists the baseline.
- Upgrades. LLM tooling releases frequently. Falling behind means missing security fixes; keeping up means regular upgrade work, including database migrations.
- Stateful dependencies. An observability backend may need an analytical database, a relational database, a queue and object storage, each with backups, monitoring and capacity planning.
- GPUs, for model hosting. Capacity planning, utilization, model upgrades, and an evaluation process to decide when a new open-weight model is good enough to switch to.
- Retention and deletion tooling. Hosted tools often have retention settings and deletion APIs built in. Self-hosted, you need to verify they work, including on backups.
- Enterprise features. SSO, audit logs or fine-grained access may sit behind a commercial license even in open-source tools. Check before assuming the free tier meets your control requirements.
For many teams, the right answer is mixed: self-host the stores that hold the most personal data for the longest (traces, vector DB, eval sets), use a gateway you control to enforce regions and redaction, and call a hosted model through a regional endpoint under a DPA with reduced retention.
Decision table
| Layer | Personal data held | Self-hosting buys you | Does not buy you | Operational weight | Common alternative |
|---|---|---|---|---|---|
| Model inference | Every prompt and completion, in transit and in provider logs | Provider removed from the flow; full control of retention and location | Top-tier model capability in every case; freedom from license terms | High (GPUs, serving, model evaluation) | Hosted model on a regional endpoint with zero or reduced retention |
| Gateway / proxy | Every prompt passes through; logs if enabled | One fewer processor; a policy enforcement point for regions and redaction | Any change to where the model call goes | Low to moderate | Hosted gateway under DPA, logging off |
| Tracing / observability | Full copies of prompts, context, tool outputs, completions | Removes the processor with the most complete copy; exact retention and access control | Less personal data in traces (that is redaction's job) | Moderate (several stateful stores) | Hosted tool in an EU region, with masking before export |
| Prompt management | Usually little; test runs may hold real data | Change control in your environment | Much residency benefit on its own | Low | Hosted, with real-data test runs disabled |
| Vector DB / documents | Chunked source corpus and embeddings | Corpus stays in region; deletion under your control | Anything for hosted embedding calls | Moderate | Managed vector DB in an EU region |
| Evaluation | Production-derived datasets, labels | Long-lived datasets kept in your environment | Anything for hosted judge-model calls | Low to moderate | Hosted, with scrubbed datasets only |
Answering the customer questionnaire
When a customer asks "does our data leave the EU?", answer per layer. A truthful answer looks like: "Model inference runs on [provider]'s EU endpoint under a DPA with [retention setting]. Traces and the retrieval index are self-hosted in [region] with [N]-day content retention. Embeddings are generated on [endpoint/region]. Engineering access is limited to staff in [locations]." That is more convincing than "we self-host", and it survives the follow-up questions. List the vendors on your sub-processor page accordingly, and make sure your DPA with the customer matches the setup you actually run.