Handling PII in AI Applications Without Breaking the Product

The first time a real customer asks how you handle their data in your AI features, most teams don't have a good answer. The second time — usually from a legal or procurement contact — the answer has to be right. This post is what we'd want a founder to know before that conversation happens.

Handling PII in AI applications isn't a compliance checkbox. It's a set of design decisions that touch storage, model providers, prompts, logs, and the product itself. Get it wrong and it's a lawsuit or a churn event. Get it right and it's a competitive edge.

What counts as PII once you add AI

Under GDPR, CCPA, HIPAA, and most modern privacy laws, PII is anything that identifies a person directly or in combination with other data. Names and emails are obvious. Less obvious: a support ticket that mentions a specific building, a rare medical condition, or a job title at a small company.

AI features widen this. A model that summarizes a customer's inbox is now processing everything in that inbox. A chatbot that references user history has access to that entire history in one prompt. Suddenly the smallest features touch the largest data sets.

Where PII leaks in an AI stack

Four places, in order of frequency.

1. The model provider's logs

Most managed model providers offer a "no training" option and separate log retention settings. Both need to be turned on. The defaults are not always the safe ones, and they change. This should be a monthly check, not a one-time setting.

If your compliance situation requires zero data retention at the model provider — healthcare, finance, some EU cases — you need a provider tier that contractually guarantees it. This is available; it's not the default.

2. Your own logs

This is where most breaches actually happen. Someone adds structured logging for debugging, PII goes into logs, logs go to a third-party observability tool, and now customer data lives in a system you don't fully control.

The fix is a redaction layer on the way into logs — regex plus a small classifier that catches names, emails, phone numbers, and IDs. Every AI-heavy project we work on ends up building this early, or wishing it had.

3. The prompt itself

Prompts often include user context — recent messages, profile fields, past actions. That context can carry PII from users other than the one sending the request. A shared workspace in your product is a shared context for the model, and cross-user leakage becomes a real failure mode.

Solve this at the prompt-building step, not later. Redact before assembling. Assume the entire prompt is retained somewhere, and design for that.

4. Model outputs

Models occasionally return data from other contexts they've been given. This is rare on managed models with proper isolation, but it happens with fine-tuned or self-hosted models trained on customer data. If you fine-tune on user data, you've introduced a new privacy risk that requires the same care as any other data store.

The patterns that actually work

Here's what we set up in AI applications we build for clients who take this seriously.

Redact before you send

Before any user text hits a model, run it through a redactor. Names to tokens, emails to hashes, phone numbers to placeholders. The model can still reason about "the user's assistant" or "their manager"; it doesn't need "Sarah at Acme Corp".

For most B2B products, a well-tuned redactor catches 90-95% of PII before it leaves your infrastructure. The remaining 5-10% is a business decision, not a technical one.

Separate PII storage from AI-accessible storage

Store real user data in one table. Store the AI-facing projection — sanitized, tokenized, minimum-necessary — in another. The AI features query the second table. The first stays untouched by every AI pathway.

This is the same pattern used in data warehousing, and it's underused in AI apps because AI feels new. It's not. The pattern still works.

Ask for consent, and mean it

If an AI feature meaningfully processes user data, users should know before they use it. Not a buried EULA — a real inline disclosure. This is good product design, not just compliance. Users trust products that tell them what's happening.

Our safety guardrails post covers more of the product side.

Log what a model saw, not just what it returned

For audit purposes, you need to be able to answer: "what data was passed to the model on request X?" That's a log of the sanitized prompt, not the raw one. Store it long enough for compliance, short enough that a breach doesn't cascade.

Model provider choices, and what they mean

Choice of provider is the single biggest privacy decision after redaction.

• Public API tiers — fine for non-sensitive use cases. Never send PII you can't tokenize.
• Enterprise or contracted tiers — offer no-training guarantees, regional data residency, and zero-retention options. Standard for most B2B AI products.
• Self-hosted or open-source models — full control, full responsibility. Worth it only when the compliance situation demands it and the team can operate the infrastructure.
• On-prem models — required in some regulated environments. Expensive, slow to iterate, sometimes the only option.

We've done all four. The right one depends on your buyer, not your engineering preferences.

Consent and disclosure without killing the product

The tension is real: too much friction and users don't opt in, too little and you're breaking a law or a trust. The pattern that works is graduated disclosure.

First interaction: a short inline notice explaining what data the AI feature uses. Not a modal, not a wall of text. One sentence, one link to the details. Users read one-sentence notices. They don't read modals.

Ongoing use: a settings page where users can revoke or scope the AI's access. This has to actually work. If a user opts out, the AI features either turn off or gracefully degrade. No hidden processing.

Mistakes we see repeatedly

Logging everything "for debugging" and forgetting. Debug logs live in production far longer than anyone plans. Redact before writing, always.

Sending user data to a fine-tuning job without a data agreement. Fine-tuning consumes user data. Consent for using the AI feature is not consent for training on that data. These are separate.

Trusting the model provider's defaults. Providers change defaults and pricing tiers. The setting that was safe six months ago may not be today. Audit quarterly.

Skipping DPIAs when you clearly need one. If you're building AI features that process EU user data at any scale, a Data Protection Impact Assessment isn't optional. Do it early, when the answers are cheap to change.

Trade-offs to expect

Some quality is lost when PII is redacted. A model that can see "Sarah at Acme Corp" personalizes better than one that sees "the user at their company". The gap is real, and for most applications, small enough that users don't notice.

Where personalization is the core value — a coaching product, a personal assistant — you accept more data exposure and invest more in security. Where personalization is a nice-to-have, redact aggressively and take the small quality hit.

Frequently asked questions

What good looks like

You should be able to answer four questions in a customer security review without hedging: what data goes to the model, where it's stored, how long it's retained, and how a user removes their data. If any of those needs a "let me check", the answer isn't ready yet.

Getting there isn't a compliance exercise. It's a design exercise, and it's easier if it starts at the beginning of a project rather than being retrofitted after a legal review. Our regulated industries post goes deeper on this for teams selling into healthcare and finance.

If you're building AI features and want a review of how you're handling user data, book a call. We'll walk through the flow and tell you what to change before your first enterprise customer asks.