AI Data Privacy: Where Your Customer Data Goes When Your App Uses AI

Cloud or on the device

The first real privacy decision here is where the model runs. Cloud models are the most capable, yet every request leaves the user’s device and your control to reach them. On device inference, using frameworks like Apple’s Core ML, keeps sensitive data local and never puts it on the wire. Choosing between the two is not only a performance question. It is a privacy question about how far your customers’ data should have to travel.

What the provider does with your data

Sending data to a model is not the end of its journey. Many providers log the requests they receive, some retain them for a period, and their terms differ on whether your inputs can be used to train future models. Reading those terms, and negotiating them where the stakes are high, is a core part of protecting that data, and teams routinely skip it.

The cost of skipping it shows up in the breach data. IBM’s 2025 report found that shadow AI, the unsanctioned use of AI tools, was a factor in 20 percent of breaches and added roughly $670,000 to the average cost. More striking, 27 percent of organizations said that over 30 percent of the data flowing into their AI systems contained private information, while only 17 percent had any technical control capable of stopping a confidential upload. Breaches that involved shadow AI also exposed customer records at a higher rate than the global average.

Where the law lands

Regulation treats this data seriously even when teams do not. Under the European Union’s GDPR, a single mishandled breach can cost up to 4 percent of worldwide revenue, and health data carries its own obligations under rules like HIPAA. Those penalties attach to exactly the customer information that an AI feature quietly ships to a third party, which makes AI data privacy a compliance problem as much as an engineering one.

Staying in control

Control is a design choice, and it starts before the first request is sent. Keep inference on the device when the feature allows it. Send the model only the data it truly needs rather than the whole record. Ask providers for zero retention terms so your inputs are never stored or reused. Encrypt data both in transit and at rest, route calls through private endpoints, and tell users plainly what leaves their device. Handled this way, AI data privacy stops being a liability hiding inside a convenient feature and becomes something you can stand behind.