Somebody forwards you a story. A firm you have heard of has lost data through an AI tool, and the question that follows is always the same one: could that happen here, and what are we doing about it.
The honest answer is usually yes, and usually not much. But the useful answer needs detail the coverage never carries. Almost every documented incident comes down to a small number of mechanisms, and each mechanism has a control that would have stopped it. Once you can name the mechanism, "could that happen here" becomes a question you answer in an afternoon rather than a feeling you have in a meeting.
This is a pattern analysis of what has gone wrong elsewhere. Establishing what is going on inside your own business is a separate exercise, set out in how to audit the shadow AI already happening in your business.
One ground rule: every incident named here is a matter of public record with a source and a date, listed at the end. Where a widely repeated story could not be stood up, it is left out and the mechanism is described instead.
Mechanism one: the work goes into an account your business does not administer
On 02 May 2023 Samsung restricted employee use of ChatGPT and other generative AI tools on company devices and networks, after engineers entered internal source code and meeting content into the tool. Bloomberg and Forbes both reported it at the time.
The mechanism is not that the tool was dangerous. It is that the account sat outside every control the business already trusted. No single sign-on, no admin console, no retention setting anyone else could change, no leaver process.
A second episode shows how thin the margin is. Until 2025 a shared ChatGPT conversation could be made discoverable by search engines through a checkbox at the point of sharing. Users ticked it without understanding the consequence, and confidential material became indexable. OpenAI removed the option on 31 Jul 2025, its chief information security officer saying the feature "introduced too many opportunities for folks to accidentally share things they didn't intend to". Nobody was defaulted into it. A consumer product simply offered a one-click affordance that a governed workspace would never put in front of a member of staff.
Why the ban fails. A policy reaches company devices and company networks. It does not reach a personal phone on a home connection, which is where a good deal of this happens. Prohibition without a replacement removes your visibility, not the behaviour.
The control that works. A governed account, with the same current models and the same speed, connected to the systems where the work already lives so people are not copying data out by hand. If the sanctioned option is slower or worse, the personal account is back within a month.
Mechanism two: a credential outlives the person, the project and the supplier
On 07 Oct 2022 Toyota disclosed that an access key to a customer data server had been sitting in a public GitHub repository since December 2017. A development subcontractor had uploaded part of the source code for the T-Connect service to its own account. The exposure ran until 15 Sep 2022 and covered 296,019 customer records. Key handling and environment separation are covered properly in is the OpenAI API safe for business data.
Nearly five years. One key, one repository, one supplier's developer.
This is not a historical curiosity. GitHub reported on 01 Apr 2025 that more than 39 million secrets were leaked across the platform during 2024, and made the point that long-lived credentials persist unnoticed and keep granting access. AI raises the rate rather than inventing the problem: debugging prompts carry connection strings and keys far more often than anyone volunteers, and every agent you build needs a credential of its own to do anything useful.
Why the ban fails. "Stop putting keys in code" has been the policy everywhere for a decade and 39 million secrets still leaked in a year. Telling people to be careful is not a control.
The control that works. Change what a leaked credential is worth. Per-integration service credentials rather than a shared key. Scoped to the specific objects and actions the job needs. Short-lived where the platform supports it. Revocable individually, so cutting off one integration does not take down four others. And a rotation you have actually rehearsed, because the first time you find out that nobody knows which systems use a key is the wrong time.
Mechanism three: output is trusted in a process where a wrong answer propagates
A Big Four member firm delivered a report to the Australian government's Department of Employment and Workplace Relations that contained fabricated academic references and a fabricated quote attributed to a Federal Court judgment. A revised version published on 26 Sep 2025 disclosed that Azure OpenAI had been used in preparing it, and the firm agreed to refund part of the fee.
The customer-facing version of the same mechanism is on the record too. In Moffatt v. Air Canada, decided by the British Columbia Civil Resolution Tribunal on 14 Feb 2024, an airline chatbot told a customer he could apply for a bereavement fare retroactively. The airline's actual policy did not allow it, and the tribunal held the airline responsible for what its own chatbot had said.
We offer no view on the law in either case. The operational reading is the same in both: a model answered a question it had no authoritative source for, and nothing in the process caught it before the answer reached someone who acted on it.
Why the ban fails. Nobody bans this one. They add a review step, which decays quickly. Reviewing fluent, plausible, correctly formatted output is genuinely hard, and a reviewer who has caught nothing in three weeks stops reading properly.
The control that works. Ground the answer, then check the part that can be checked. Where a question has an authoritative source, the system should retrieve from it and cite what it used rather than recall it. Where a citation, a figure or a policy statement appears in the output, verify it programmatically against that source rather than asking a person to notice. Put the human check where the answer becomes expensive, which is usually where it leaves the building, not at every intermediate step.
Mechanism four: your data leaves through a supplier's AI tooling, not yours
This is the one most businesses have no answer to, and it has the clearest recent example.
From 08 Aug 2025 to at least 18 Aug, an actor Google Threat Intelligence tracks as UNC6395 used OAuth tokens belonging to the Salesloft Drift AI chat agent's Salesforce integration to export data from corporate Salesforce instances. Google published its advisory on 26 Aug 2025 and updated it on 28 Aug to confirm the compromise reached other integrations, including access to Google Workspace accounts.
Cloudflare disclosed its own exposure on 02 Sep 2025: reconnaissance from 09 Aug, access on 12 Aug, support case text, contact details and account information taken between 12 and 17 Aug. It disabled the integration and rotated credentials across every third-party Salesforce integration.
Note what did not happen. Nobody broke into Cloudflare. Nobody broke into Salesforce. A supplier's AI product held a token into its customers' CRM, and that token was the asset.
Why the ban fails. Banning third-party integrations is not available to you, because the useful ones are how the business runs. The usual substitute is a supplier questionnaire, which records an intention on a date. The token keeps working regardless of what the questionnaire said.
The control that works. Inventory integrations by token and scope, not by vendor name. For every connected application, know what it can read, what it can write, who approved it and when the credential was last rotated. Treat a supplier's AI features as a sub-processor question in the contract, with a notification obligation when they change. And keep the blast radius small: an integration that needs to read contacts should not also be able to read every closed support case. The same discipline applied to your own builds is covered in governing AI write access to your CRM.
Mechanism five: logs and records hold far more than anyone intended
Two findings from the same period make this concrete.
Cloudflare found 104 of its own API tokens inside the support case text taken in the Drift incident, and rotated all of them. Customers had pasted credentials into support tickets, which nobody thinks of as a credential store until one is exfiltrated.
On 29 Jan 2025 Wiz Research reported a publicly accessible ClickHouse database belonging to DeepSeek, requiring no authentication, holding over a million log entries that included chat history and secret keys. DeepSeek secured it after disclosure.
Retention is the second half of this. On 13 May 2025 a US magistrate judge directed OpenAI to preserve and segregate output log data that would otherwise have been deleted. That going-forward obligation was lifted on 09 Oct 2025, with previously preserved data retained. The point is not the case. It is that a deletion schedule is a policy operating inside a legal system, and policies can be suspended by events you do not control.
Why the ban fails. Turning logging off leaves you unable to debug anything or evidence anything, which is a worse position than the one you started in.
The control that works. Decide what a prompt log is for before you build, then log the minimum that serves it. Redact at the point of capture rather than the point of review. Set a retention period deliberately instead of inheriting a default of forever, and apply the same rules to support tickets and their attachments. Where material genuinely cannot be retained by anyone, keep it out of the vendor's hands rather than relying on a promise to delete it.
The five in one table
| Mechanism | What it looks like in your business | The control | Usual owner |
|---|---|---|---|
| Ungoverned account | Personal AI subscriptions, reclaimed on expenses | A governed account that is genuinely faster | IT with the budget holder |
| Credential outlives its purpose | One long-lived key, broad scope, unknown users | Per-integration scoped credentials, rehearsed rotation | Engineering |
| Unchecked output | AI drafts going to clients, customers or regulators | Grounded retrieval plus a programmatic check at the exit point | The process owner, not IT |
| Supplier's AI tooling | Connected apps holding tokens into your CRM | Integration inventory by token and scope, sub-processor clause | Procurement with IT |
| Over-broad logs and retention | Full prompts, customer records, secrets in tickets | Redact at capture, deliberate retention period | Engineering with the DPO |
None of the five require a new product, and four are configuration and process rather than software.
Where this leaves you
Notice what is missing from that table: any suggestion that the answer is to use AI less. In every documented case above, the tool did roughly what it was asked to do. The failure was in how it was connected, credentialed, reviewed or logged.
That is why this is an engineering and governance question rather than a compliance one. Whether a given exposure creates an obligation for your business is a question for your own advisers and we do not answer it. What we answer is which controls belong in the build, what they cost, and which of your existing integrations should have been scoped more tightly a year ago.
We are an OpenAI Select Partner and our OpenAI implementation work usually starts with the integration inventory rather than a build. We run the same work model-neutral where the right answer is a different vendor, or the AI already sitting inside software you pay for. We do not resell licences and we do not mark up your usage, so we have no reason to sell you a build where a subscription and a tighter configuration would do.
The decision in front of you is narrow. Take the five mechanisms, and for each one write down the name of the person who could answer "how would we know" within a day. Wherever that box is empty, that is your first piece of work, and it is almost certainly the integration inventory.
A step back, and the other routes available, are in the wider picture of OpenAI for UK businesses.
If you would rather have it done properly and written up in a form you can hand to a client or a board, request a quote.
Stay Updated with Our Latest Insights
Get expert HubSpot tips and integration strategies delivered to your inbox.
