Taking a ChatGPT prototype to production: what actually breaks

The demo went well. Somebody in the business built a thing that reads the incoming orders, or drafts the quotes, or summarises the calls, and it worked. People were impressed, correctly, because it is genuinely impressive. And then someone senior said the sentence that starts the expensive part: can we roll this out to the whole team, or put it in front of customers.

What you are discovering, or are about to, is that the distance between a thing that works once and a thing that works every day is much larger than anyone estimated in the room where the demo happened. Not because the demo was fake, but because a demo and a system are different objects, and almost none of the work that separates them is AI work.

Here is what that work consists of, in the order it tends to bite.

A demo is one run, on an example the builder chose

The prototype was run by the person who built it, on an input they picked, with the option to try again if it came out odd. Every one of those conditions disappears in production.

Production means somebody pastes in a forty-page PDF, or sends an order with a line item that does not exist in your catalogue, or writes in a way the builder never anticipated. And the same input has to produce the same answer on Tuesday that it produced on Monday. If it does not, nobody will trust the output, they will start checking every one by hand, and that removes the entire point.

Two things close most of this gap, and neither is glamorous. The first is enforcing the shape of the answer: OpenAI's Structured Outputs lets you require the model's response to match a JSON schema exactly, rather than merely be valid JSON. That converts a large class of "the model said something strange" into "the call failed", which your software can handle.

The second is a test set, and it is the thing prototypes almost never have. Thirty to fifty real examples, including the ugly ones, with the right answer written down next to each, run before and after every prompt change and every model change. Without it, every subsequent change is a matter of faith and you cannot answer a simple question from your own board: is it better than it was last month.

One warning if you are commissioning this now. Do not let anybody build that harness on OpenAI's own Evals product. OpenAI's deprecations page states that existing evals become read-only on 31 Oct 2026 and that the Evals dashboard and API shut down on 30 Nov 2026, with Promptfoo named as an alternative. Your test set should live in your own repository regardless.

Most promising prototypes die on the data, not on the AI

This is the one to plan for, because it is the most common reason a prototype that impressed everybody never becomes a system.

The prototype ran on a clean sample: a handful of records the builder chose, or a spreadsheet they tidied up first without really registering that they had done it. Production runs on your actual CRM, with duplicate companies, half-empty fields, free text where a picklist should be, three spellings of the same customer, and a long tail of records nobody has touched since 2019.

The model does not fix any of that. Worse, it does not fail on it either. It produces a confident, plausible answer derived from bad input, which is more dangerous than an error, because an error gets noticed.

So before committing a budget, run one test. Take one hundred real records at random, not selected, and run the prototype over them. Count how many produce an answer you would let out of the building unchecked. If that number is sixty, you do not have a model problem. You have a data project sitting in front of your AI project, and it needs to be scoped, costed and staffed as one. That is unwelcome news in the week after a good demo, and much less unwelcome than it will be four months into a build.

Production means the output lands in a system, and fails visibly when it cannot

In the demo, a person copied the output somewhere. That person was quietly doing the error handling: noticing when it looked wrong, fixing the formatting, deciding not to paste the odd one. None of that is in the specification, because nobody wrote it down.

Production has to do those jobs explicitly, which means three things beyond the model call. The output lands in the destination system with its structure enforced. There is an explicit route for "not confident enough" that sends the item to a person rather than writing a guess into your CRM. And repeated work is safe, so the same order arriving twice does not create two records.

In that class of work, the model reading the order is the straightforward half. The engineering is in deciding what a line of a customer's text corresponds to in your product data, and what the system does when it corresponds to nothing.

If you have not built anything yet and are still choosing between the chat interface and the API, that is a different and earlier decision, and we have set it out in what the OpenAI API does that ChatGPT cannot.

At volume, the shape of the workload decides the bill

A prototype's running cost is invisible: a few pounds on somebody's card, for twenty runs. At production volume it becomes a real line, and what determines it is not how clever the system is. It is three design decisions nobody made during the prototype, because nobody was counting. What the whole thing costs, and where the money actually goes, is in what an OpenAI integration costs.

Which model does which task. The GPT-5.6 family exposes three models in the API: Sol with the highest reasoning ceiling, Terra as the balanced default, and Luna as the fast, lowest-cost option. (ChatGPT also offers GPT-5.6 Sol Pro, which OpenAI makes selectable to Pro and Enterprise users for its highest-quality results. It is a ChatGPT model choice rather than one of the three API models, so it does not change what you build against.) Prototypes are almost always built entirely on the most capable one, because during a demo that is free and safe. In production most workloads are a mix, and classification, extraction and routing rarely need the top variant. Splitting the work by task is usually the largest single saving available.

What context repeats. OpenAI's prompt caching works automatically with no code changes, on prefixes of at least 1,024 tokens for GPT-5.6 and later, and a cached prefix stays eligible for reuse for at least thirty minutes. There is a catch worth knowing before you design the prompt: on GPT-5.6 and later, cache writes are billed at 1.25 times the uncached input rate. A system that sends a slightly different prefix every call pays that premium repeatedly and never earns the read discount. Put the stable part of the prompt first, and keep it stable.

What does not need an instant answer. OpenAI's Batch API takes asynchronous groups of requests on a twenty-four hour turnaround, at a stated 50% discount against synchronous calls and with more rate-limit headroom. Nightly enrichment, back-office classification and bulk document processing belong there. Only the things a person is waiting for need the synchronous path.

Get a cost per transaction from a real sample before you commission the build, not a projection afterwards, and set per-project spend limits and alerts on day one. The first surprising bill is usually a loop, not growth.

The parts nobody demos: keys, logs and a named owner

The prototype is running on somebody's personal key, quite possibly on their own account with their own card. That arrangement fails two ways: it does not survive a security review, and it stops working the day that person leaves.

OpenAI's own production guidance is the floor here. Keys belong in environment variables or a secret management service, never in code or a public repository. Run separate projects for staging and production, limit who has access to the production project, and set custom rate and spend limits per project. For a system rather than a person, use a service account, so the key is not tied to an individual at all. Roles at organisation and project level, plus permissions on the key itself, are what stop a reporting job being able to write to your CRM.

Then decide what gets logged, because a silent failure is invisible by definition. At minimum: every call, with its input, its output, the model used, token counts, latency and the outcome (written, rejected, or escalated to a person), retained long enough to investigate a complaint. Alert on two things. Error rate, obviously. And volume falling to zero, which is what a broken integration usually looks like: not errors, silence.

Finally, one named person owns it, the account is in the company's name, and the source code is in your repository rather than a supplier's. If you cannot name that person, you are not ready to put this in front of customers.

The platform will move, and you can make that a scheduled task

Something you depend on will be deprecated. This is not a risk to hedge against, it is a certainty to design for, and OpenAI publishes the dates well in advance. As of 08 Aug 2026 its deprecations page carries several: the Assistants API is removed on 26 Aug 2026, the v1/prompts API and reusable prompt objects shut down on 30 Nov 2026, Agent Builder shuts down on 30 Nov 2026, the Videos API and the Sora 2 models are removed on 24 Sep 2026, and the DALL-E model snapshots were removed on 12 May 2026.

Three design choices turn that from an emergency into a diary entry. Keep prompt text in your own codebase rather than in vendor dashboard objects, which is what OpenAI itself now recommends. Keep the model name in configuration rather than scattered through the code, so swapping variants is a change of one value. And keep the test set from the first section, because that is what makes a forced model change an afternoon's verification instead of an act of faith.

Then put a quarterly reminder in somebody's calendar to read the deprecations page. It takes ten minutes and it is the cheapest insurance in this post. If you suspect you are already exposed on the nearest of those dates, we have set out how to check and what the options are in what to do about the Assistants API retirement.

Some prototypes should not graduate, and that is a real answer

Everything above costs money, and for a good number of prototypes the honest recommendation is not to spend it. Four questions settle it, and you can answer them without an engineer in the room.

If a prototype saves one person twenty minutes a week and a wrong answer is obvious and harmless, it is a personal productivity tool. Leave it as one. Industrialising it will cost more than it returns, and the version that survives a security review will be slower and more annoying to use than the version they have now. We say so rather than take the work.

The awkward cases are high frequency and low consequence. There, the useful question is whether the value is really in the AI at all. If what the prototype does is extract five fields from a predictable document, the production answer is sometimes a smaller, cheaper and duller piece of software with a model used sparingly inside it. If you are still working out whether what you have needs to become a system at all, that comparison is the subject of Custom GPTs versus a properly built agent.

The decision in front of you

Before you commission anything, do two things that cost you a day. Run the prototype over one hundred random real records and count how many answers you would let out of the building. Then get a cost per transaction from that same sample.

Those two numbers turn a vague enthusiasm into a decision with three defensible answers: build it properly, build the smaller and duller version, or leave it as somebody's personal tool and go and do something else.

A step back, and the other routes available, are in the wider picture of OpenAI for UK businesses.

We are an OpenAI Select Partner and this is the work we do, so treat that closing paragraph with the scepticism it deserves and check it against how we behave. We take no margin on your usage, so we have no reason to talk you into a larger system than the problem justifies. If the two numbers say your prototype should stay a prototype, we will tell you so, and it will not cost you anything. If they say otherwise, request a quote and tell us what the prototype does and who wants it rolled out, or read what a production build involves on our OpenAI implementation page.