Every enterprise AI deal looks the same in the room where it's sold: a crisp demo, an impressive benchmark, a confident roadmap. Then you sign — and the gap between the demo and your Tuesday-morning reality starts to show.
I've been on both sides of that room. As Principal Product Manager on ServiceNow's AI Platform, I shipped Generative AI into production for large enterprises. Earlier, I took Luma, an AI chatbot platform, to Leader in The Forrester Wave at ServiceAide. I know which questions a vendor hopes you won't ask — because they're the ones we prepared hardest to answer. AIAUTOMIC sells no software and takes no referral fees, so I can say plainly what those questions are.
Here are five. Notice that none of them are about the model.
1. "What does this cost at our production volume — not the demo's?"
The demo runs on cherry-picked inputs and a handful of users. Your production runs on messy data, long contexts, retries, and peak-hour load. Those are different economies.
Ask for a written cost estimate at your real monthly volume — including retries, worst-case context length, and concurrency. This matters most for anything "agentic": a single user action can quietly fan out into dozens of model calls, and the bill scales with it. A vendor who can't model your cost curve before you sign will be just as surprised as you are when the invoice arrives.
2. "Show me a customer in production for a year — at our scale."
MIT's NANDA initiative found that 95% of enterprise GenAI pilots deliver no measurable P&L impact. A polished pilot proves almost nothing; the hard part is what happens after month three, when novelty fades and the workflow has to actually hold.
So ask for a reference that has been live for twelve months or more, at a scale comparable to yours — and ask to speak with them directly. The most useful question you can put to that reference is simple: what broke, and how long did it take the vendor to fix it?
3. "When the model is wrong, what happens — and who's accountable?"
In production, a wrong answer isn't an edge case — it's a Tuesday. What separates a serious platform from a demo is what it does on that Tuesday.
Ask concretely: are there confidence thresholds that trigger a human hand-off? Is there an audit trail — a receipt showing what the system did and why? Can you roll back? Where does liability sit when the agent acts on bad output? If a vendor treats hallucination as a communications problem rather than an engineering one, that tells you everything about how the next incident will go.
4. "Where does our data live, and does it train your models?"
This is the question that belongs in the contract, not the sales call. Get specific and get it in writing: data residency, retention windows, whether your prompts and outputs are used for training, who owns the IP in the generated results, and which subprocessors touch your data.
"We take security seriously" is not an answer. A clause is.
5. "What do we keep if we leave?"
Every AI vendor is happy to talk about onboarding. Ask instead about the exit. Can you export your data, your prompts, your fine-tunes, and your evaluation sets in a usable form? How portable is the solution if you switch models or providers next year?
The best vendors make leaving easy — precisely because they aren't worried you will. Lock-in is what a vendor reaches for when the product can't hold you on its own merits.
The through-line
A great demo is optimized for the day you sign. A great partner is optimized for your worst day in production. Every one of these five questions is designed to move the conversation from the first to the second.
You don't need to be an AI expert to ask them — you need to be willing to sit in the silence after each one. The quality of the answer, and how quickly it arrives, will tell you most of what you need to know before anyone reaches for a pen.
Evaluating a vendor and want a second set of eyes from someone who has sat on the other side of the table? That's part of what AIAUTOMIC does — vendor-neutral, with no software to sell you.