How to Read an AI Vendor Pitch Without Getting Sold

Every AI vendor pitch looks like it will change your business. Some of them will. Most won't. The difference isn't obvious in the demo, and it isn't in the case studies. It's in the six questions vendors dodge. This post is a short field guide for executives sitting through more AI pitches than they'd like.

The pitch pattern that repeats

Read enough AI vendor decks and you'll notice a template. Big-name customer logos. A demo that starts with a happy path. A case study with a percentage improvement in a bold font. A pricing slide with three tiers, "Enterprise" priced by request.

None of this is a lie. It's the compressed, best-case version of a product. The problem is that AI products vary more between "demo case" and "your case" than most other software. So the question isn't whether the demo works — it's how the vendor behaves when the demo doesn't fit.

Six questions that shorten the sales cycle

1. What happens on our first 10 real inputs — not a curated demo?

The best question you can ask. Vendors that are confident will offer to run their product against your real data (anonymised if needed) in the second meeting. Vendors that aren't will find reasons to delay, propose a POC that costs money, or send you a sample dataset instead.

You want honest numbers on your actual inputs. If a vendor can't produce them within two weeks, note it. That's information.

2. What's the failure mode when it's wrong?

Every AI product is wrong sometimes. The interesting question is what "wrong" looks like. Is it a small hallucination that's easy to catch? Is it a silent misclassification that ends up in a customer email? Is it a decision that's expensive to reverse?

A vendor that can describe the failure modes clearly is a mature product team. A vendor that says "our accuracy is 98%" without describing what the 2% looks like is selling you a number without a context.

3. How is model cost priced — and what changes if our usage doubles?

AI vendors price three ways: per user, per action, or per token. The billing model matters more than the headline price. A per-user product with unlimited actions looks expensive at 10 seats and cheap at 100. A per-token product looks cheap in the demo and expensive in production.

Ask for a cost projection at 3x your expected usage. If the number doesn't hold, you've found the ceiling of the deal.

4. Where does our data live, and who trains on it?

Three specific sub-questions:

• Where is our data physically stored?
• Is our data used to train their models, or shared with the underlying model providers?
• What happens to our data when we cancel?

Reasonable answers exist for all three. Vagueness on any of them is a flag. Enterprise buyers already know this; smaller companies sometimes discover it after the contract is signed.

5. Who supports us when it breaks?

Not "do you have support." Specifically: what's the SLA on the response, what's the SLA on a fix, and who's on the phone at 2 a.m. when the model produces something embarrassing?

Vendors with mature support tell you the answer without hedging. Newer products often have "we'll be there for you" language that translates to "one engineer, business hours, best effort."

6. Can we take our data and outputs with us when we leave?

The lock-in question. Every AI vendor should be able to export your inputs, outputs, and any structured configuration in a machine-readable format. If the answer is "we don't currently support export," that's a two-year problem waiting to happen.

Signals that a pitch is worth pursuing

• The demo starts with a hard case, not a happy path
• Failure modes are named voluntarily, not extracted under questioning
• Pricing is transparent enough to model at 3x your usage
• References include a customer who churned and why
• The product team is on the sales call at some point, not just an AE
• The vendor asks you about your workflow before showing their product

Signals that suggest a stall or a bad fit

• Every reference customer is at least 10x your size
• The demo data is the same across every prospect meeting
• ROI claims are given without a baseline number
• The AE describes technical questions as "roadmap conversations"
• The pricing sheet has one column labeled "Enterprise — Custom"
• Support tiers exist but SLAs are not documented

None of these is a dealbreaker alone. Three or more, and you're looking at a product that will be more expensive to run than the demo suggests.

How to structure the evaluation itself

The evaluation process for an AI vendor is different from other software. Some suggestions from what we've seen work:

Week 1: Real data test

Provide a small anonymised dataset that includes your hardest cases. Ask the vendor to run their product against it and share the outputs. Grade the outputs by hand.

Week 2: Workflow fit review

Have the team that will actually use the product sit through a demo and try to break it. Not the buyer — the operator. What they say afterward is the closest thing to a real predictor of adoption.

Week 3: Reference and support probe

Talk to at least two references. Ask both: "When did the product not work, and how did the vendor respond?" That's the useful question, not "are you happy with it."

Week 4: Contract review with an eye for costs and exit

Focus on billing at 3x usage, data ownership, and exit provisions. This is where lawyers earn their fees.

Common misjudgments

Trusting the accuracy percentage

Accuracy without a task definition is a number without a unit. 92% accurate at what? On what dataset? Measured how? Any percentage that isn't accompanied by those three anchors should be treated as marketing copy, not data.

Buying the roadmap

Vendors will happily promise the feature you need for next quarter. Sometimes it lands. Often it doesn't. Assume the product will look next year exactly like it looks today — if that's still worth buying, the deal is real.

Ignoring integration cost

The vendor's price is often the smallest number in the total cost. Integration, training, adoption, and internal ops time are usually 2-3x the license. Model the total.

Underweighting change management

The most technically impressive AI product loses to the mediocre one that fits into a workflow the team already uses. Where the product lives (Slack, CRM, inbox, dashboard) often matters more than what it does.

When to buy versus build

A short heuristic:

• Buy when the workflow is standard and the differentiation between vendors is small
• Build when the workflow is core to your business and the data is proprietary
• Hybrid when you can use a vendor for 80% of the workflow and build the 20% that's unique

The mistake we see most often is companies building what they should have bought. AI is exciting, so the impulse to build is high. Buying is often the right answer, especially for the first two or three initiatives.

Frequently asked questions

How long should an AI vendor evaluation take?
Four to six weeks for a mid-sized purchase. Any faster and you haven't tested the product against real data. Any slower and the internal momentum decays.

Should we do a paid POC?
Only if the alternative is nothing. A paid POC that answers a specific question is fine. A paid POC that answers "does the product work" is usually the vendor charging you to prove their own claims.

What's a realistic accuracy expectation for AI vendors?
Depends entirely on the task. For structured extraction from clean documents, 90%+ is realistic. For open-ended reasoning, 70% is often the honest number. Any vendor claiming 95%+ across broad tasks is defining "accuracy" loosely.

How do we evaluate an AI vendor if we're not technical?
Bring in a technical advisor for the second meeting. Someone with AI implementation experience will spot the three or four questions that would have taken a non-technical buyer weeks to arrive at. Even a two-hour engagement pays for itself.

Where this fits

Vendor evaluation is a downstream question from an AI roadmap. If you haven't scored the underlying opportunity yet, our post on prioritising AI opportunities is the earlier step. If you're inside a broader diagnostic, the AI Audit includes a competitive gap analysis that names the vendors worth evaluating for each opportunity.

Working with us

We sit on plenty of vendor evaluations as the technical advisor for our audit clients. If you're staring at three shortlisted AI vendors and want an independent view, that's usually a two-hour engagement that saves a two-year commitment.