Sign Nothing Until You Ask These 12 Questions: A Founder’s AI Chatbot Buyer Guide

Here is the uncomfortable truth: demos lie by design. They run on curated scripts, perfect test data, and a salesperson who knows exactly which three questions to type. The pricing pages lie by structure — you are not buying software, you are buying a unit you don’t understand yet: credits, resolutions, conversations, seats. And the comparison articles lie by incentive — many of them are pay-to-rank. So forget the demos and the listicles. Ask the twelve questions below, in this order, before you sign anything.

1. Why chatbot buying is broken: the demo trap and the pricing fog

The demo trap works because it exploits the wrong sample. A salesperson demos the platform on data the vendor chose — clean FAQs, happy-path questions, a knowledge base that was obviously built for the occasion. What you never see in a demo: your actual mess. Your 400 FAQ pages that contradict each other. Your customers who write in sentence fragments, in three languages, with typos. Your edge cases, which are not edges at all — in ecommerce, the weird questions are the business. Any buyer guide that doesn’t tell you to distrust the demo is selling you something.

The pricing fog is the second trap. Vendors sell you a metering unit and let you misjudge it. “Credits” that expire monthly. “Resolutions” defined by the vendor’s own (generous) criteria. “Conversations” that count a bot greeting as a conversation. “Seats” for agents who just need to look at a transcript once. By the time your store scales, the unit economics of the tool have quietly changed underneath you — and switching costs are high, because your bot is now trained on months of your data. This is why the diligence has to happen before the signature, not after the first renewal shock.

The third trap is the easiest to fall into: the feature checklist. Vendor A has 47 integrations, Vendor B has 52. So what? A checklist can’t tell you whether the Shopify integration syncs order status in real time or just lets you paste an order number into a widget. It can’t tell you whether the “CRM integration” is a webhook you configure yourself or a button somebody clicked. Checklists measure marketing effort. The twelve questions below measure operational reality.

2. Data and training: who owns your conversations, and how does the bot actually learn?

Questions 1–4 of the scorecard live here, because this is where most chatbot purchases quietly die. Your conversations with customers are a company asset — they contain pricing questions, complaints, feature requests, and objections. Some vendors claim broad rights to train their models on your data. Ask directly: who owns the conversation transcripts, are they used to train the vendor’s shared models, and can you opt out completely? If the answer involves a “trust us” slide, walk away. Data ownership should be a line in the contract, not a paragraph in a privacy center.

Next, ask how the bot actually learns. There are two fundamentally different products sold under the name “AI chatbot.” The first is trained on your content — your help docs, product pages, policies — and answers from that grounding. The second is a generic model wrapped in your brand colors, improvising answers with varying confidence. The second type is the one that hallucinates your return policy. Ask: does the bot answer only from my knowledge base, or does it generate from general knowledge? Can I see exactly which source a given answer came from? Source-grounded answers are auditable; improvised answers are lawsuits waiting for a coupon code.

Hallucination control is the third question, and the one vendors are least prepared for. Every model makes things up; the question is what the platform does about it. Ask for specifics: confidence thresholds, fallback behavior when no source matches, the exact wording the bot uses when it doesn’t know. “I don’t know, let me connect you with the team” is an acceptable answer. A confidently wrong shipping time is not. The discipline of training a chatbot to match your brand voice is inseparable from this: tone control without factual grounding is just a well-dressed liar.

The fourth question is about the learning loop: when the bot gets something wrong, what happens? In the best platforms, every correction you make in the transcript feed improves the next answer — a human reviews, edits, and the system learns. In the worst, you file a ticket and the vendor’s team retrains something, someday. Ask to see the correction workflow live, on your own data. If improving the bot requires vendor intervention, you haven’t bought software — you’ve bought a consulting retainer with a login screen.

3. Integration depth: APIs and live data, not screenshots of logos

Questions 5–7 concern what the bot can actually do versus what it can talk about. A chatbot that can only answer FAQs is a searchable help center with extra steps. The value compounds when the bot can act: check an order status, look up stock, start a return, update a subscription, book a slot. That requires real integration — authenticated API calls into your stack, not a widget that links to your tracking page. Ask the vendor to show, on a live store, the bot resolving a “where is my order” query end to end. If the demo suddenly needs “a quick custom setup for that,” you’ve learned everything.

For ecommerce specifically, the Shopify question matters more than any other integration. Your bot lives on your storefront; it must know the storefront. Ask: does it sync product catalog, inventory, and order data in real time, or does it read a static export? Can it surface a product recommendation based on live availability? The Shopify chatbot comparison landscape is full of apps that claim “deep integration” and deliver a chat bubble. Real depth means the bot acts as a storefront employee with register access, not a greeter at the door.

Then there’s the CRM and helpdesk layer. When a conversation escalates, does the full transcript land in your helpdesk ticket with context attached, or does your agent start from “hello, how can I help?” again — the single most churn-inducing sentence in customer service? Ask how the handoff payload works: tags, sentiment, customer identity, cart contents. A proper chatbot API integration passes structured context so the human picks up mid-conversation, not at the beginning. If the “integration” is a daily CSV email, you have your answer.

4. The total-cost autopsy: what credits, resolutions, and seats really mean

Questions 8–10 are the pricing autopsy, and they require a spreadsheet, not a brochure. First: define the metering unit in plain language, then model it against your growth. A “resolution” sounds generous until you learn the vendor defines it, and that a bot saying “here’s our shipping policy” counts as resolved even when the customer immediately opens a ticket. A “conversation” sounds simple until greetings and abandoned widgets count toward it. Ask for the exact counting rules, in writing, and then project: what does this cost at 2x your current traffic? At 5x? The vendors that win this question answer it eagerly. The ones that dodge it are telling you the margin is in your confusion.

Second: map the seat trap. Many platforms charge per agent seat, which quietly taxes you for transparency — every person who needs to peek at a transcript, review a handoff, or correct a bad answer becomes a billable unit. Ask whether read-only, reviewer, and analyst roles are free, and what the minimum seat commitment is. A platform that charges you to look at your own data is building its business model on your curiosity.

Third: price the exit. Ask what happens to your training data, conversation history, and configured flows if you leave. Can you export transcripts in a usable format? Does the knowledge base you’ve built port out, or does it evaporate? This question feels premature in week one and becomes the most important question in month eleven. A vendor confident in its product makes leaving easy; a vendor confident in its lock-in makes exporting a “custom request.” Treat any friction here as a preview of the renewal negotiation.

5. Human handoff and escalation: the metric that predicts churn

Question 11 is about the moment of failure, because every bot fails and the failure is where customers are won or lost. Gartner’s 2026 outlook suggests agentic AI will resolve the vast majority of common service issues without human help by 2029 — but the flip side of that statistic is that the remaining issues are the hard, emotional, high-stakes ones. A lost package two days before a birthday. A double charge. A product that arrived broken. These are the conversations where a bad handoff costs you the customer, and where Klaviyo’s finding — that 74% of consumers have come to expect personalized brand experiences — cuts hardest. A generic “an agent will be with you shortly” is the opposite of personal.

So ask: what exactly triggers an escalation, and who designs those triggers? The best platforms let you define escalation rules — sentiment drops, repeated questions, specific intents like “refund” or “cancel” — and route with full context to the right queue. The worst have a single “talk to human” button and a shared inbox. Ask to see the escalation analytics: how often does the bot hand off, what’s the average wait after handoff, and what share of handed-off conversations resolve without the customer repeating themselves? That last metric — the no-repeat rate — is the single best predictor of whether the bot is helping or just adding a step.

And here’s the stat that should sharpen your skepticism about full automation: Gartner found that about half of the companies that cut customer service staff in favor of AI expect to rehire by 2027. The lesson isn’t that AI failed — it’s that companies automated the handoff away along with the humans. Your evaluation question is not “how much can the bot handle alone” but “how gracefully does it fail.” The bot that escalates brilliantly is worth more than the bot that deflects 5% more tickets and strands the rest. This is also where WISMO ticket deflection done right proves its model: routine status questions disappear into automation while anything with emotional weight reaches a human fast, with context.

6. The 12-question scorecard, red flags, and a 30-day pilot that actually proves value

Here is the scorecard, distilled. Ask every vendor all twelve; score 0–2 per question; sign nothing below 18.

Data & training (Q1–Q4): (1) Who owns conversation transcripts, and can we fully opt out of model training? (2) Does the bot answer only from our knowledge base — can we see the source of each answer? (3) What are the exact hallucination controls: confidence thresholds, fallback wording, unknown-answer behavior? (4) Show us the correction workflow live: when the bot is wrong, how do we fix it and how fast does it learn?

Integration depth (Q5–Q7): (5) Demo a “where is my order” resolution end to end on a live store — no custom setup. (6) For Shopify: real-time sync of catalog, inventory, and orders, or static exports? (7) What exactly lands in the helpdesk ticket on escalation — full transcript, tags, cart, identity?

Total cost (Q8–Q10): (8) Define the metering unit precisely, in writing — then model our cost at 2x and 5x traffic. (9) Are read-only and reviewer roles free? What’s the minimum seat commitment? (10) If we leave, what exports in what format — transcripts, knowledge base, flows?

Handoff (Q11–Q12): (11) Show us escalation triggers, routing rules, and the no-repeat rate after handoff. (12) What does the implementation timeline look like on our data — and what does week-one performance typically look like versus month three?

Red flags that end the conversation immediately: the vendor won’t put data ownership in the contract; “AI-powered” with no explainable source grounding; pricing that can’t be modeled on a napkin; a demo that avoids your real questions; case studies with no named customer; and the classic — a discount that expires this Friday. Genuine enterprise software doesn’t need a countdown timer.

Then run the pilot that proves value instead of promising it. Thirty days, on your live traffic, with your real catalog and your real edge cases — not a sandbox, not a staging store. Week one: instrument the baseline (ticket volume, first-response time, resolution rate) before the bot touches anything. Weeks two and three: the bot handles the routine tier while you review every transcript daily and correct it — this is also where A/B testing chat scripts earns its keep, because your first scripts will be wrong. Week four: measure against the baseline on three numbers only — deflection of routine tickets, no-repeat rate after handoff, and customer satisfaction on bot-handled conversations. If those three move, sign. If the vendor won’t agree to a live-traffic pilot with a clean exit, that is your answer.

One practical note: the easiest pilots are the ones you can start without procurement. Platforms with a genuinely free entry tier let you run this entire evaluation on your real store before anyone in finance learns the vendor’s name. Oscar Chat is one such option — a free plan means the 30-day test costs you nothing but the hour it takes to set up. That’s the right price for an evaluation: the vendor proves value before you prove budget.

The Takeaway

Chatbot purchases fail on evaluation, not technology. The demo is a curated performance, the pricing unit is a fog machine, and the feature checklist measures marketing budgets. The twelve questions cut through all three: who owns your data, how the bot learns and admits ignorance, whether integrations are real or decorative, what the meter actually meters, how gracefully it fails, and what leaving costs. Run them in order, score honestly, and demand a 30-day live-traffic pilot with a clean exit — because the only chatbot demo worth trusting is thirty days of your own customers talking to it. Sign nothing until the numbers from your store, not the slides from their deck, tell you to.

FAQ