How to evaluate an AI tool for your business in 15 minutes — a checklist to avoid buyer's remorse

Somewhere between the demo call and the monthly invoice, most AI tool purchases go wrong. Not because the tool is useless — usually it does something — but because the thing it does isn't quite the thing you needed, the data ends up somewhere you didn't want it, and the support you were promised turns out to be a help-centre article from 2023. Six months in, you've spent money, you've spent time, and you're quietly hoping your team has stopped using it so you can cancel without anyone noticing.

This post is the 15-minute filter that sits between the demo and the decision. It won't guarantee you pick the right tool. It will stop you from picking an obviously wrong one. Run through these questions before you hand over a credit card — or before you let someone on your team hand over a credit card on your behalf.

The questions are blunt. That's the point. If a vendor can't answer them plainly, that's an answer too.

Where does your data actually go — and does it stay in Australia?

Ask the vendor directly: which country or region stores your data, and which region processes it? These are different questions. A tool can be headquartered in Sydney and still route your documents through US-based inference servers. Processing location matters as much as storage location.

For most Australian small businesses under $3M in annual turnover, the Privacy Act 1988 doesn't directly apply to you — but that doesn't mean data residency is irrelevant. If you provide a health service, hold a Commonwealth contract, or buy or sell personal information, the Act applies regardless of your size. And even if it doesn't, your clients may have their own requirements about where their personal information lands.

The specific question to ask: "Is data processed on AWS Sydney (ap-southeast-2), Azure Australia East, or another AU-region endpoint — or does it leave Australia at any point?" A vendor who can't answer that cleanly is a vendor who hasn't thought about it. More detail on how to read the answers is in the earlier post on AI privacy questions to ask before you sign anything.

Also ask whether your data is used to train their models. Many consumer-tier tools have training opt-outs buried in account settings. Business tiers often exclude it by default — but often is not always, and you should confirm it in writing, not by reading the FAQ.

Does it actually connect to the software you already use?

Integration demos are the most optimistic moment in any software sales process. The vendor shows you their side of the connection. They do not show you the three manual steps, the CSV export, or the Zapier zap that makes it look seamless. Before you leave the demo, ask for the exact connection method — native API, webhook, middleware, or manual export — and what breaks when the other system changes.

The concrete questions: "Does this connect natively to Xero, MYOB, or whichever accounting system we use? What happens when Xero releases an update — is there an engineer on your side, or do we wait?" If the answer involves a third-party connector tool you'd have to pay for separately, price that in now.

Also ask about what the integration actually does versus what it reads. Some tools connect to your CRM but only pull data — they can't write back. That's fine if reading is all you need, but it's not a two-way integration, and calling it one in the demo is misleading. Get specific about which fields flow which direction, and what happens to records that don't match cleanly.

The integration question matters most for trades, retail, and professional services — the businesses where the back-end system (job management, POS, practice management) is the whole point. An AI tool that floats above your existing software without connecting to it is probably just a better way to type.

What does support actually look like after you sign up?

During the sales process, support is described as responsive, knowledgeable, and available. After you sign, support is an email queue, a bot triage, and a wait. The question to ask before you sign: "If I have a problem on a Tuesday afternoon in AEST, what are my options, and what's the realistic response time?"

If the vendor is US-based and support is US-hours only, that's a 14 to 19 hour gap, depending on the US time zone and the time of year. For a tool handling your quoting, your invoicing, or your client intake, that gap matters. It doesn't mean the tool is wrong for you — it means you need to know in advance and plan for it.

Ask whether there's a dedicated onboarding window or whether setup is self-serve. Most SME-tier AI tools are self-serve with a handful of tutorial videos and a community forum. That's workable if the tool is simple. It's not workable if the tool touches a critical workflow and has a steep configuration curve. If onboarding requires you to hire a consultant to set it up properly, that cost belongs in the total price.

A useful final question: "Can I speak to a customer who's been using this for more than 12 months?" Not a case study. An actual conversation. Vendors who can arrange this quickly are usually vendors with customers who stayed. Vendors who deflect it usually aren't.

What does the contract actually commit you to?

Read the subscription terms before you sign, not after. Four things to look for specifically.

1. Annual vs monthly billing. A tool that charges monthly but locks you into an annual commitment if you want the advertised price isn't monthly billing — it's annual billing with a refund policy. Know what you're actually committing to before the trial period ends.

2. Price escalation clauses. Some contracts allow the vendor to increase pricing with 30 days' notice. Others lock pricing for 12 months. If you're budgeting AI tooling as a line item, the difference between these two scenarios is significant over a two-year horizon.

3. Data return and deletion on exit. When you cancel, what happens to your data? Can you export it, and in what format? How long does the vendor retain it after account closure? A vendor who can't answer this is a vendor who hasn't built a clean exit process — which tells you something about how they think about their relationship with you.

4. Liability caps. Most SaaS contracts cap the vendor's liability at the fees you paid in the prior 12 months. For a $99/month tool, that's $1,188 of protection. If the tool processes your client records and something goes wrong, $1,188 is not much. This doesn't mean don't use SaaS tools — it means understand what you're carrying and make sure your own insurance covers the gap.

Is the pricing actually what it looks like on the pricing page?

AI tool pricing pages are written to show the best-case number. The realistic number is usually higher once you account for seat limits, usage caps, feature gates, and add-ons. Before you commit, build out the realistic monthly cost for your actual team size and actual usage volume.

The specific questions: "At our team size and expected usage, which plan would we be on? Are there usage caps on that plan — document processing limits, API call limits, message limits? What happens when we hit the cap — do we get charged per overage, or does the feature stop working?"

Per-overage charging is common in AI tools because the underlying model costs are real and variable. That's fair. But you need to know whether heavy months cost you 20% more or 200% more. Ask for the overage rate explicitly, and run the numbers against a realistic high-usage scenario, not the average.

Also check whether the features shown in the demo are in the plan you're being sold, or whether they're in a higher tier. It's a common pattern: the demo shows the full product, the proposal quotes the entry tier, and the features you came for are a plan upgrade. Confirm the specific features you need are included in the plan you're pricing.

Does it actually do what it claims, or does it do a version of it?

This is the hardest question to answer in 15 minutes, but it's the most important one. AI tools often describe capabilities in terms of what the underlying model can do, not what the tool has actually built. "AI-powered" on a landing page can mean anything from a fully integrated workflow to a ChatGPT API call with a company logo on top.

The test is to run your actual data through it before you commit. Not a demo document. Not a clean example. Your messiest supplier invoice, your most chaotically formatted client intake form, your worst-formatted spreadsheet export. If the vendor won't let you test with real data before signing, ask why. The answer tells you something.

Also ask about accuracy on real-world input. Most AI tools are trained and benchmarked on clean data. Your data isn't clean. A tool that's 95% accurate on a benchmark dataset might be 70% accurate on the actual documents your business generates — and at 70%, you're reviewing everything anyway, which defeats the point. The relevant metric is accuracy on your data, not accuracy on the vendor's test set. If you can get a two-week trial on a real workflow before any financial commitment, take it.

If you're building something more bespoke — a workflow that actually connects your existing systems rather than sitting next to them — the AI automation service page covers how that's different from buying an off-the-shelf tool.

What should you do if a tool passes all of these?

Start with a single workflow, not a whole business function. Pick the one area where the tool's value is most concrete and most measurable for you — invoice extraction, meeting transcription, intake form parsing, whatever it is — and run it for 30 to 60 days before expanding. Most buyer's remorse comes from buying broadly before proving value narrowly.

Keep a simple log: what did the tool handle correctly, what did it get wrong, and how much time did you or your team spend fixing the errors? That log is worth more than any vendor benchmark. After 30 to 60 days, you'll know whether the tool earns its place or whether it's quietly creating work rather than removing it. Either way, you'll know — which is better than finding out six months in when you're already locked into an annual plan.