Eleven questions to ask when someone pitches you an AI project

A vendor, or your own team, is presenting an AI project. You have forty minutes and no technical background. These are the questions that reliably separate a considered proposal from an enthusiastic one.

They're ordered roughly by how often the answer turns out to be revealing.

On the problem

1. What happens today, step by step, and which of those steps is the expensive one?

If nobody can describe the current process in detail, nobody has measured it, which means nobody will be able to tell you afterward whether this worked. The manual process you have now is the specification, the baseline, and the test set all at once.

2. What does success look like as a number, and what is that number today?

"Improved efficiency" is not an answer. "Ninety seconds per ticket, down from four minutes" is. Without a baseline you can't tell improvement from noise, and in six months you'll be arguing about impressions.

3. Would a simpler thing work?

Some proportion of AI proposals are a search feature, a form, or a better-organized wiki with a language model bolted on top. The question costs nothing to ask and occasionally saves a year.

On the failure modes

4. When it gets one wrong, what happens?

The single most important question in the list. It sets how much accuracy you actually need, whether a human has to sit in the path, and what the real risk is. A wrong draft email is a nuisance. A wrong dosage or a wrong refund is not.

5. Who notices when it's wrong, and how?

If the answer is "the user," ask whether the user is positioned to. Someone asking a question generally can't evaluate the answer; that's why they asked.

6. Is it working from our documents, or from what it learned in training?

If it's answering out of its own training, it will be confidently wrong about anything specific to you, and it won't know about anything recent. If it's working from your documents, the follow-up is which ones, who keeps them current, and what happens when they change.

7. What's the plan for when quality drops six months in?

These systems degrade quietly as the input mix drifts and source documents go stale. Nothing errors. If there's no named owner running periodic quality checks, you're buying something that will decline without anyone noticing.

On cost

8. What does one transaction cost, and what's inside that number?

You're billed for the text going in as well as the text coming out, and the input is usually much larger than people assume. Instructions, retrieved documents, and conversation history all count toward it.

9. Does cost scale with users, with usage, or with conversation length?

These behave very differently as you grow. Conversation-length scaling surprises people in particular, because every turn resends everything before it.

10. Which parts of every request are identical, and are we getting the repeat discount on them?

Repeated content can be billed at roughly a tenth of the normal rate if the system is built so the unchanging part comes first. It's an architecture decision rather than a later optimization, so ask before it's built.

On the thing everyone forgets

11. Who is going to use this, and what have we asked them?

Most AI projects that fail don't fail technically. They work, and nobody adopts them, because the workflow they replaced was fine, or the new one is slower, or people don't trust the output and check it by hand anyway. At which point you've added a step rather than removed one.

Microsoft's AI Transformation Leader blueprint gives adoption strategy a fifth of the exam: adoption teams, common barriers, champions programs. That weighting is not an accident. It's the part that decides whether any of the rest mattered.

How to read the answers

You are not assessing technical correctness. You're listening for three things.

Specificity. "It's very accurate" versus "94% on a 200-case set we built from last quarter's tickets, and here's where it falls over." The second person has done the work.

Awareness of failure. Anyone who has actually shipped one of these can tell you how it goes wrong without being prompted. Presenting only upside means either inexperience or a sales posture, and it doesn't much matter which one you're looking at.

Willingness to say "I don't know." The strongest signal in the room, and the one people underrate most. Someone who says "we haven't measured that yet" is telling you the truth about one thing, and is probably telling you the truth about the rest.