AI Quality Is a Product Problem, Not Just a Model Problem

A model can give a correct answer and still deliver a bad product experience.

When teams build AI products, a lot of attention goes into choosing the model. Which one is more accurate? Faster? Cheaper?

Those are important questions. But they don't answer the most important one:

Is the AI actually solving the customer's problem?

A correct answer isn't always a useful answer

A customer asks for the number of employees at a company. Your AI returns 250.

Is that a good answer? It depends.

Maybe the company had 250 last year and has 400 today. Maybe the customer just needs to know if it has more than 200. Maybe their sales team uses that number to decide which accounts to prioritize.

Same answer. Fine in one context, a problem in another.

Not all mistakes are equally important

At Clearbit and HubSpot, I worked with company data covering millions of domains and dozens of attributes. They don't behave the same way. A headquarters address can stay unchanged for years. Employee count changes much more often.

So before improving everything, I'd start with three things:

• Relevance: does this matter to the customer's workflow?
• Reliability: is it correct and current enough?
• Impact: what happens when it's wrong?

A better model won't fix everything

An AI answer can fail in 5 ways: invented, grounded but wrong, correct but outdated, correct but irrelevant, or inconsistent.

Each one needs a different fix. Sometimes the problem isn't the model at all.

On AI Data Enhancement at HubSpot, hallucination went from around 80% to under 2%. Read the one-pager.

Before setting an accuracy target, I'd ask

1. What decision is the customer making with this output?
2. What happens if the answer is wrong? Can they see the error and correct it?
3. Is it better to return no answer than a potentially wrong one?

Sometimes the best product decision is to return: "We don't have enough reliable information."

The real question

It isn't whether your AI works. It's whether it works for your customers.

Model quality tells you how well the technology performs. Product quality tells you whether that performance creates value. You need both.

Building an AI product? Send me an AI feature that isn't delivering the results you expected. I'll tell you the first three things I'd investigate.