Ale CabreraHubSpot · 2024
AI Data Enhancement
AI-powered enhancement of company data across 80 million domains and 50+ attributes, built so that more coverage did not mean less trust.
Outcome
- 100%data coverage
- 80% → <2%hallucination, from the start to release
- 50+attributes, rolled out in 3 steps: top 2, top 5, then all
- ↑real-time data delivered faster, with fewer errors and shorter resolution time
- Coverage was stuck around 50%. Traditional approaches and providers could not fill the rest fast enough or at a reasonable cost.
- AI could raise coverage and speed, but generating more data has no value if customers cannot trust it.
- Attributes behave differently. A headquarters address stays stable for years. Employee count changes all the time.
- So quality could not have one single definition.
- Ran Llama, GPT and Claude against the same evaluation set, comparing quality, cost and coverage.
- Llama was attractive on cost. GPT and Claude were more expensive at that scale.
- An AI call for everything was too expensive and too slow. Developed embeddings to save budget and time.
- Hallucinated data and stale data were two different problems and needed different detection.
- Used customer feedback to decide which attributes mattered, and started with the top 2, revenue and employee count, instead of all 50+.
- Shipped v1 at 80% coverage on the top 2 attributes: revenue and employee count.
- Expanded to the top 5, adding HQ, industry and public or private status.
- Then rolled out to all attributes.
- Detected hallucinations with manual validation. Humans reviewed samples before each step.
- Measured freshness with manual validation plus metrics.
- Brought hallucination from around 80% to under 2%.
My Role
Owned the product strategy, the validations, the prioritization and the rules, all based on customer feedback.
Engineering Collaboration
Engineering and data science built the models, the embeddings and the pipelines. I brought the customer feedback, the validation rules and the priorities. The validation results decided when each rollout step was ready.
Problem & Risk
More generated data is only useful if it is right. The value risk was wrong or stale data reaching customers. The business risk was model cost at 80 million domains.
Customer Discovery & Validation
- Customer feedback on which attributes mattered most.
- One evaluation set used to compare all three models.
- Manual validation of samples for hallucination and freshness.
- Early outputs were wrong around 80% of the time, which set the bar for what had to be fixed before release.
Context
Company data enrichment across about 80 million domains, with coverage at about half and 50+ attributes to fill.
Timeline
What We Did
- Compared Llama, GPT and Claude on one evaluation set.
- Replaced an AI call for everything with embeddings, to save budget and time.
- Defined quality per attribute: correctness, groundedness, coverage, freshness.
- Validated samples by hand for hallucination and freshness.
- Wrote the validation rules from customer feedback.
- Rolled out by attribute priority: top 2, top 5, all.
one evaluation set, 3 models
Llama, GPT, Claude compared on quality, cost, coverage
top 2 live: revenue, employee count
top 5 live
all 50+ attributes live
Who Benefits
Customers get more coverage on the attributes they care about, in real time, with data they can trust.
Next Steps
Increase accuracy.