data-catalog.mdZapier · 2021
Building a Self-Serve Data Catalog
ShippedTeam: Data3 weeks
TL;DR
- Centralized, validated, searchable data documentation in Looker.
- Turned a scattered, confusing data landscape into a self-service resource for the whole company.
Outcome
- ↓time to resolve data requests
- ↑new-hire self-service, no data-team bottleneck
Saw It
- Data team members, power users, and new hires kept independently asking for a "single source of truth."
- Every vendor we evaluated pointed to the same root cause: no organized metadata layer.
↓→
Figured It Out
- Rather than wait months for procurement, scoped a lightweight ETL.
- Used the metadata already being gathered for the vendor evaluation.
- Turned the eval itself into a live validation vehicle.
↓→
Did It
- Shipped v1 in Looker in 2 weeks.
- When feedback interviews called it "confusing," owned the restructure into v2 rather than treating the miss as final.
My Role
- Owned the build-vs-buy evaluation and scoped the metadata ETL as the validation vehicle.
- Ran the vendor conversations.
- Synthesized the "confusing and overwhelming" feedback into the v2 restructure.
- Made the call to ship a stopgap rather than wait on procurement.
Engineering Collaboration
- Partnered directly with a data engineer on the ETL design.
- Trade-off discussion: speed of extraction vs. completeness of metadata coverage.
- Chose speed to get a testable v1 in front of users faster, backfilling coverage in v2.
Full breakdown
Problem & Risk
Usability Risk
- Data documentation was scattered across formats.
- Every data request required manual triage, slowing onboarding.
- Data team wasting time finding and validating data sources rather than analysis.
Customer Discovery & Validation
Evidence
- 20 internal customer interviews: repeated, unprompted requests for a "single source of truth" from data team members, power users, and new hires.
- "Build vs. buy" vendor evaluation: same root problem to solve, no organized metadata layer.
Two different sources agree before we wrote a line of code.
Context
- Mid-evaluation on catalog vendors, needed a decision, not just a wishlist.
- Used the metadata gathered during vendor scoping to test the actual need in production.
- Solved the immediate pain shipping a good quality usable product, while generating real evidence for the build-vs-buy process and definite tool.
Timeline
git log --oneline⏱ 3 weeks to v2
kickoff: vendor "build vs. buy" evaluation beginsWeek 0
discovery: metadata mapped across full data ecosystemWeek 1
ship v1: Looker catalog live, basic filtersWeek 2
feedback: feedback interviews, users flag v1 as "confusing and overwhelming"Week 3
ship v2: restructured by user need, feedback loop added, simplified main UXNext
What We Did
- Built a lightweight ETL to extract and map the entire data ecosystem, exposed through Looker with filters for searchability.
- First-round usage testing called it "confusing and overwhelming," a signal we didn't ignore.
- Rebuilt into separate views by user need, added a feedback + feature-request loop.
~ rough sketch, not an actual screenshot ~
data glossary
every event + table, defined
documentation
models + FAQs, linked
data assets
source + type, discoverable
lineage
(soon) pipeline map
quality + health
live monitoring status
taxonomy
naming + classification
Who Benefits
- New hires onboard faster without waiting on a data team member.
- Data scientists decreasing time to find sources.
- Non-technical stakeholders self-serve insights instead of filing tickets.
- The business gained validated, production-tested evidence for the vendor decision, de-risking a procurement spend before it happened.
Next Steps
- Complete the build-vs-buy evaluation.
- Migrate to a more sophisticated catalog solution using the validated metadata model as the foundation.