Join the Clarity open source harness waitlist: https://heyclarity.dev/
A 95% accurate AI judge can still be completely useless. If errors only show up 5% of the time, saying “no errors” every time already scores 95%, which is why you measure precision and recall instead.
I sat down with Hamel Husain, who has taught over 5,000 builders how to measure and improve AI products, to go through his best evals memes and the real mistake behind each one. We cover why a 100% pass rate is a warning sign, why generic scores like coherence and helpfulness tell you nothing, how to sample your traces, and why you should build your own annotation app.
If you’re an AI builder or a tech exec who has a dashboard full of scores nobody can explain, you need to watch this.
- hamel.dev hamel.dev
Hosted by Robert Ta and Jonathan McCoy, co-founders of Clarity, reporting from the front lines of building an AI company.