The nutritionists almost killed the project.
We had a team of eight in-house nutritionists who had been running WhatsApp consultations manually for over a year. When I told them we were shipping an AI agent to do the same thing, the first reaction was not excitement. It was “so you are replacing us.”
They were right to feel that way. From their side, some product guy walks in and says we trained an AI on your expertise, and now it talks to your users.
The first two weeks were rough
They found every wrong answer the agent gave and screenshotted it to the founders. Some of those screenshots were valid. The agent was confidently telling a user with acidity to eat citrus. Stuff a nutritionist would never say.
The instinct is to call that resistance. It is not. It is signal, delivered badly.
What unblocked it
We made them the eval team. Not as busywork. Genuinely. They became the people who decided whether a response was good enough. Their rubric, their call.
Suddenly they were not being replaced. They were the quality bar the AI had to clear.
The second-order effect was the important one. They stopped flagging individual failures and started flagging patterns. “It keeps getting fermented foods wrong for PCOS users” is far more useful than a screenshot of one bad response.
Month three
By month three they were requesting features. “Can it ask about their cycle day before recommending probiotics?”
That is when I knew it worked. The team that was most threatened by the agent had become its most invested reviewer.
The lesson generalises. The people closest to the domain are the best eval authors you have. The trick is giving them ownership over quality instead of asking them to defend their jobs.