Shivam Tiwari

The gap between an agent demo and production

Tue, 15 Sep 2026 00:00:00 +0000

Most agent projects die somewhere between the demo and the first real customer. Here is what actually closes that gap.

Your nutritionists should own your evals

Sun, 02 Aug 2026 00:00:00 +0000

The team most threatened by your AI agent is the team best positioned to define whether it is good enough. Here is how that turned into our quality bar.

About

Mon, 01 Jan 0001 00:00:00 +0000

I am an AI Product leader with 7 years shipping agentic AI, conversational AI and NLP systems at consumer scale. Right now I lead Product and AI at The Good Bug, a Series B consumer health company, where I took the AI and Product org from 0 to 1.

My career started as a founder. I co-founded Apli.ai, which grew to 1,200+ institutions and 6.5L+ candidates before being acquired by Edulab. From there I went to Lenskart, where I shipped a GPT-4 powered eyewear recommendation engine and the Face Shape AI that lifted conversion 2.5x. That work taught me that the hardest part of AI shipping is rarely the model. It is deciding where the model should stop and a human should start.

Experience

Mon, 01 Jan 0001 00:00:00 +0000

AI Product leader with 7 years shipping agentic AI, conversational AI and NLP systems at consumer scale. Founder-to-exit, full-stack AI ownership across model selection, evals, inference cost and P&L.

Head of Product & AI

The Good Bug · Series B, INR 100 CR · Reporting to the founder · Mar 2025 – Present

Defined and shipped Bugzy, an agentic AI Gut Coach on WhatsApp with RAG, multi-turn memory and hallucination guardrails; scaled to 200K+ users in 6 months with a 1.5x retention uplift.
Architected Goodsy as a multi-step AI agent with tool-use, memory and HITL escalation; autonomously resolves 65% of 24,000 monthly support tickets, saving INR 3CR+ in annual support opex.
Owned model selection and migration strategy across GPT-4o, Llama 3.2 and Mistral using RAGAS faithfulness and answer-relevancy evals; selected Llama 3.2 11B for a 30% inference cost reduction at 85%+ quality parity.
Built the AI and Product org 0-to-1; hired a 10-member team in 6 months across Product, Design and Engineering; owned the full annual budget spanning compensation, LLM infrastructure and vendor tools.
Set the quarterly AI roadmap and OKRs across Product, Design and Engineering; led model performance reviews and GenAI strategy directly with the co-founding team.

Senior AI-Product Manager

Lenskart · NSE/BSE Listed, $8B IPO 2025 · Feb 2023 – Feb 2025

Projects

Mon, 01 Jan 0001 00:00:00 +0000

A selection of things I have shipped and things I am building. Some are work, some are bets.

Bugzy

WhatsApp AI gut health coach · at The Good Bug

An agentic health coach on WhatsApp built with RAG, multi-turn memory and hallucination guardrails. Scaled to 200K+ users in 6 months with a 1.5x retention uplift. The hardest part was multi-turn memory on a channel with no session concept, which we solved with a summarise-and-tag memory layer plus a 200+ scenario eval set.

Research, Speaking & Teaching

Mon, 01 Jan 0001 00:00:00 +0000

Research & writing

I write about agentic AI systems, LLM evaluation and AI product strategy, mostly for an audience of D2C founders and product managers. My work focuses on the gap between an agent demo and a production system: memory design, evaluation methodology, inference cost and where autonomy should stop.

Selected themes:

Evaluation as infrastructure — building faithfulness and relevancy eval harnesses (RAGAS, regression suites) that gate every model swap.
Memory architecture — retrieval and summarisation layers that give stateless channels like WhatsApp durable multi-turn memory.
Model economics — treating foundation model selection as a margin decision, benchmarked on cost per resolution rather than raw quality.

Speaking

Google for Startups — Lead Instructor, teaching AI product and rapid prototyping to early-stage founders across India.
IIT Bombay — Visiting Faculty, guest sessions on generative AI and product.
Masters’ Union and Mesa School of Business — guest lectures on AI product management.
20+ events across India on generative AI, agentic systems and product strategy.

Teaching

I teach practitioners how to go from an AI prototype to something they can trust in production, covering prompting, retrieval, evaluation and the product decisions that surround them.