[{"content":"Every company I talk to is building an agent. Very few have one in production.\nThe demo is easy now. You wire an LLM to a few tools, add a system prompt, and it looks magical in a screen recording. The production version is a different sport. The gap between the two is where most agent projects quietly die, and it is almost never the model that kills them.\nWhat actually breaks Memory on channels that have no memory. We built an AI gut-health coach on WhatsApp, a channel with no session concept. A user asks about bloating on Monday and says \u0026ldquo;is it better now\u0026rdquo; on Thursday. The agent needs to know what it means. Stuffing the last N messages into the prompt works until users have been chatting daily for weeks, at which point context bloats, costs spike, and the model starts confusing its own instructions with the user\u0026rsquo;s history.\nModel choice as a margin decision. Migrating from GPT-4o to Llama 3.2 was not an infra call. It was a cost-per-resolution call. That only works if you have been running faithfulness and relevancy evals religiously, otherwise you are guessing and shipping a regression.\nThe autonomy boundary. The hardest conversation is never architecture. It is deciding where the AI should stop and a human should start. That line moves every month as models improve, and someone has to own it.\nWhat closes the gap Three things, in order of impact:\nEvals before launch, not after the first bad answer. A test set of real multi-turn scenarios, run on every change to the logic that matters. Escalation paths designed up front. Design the human-in-the-loop before someone is angry, not in response to it. Treat the whole thing as a P\u0026amp;L object. Latency, cost per resolution, and headcount saved belong in the same spreadsheet. None of this is glamorous. The model was fine way earlier than you think. The product questions around it are the actual work.\n","permalink":"https://www.shivamtiwari.si/posts/agent-demo-to-production/","summary":"Most agent projects die somewhere between the demo and the first real customer. Here is what actually closes that gap.","title":"The gap between an agent demo and production"},{"content":"The nutritionists almost killed the project.\nWe had a team of eight in-house nutritionists who had been running WhatsApp consultations manually for over a year. When I told them we were shipping an AI agent to do the same thing, the first reaction was not excitement. It was \u0026ldquo;so you are replacing us.\u0026rdquo;\nThey were right to feel that way. From their side, some product guy walks in and says we trained an AI on your expertise, and now it talks to your users.\nThe first two weeks were rough They found every wrong answer the agent gave and screenshotted it to the founders. Some of those screenshots were valid. The agent was confidently telling a user with acidity to eat citrus. Stuff a nutritionist would never say.\nThe instinct is to call that resistance. It is not. It is signal, delivered badly.\nWhat unblocked it We made them the eval team. Not as busywork. Genuinely. They became the people who decided whether a response was good enough. Their rubric, their call.\nSuddenly they were not being replaced. They were the quality bar the AI had to clear.\nThe second-order effect was the important one. They stopped flagging individual failures and started flagging patterns. \u0026ldquo;It keeps getting fermented foods wrong for PCOS users\u0026rdquo; is far more useful than a screenshot of one bad response.\nMonth three By month three they were requesting features. \u0026ldquo;Can it ask about their cycle day before recommending probiotics?\u0026rdquo;\nThat is when I knew it worked. The team that was most threatened by the agent had become its most invested reviewer.\nThe lesson generalises. The people closest to the domain are the best eval authors you have. The trick is giving them ownership over quality instead of asking them to defend their jobs.\n","permalink":"https://www.shivamtiwari.si/posts/nutritionists-should-own-evals/","summary":"The team most threatened by your AI agent is the team best positioned to define whether it is good enough. Here is how that turned into our quality bar.","title":"Your nutritionists should own your evals"},{"content":"I am an AI Product leader with 7 years shipping agentic AI, conversational AI and NLP systems at consumer scale. Right now I lead Product and AI at The Good Bug, a Series B consumer health company, where I took the AI and Product org from 0 to 1.\nMy career started as a founder. I co-founded Apli.ai, which grew to 1,200+ institutions and 6.5L+ candidates before being acquired by Edulab. From there I went to Lenskart, where I shipped a GPT-4 powered eyewear recommendation engine and the Face Shape AI that lifted conversion 2.5x. That work taught me that the hardest part of AI shipping is rarely the model. It is deciding where the model should stop and a human should start.\nHow I think about AI products The demo is easy now. The production version is a different sport, and the gap between the two is where most agent projects die.\nEvals are infrastructure, not a chore. Faithfulness and relevancy harnesses gate every model swap. If you have not been running them before the change, you are guessing. Model choice is a margin decision. Cost per resolution sits in the same spreadsheet as latency and headcount saved. Quality parity at lower cost is a product win. Design the autonomy boundary first. Decide where the AI stops and a human starts before launch, not after the first angry customer. Domain experts own quality. The team closest to the problem makes the best eval authors. Give them ownership, not a threat. What I do day to day Full stack AI ownership: model selection, retrieval and memory architecture, eval pipelines, inference cost, and the P\u0026amp;L behind all of it. I have run migrations across GPT-4o, Llama 3.2 and Mistral, and I build the teams and roadmaps that ship them.\nBeyond the day job Lead Instructor at Google for Startups, teaching AI product and prototyping to early-stage founders across India. Visiting Faculty and speaker at IIT Bombay, Mesa School of Business, Masters\u0026rsquo; Union and 20+ events on Gen AI and Product. Open source contributor with merged PRs in repos like Ollama, LangChain, SWE-Agent, Deno and Jaeger. Writing about agentic AI, evals and AI product strategy for ~24K LinkedIn followers, mostly D2C founders and PMs. Currently Based in Abu Dhabi. Building AI products, consulting through Nexhumans, and tinkering on side projects like an AI voice companion for elderly parents and a macOS workflow agent.\nThe fastest way to reach me is LinkedIn, or email at iitb.shivam@gmail.com.\n","permalink":"https://www.shivamtiwari.si/about/","summary":"\u003cp\u003eI am an AI Product leader with 7 years shipping agentic AI, conversational AI and NLP systems at consumer scale. Right now I lead Product and AI at \u003cstrong\u003eThe Good Bug\u003c/strong\u003e, a Series B consumer health company, where I took the AI and Product org from 0 to 1.\u003c/p\u003e\n\u003cp\u003eMy career started as a founder. I co-founded \u003cstrong\u003eApli.ai\u003c/strong\u003e, which grew to 1,200+ institutions and 6.5L+ candidates before being acquired by Edulab. From there I went to \u003cstrong\u003eLenskart\u003c/strong\u003e, where I shipped a GPT-4 powered eyewear recommendation engine and the Face Shape AI that lifted conversion 2.5x. That work taught me that the hardest part of AI shipping is rarely the model. It is deciding where the model should stop and a human should start.\u003c/p\u003e","title":"About"},{"content":"AI Product leader with 7 years shipping agentic AI, conversational AI and NLP systems at consumer scale. Founder-to-exit, full-stack AI ownership across model selection, evals, inference cost and P\u0026amp;L.\nHead of Product \u0026amp; AI The Good Bug · Series B, INR 100 CR · Reporting to the founder · Mar 2025 – Present\nDefined and shipped Bugzy, an agentic AI Gut Coach on WhatsApp with RAG, multi-turn memory and hallucination guardrails; scaled to 200K+ users in 6 months with a 1.5x retention uplift. Architected Goodsy as a multi-step AI agent with tool-use, memory and HITL escalation; autonomously resolves 65% of 24,000 monthly support tickets, saving INR 3CR+ in annual support opex. Owned model selection and migration strategy across GPT-4o, Llama 3.2 and Mistral using RAGAS faithfulness and answer-relevancy evals; selected Llama 3.2 11B for a 30% inference cost reduction at 85%+ quality parity. Built the AI and Product org 0-to-1; hired a 10-member team in 6 months across Product, Design and Engineering; owned the full annual budget spanning compensation, LLM infrastructure and vendor tools. Set the quarterly AI roadmap and OKRs across Product, Design and Engineering; led model performance reviews and GenAI strategy directly with the co-founding team. Senior AI-Product Manager Lenskart · NSE/BSE Listed, $8B IPO 2025 · Feb 2023 – Feb 2025\nLead Product Manager, John Jacobs (Lenskart) · Reporting to the CEO · Feb 2023 – Mar 2024\nShipped a GPT-4 powered eyewear recommendation engine using face-shape detection and prompt chaining, evaluated across 500+ test cases before launch, lifting conversion from 3% to 7.5% (2.5x uplift) in 3 months. Originated the AI stylist concept behind Lenskart B, built the business case projecting ₹25CR revenue in 7 months, and designed a lifestyle-based frame assistant that delivered 8x conversion uplift in prototype testing. Architected a Shopify-like multi-storefront platform for Lenskart Brands, enabling rapid launch of tailored storefronts within hours at a 30% cost reduction. Led a cross-functional pod of 1 PM, 1 GM and 12 Engineers, delivering 48% YoY revenue growth. Built and led GTM for online eye tests reaching 1,000+ users daily at 70% accuracy; launched affiliate and partnership channels with 12 partners in one quarter, driving 22% user acquisition growth. Achieved 28% higher conversion by streamlining the PLP-to-cart flow and simplifying prescription lens selection. Cofounder \u0026amp; CEO Apli.ai · Acquired by Edulab · Aug 2019 – Dec 2022\nBuilt and exited Apli.ai via acquisition by Edulab, scaling from zero to 6.5L+ candidates, 1,200+ institutions and 400+ companies in 18 months; drove 150% revenue growth pivoting to a hiring-success model. Designed an NLP-based job-matching engine as a ranking problem with A/B tested scoring logic, improving match accuracy 35% and interview success rates 40%. Led a 15-member cross-functional team across product, engineering and growth; raised an angel round from leaders at MIT, IIT Bombay and Impact Fund; selected for the IIT Startups Accelerator Cohort 8 out of 2000+ applications. Skills Core: Agentic AI Systems, RAG Architecture, LLM Evaluation (RAGAS, regression evals), Foundation Model Selection and Migration, Prompt Engineering, Inference Cost Optimization, Vector Databases, Conversational AI, HITL Design, MLOps/LLMOps, Responsible AI, 0-to-1 Product Building, AI Product Strategy, Cross-functional Leadership, OKRs, P\u0026amp;L Management, A/B Testing, GTM, Stakeholder Management.\nTools: Figma, SQL, Jira, Confluence, Notion, Shopify, Appsflyer, Google Analytics, Claude Code, GitHub, Vercel.\nEducation IIT Bombay · IDEAS Program (Preincubation), Desai Sethi School of Entrepreneurship · 2020\nMumbai University · Mechanical Engineering, Lokmanya Tilak College of Engineering · 2019\n","permalink":"https://www.shivamtiwari.si/experience/","summary":"\u003cp\u003eAI Product leader with 7 years shipping agentic AI, conversational AI and NLP systems at consumer scale. Founder-to-exit, full-stack AI ownership across model selection, evals, inference cost and P\u0026amp;L.\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"head-of-product--ai\"\u003eHead of Product \u0026amp; AI\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003eThe Good Bug\u003c/strong\u003e · Series B, INR 100 CR · Reporting to the founder · Mar 2025 – Present\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eDefined and shipped \u003cstrong\u003eBugzy\u003c/strong\u003e, an agentic AI Gut Coach on WhatsApp with RAG, multi-turn memory and hallucination guardrails; scaled to \u003cstrong\u003e200K+ users\u003c/strong\u003e in 6 months with a 1.5x retention uplift.\u003c/li\u003e\n\u003cli\u003eArchitected \u003cstrong\u003eGoodsy\u003c/strong\u003e as a multi-step AI agent with tool-use, memory and HITL escalation; autonomously resolves \u003cstrong\u003e65% of 24,000 monthly support tickets\u003c/strong\u003e, saving INR 3CR+ in annual support opex.\u003c/li\u003e\n\u003cli\u003eOwned model selection and migration strategy across GPT-4o, Llama 3.2 and Mistral using RAGAS faithfulness and answer-relevancy evals; selected Llama 3.2 11B for a \u003cstrong\u003e30% inference cost reduction\u003c/strong\u003e at 85%+ quality parity.\u003c/li\u003e\n\u003cli\u003eBuilt the AI and Product org 0-to-1; hired a \u003cstrong\u003e10-member team in 6 months\u003c/strong\u003e across Product, Design and Engineering; owned the full annual budget spanning compensation, LLM infrastructure and vendor tools.\u003c/li\u003e\n\u003cli\u003eSet the quarterly AI roadmap and OKRs across Product, Design and Engineering; led model performance reviews and GenAI strategy directly with the co-founding team.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"senior-ai-product-manager\"\u003eSenior AI-Product Manager\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003eLenskart\u003c/strong\u003e · NSE/BSE Listed, $8B IPO 2025 · Feb 2023 – Feb 2025\u003c/p\u003e","title":"Experience"},{"content":"A selection of things I have shipped and things I am building. Some are work, some are bets.\nBugzy WhatsApp AI gut health coach · at The Good Bug\nAn agentic health coach on WhatsApp built with RAG, multi-turn memory and hallucination guardrails. Scaled to 200K+ users in 6 months with a 1.5x retention uplift. The hardest part was multi-turn memory on a channel with no session concept, which we solved with a summarise-and-tag memory layer plus a 200+ scenario eval set.\nGoodsy Agentic customer experience system · at The Good Bug\nA multi-step agent with tool-use, memory and human-in-the-loop escalation. It processes refunds, modifies orders and changes subscriptions. Autonomously resolves 65% of 24,000 monthly tickets, saving INR 3CR+ in annual support opex. Shipped with a tiered autonomy model: small refunds the agent handles directly, larger ones queue for human confirmation.\nFace Shape AI Eyewear recommendation engine · at Lenskart\nA GPT-4 powered recommendation engine combining face-shape detection with prompt chaining. Evaluated on 500+ test cases and lifted conversion from 3% to 7.5% (2.5x) within 3 months.\nApli.ai Hiring platform · co-founded, acquired\nAn NLP-based job-matching engine built as a ranking problem with A/B tested scoring. Grew to 1,200+ institutions, 6.5L+ candidates and 400+ companies in 18 months, then acquired by Edulab.\nCallHome AI voice companion for elderly parents · side project\nDaily outbound Hindi AI calls to elderly parents of NRI families, with WhatsApp summaries and a weekly digest. Stack: Sarvam AI, VAPI, Claude API, Exotel, Supabase, WhatsApp Business API.\nSparq Kids AI companion · side project\nA dual-account kids AI app (parent B2C + school B2B), built to COPPA/FERPA/UK Children\u0026rsquo;s Code standards. Stack: React Native/Expo, Supabase, Claude Sonnet, DALL-E 3, Next.js.\nMimic macOS desktop agent · side project\nA macOS agent that passively observes screen workflows and automates them. Thesis: AI workflows break at the UI/prompt layer, and the fix belongs at the OS level. Applied to YC; continuing to build.\nNexhumans AI-native post-purchase ops · consulting\nAn AI-native post-purchase operations agency for Indian D2C brands, priced on outcomes.\nInterested in any of these or want to work together? Reach me on LinkedIn.\n","permalink":"https://www.shivamtiwari.si/projects/","summary":"\u003cp\u003eA selection of things I have shipped and things I am building. Some are work, some are bets.\u003c/p\u003e\n\u003ch2 id=\"bugzy\"\u003eBugzy\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003eWhatsApp AI gut health coach\u003c/strong\u003e · at The Good Bug\u003c/p\u003e\n\u003cp\u003eAn agentic health coach on WhatsApp built with RAG, multi-turn memory and hallucination guardrails. Scaled to \u003cstrong\u003e200K+ users in 6 months\u003c/strong\u003e with a 1.5x retention uplift. The hardest part was multi-turn memory on a channel with no session concept, which we solved with a summarise-and-tag memory layer plus a 200+ scenario eval set.\u003c/p\u003e","title":"Projects"},{"content":"Research \u0026amp; writing I write about agentic AI systems, LLM evaluation and AI product strategy, mostly for an audience of D2C founders and product managers. My work focuses on the gap between an agent demo and a production system: memory design, evaluation methodology, inference cost and where autonomy should stop.\nReading notes and essays on the blog Google Scholar Selected themes:\nEvaluation as infrastructure — building faithfulness and relevancy eval harnesses (RAGAS, regression suites) that gate every model swap. Memory architecture — retrieval and summarisation layers that give stateless channels like WhatsApp durable multi-turn memory. Model economics — treating foundation model selection as a margin decision, benchmarked on cost per resolution rather than raw quality. Speaking Google for Startups — Lead Instructor, teaching AI product and rapid prototyping to early-stage founders across India. IIT Bombay — Visiting Faculty, guest sessions on generative AI and product. Masters\u0026rsquo; Union and Mesa School of Business — guest lectures on AI product management. 20+ events across India on generative AI, agentic systems and product strategy. Teaching I teach practitioners how to go from an AI prototype to something they can trust in production, covering prompting, retrieval, evaluation and the product decisions that surround them.\nAI product and prototyping curriculum for founders (Google for Startups) Agentic AI and LLM evaluation modules (executive and MBA audiences) Guest workshops on AI product management Open source I contribute to widely used AI and infrastructure projects. Merged PRs across repos totalling 500K+ stars:\nProject Stars Link Ollama 166K github.com/ollama/ollama LangChain 131K github.com/langchain-ai/langchain Deno 106K github.com/denoland/deno Backstage 33K github.com/backstage/backstage Biome 24.2K github.com/biomejs/biome Jaeger 22.6K github.com/jaegertracing/jaeger SWE-Agent 18.9K github.com/SWE-agent/SWE-agent See my work on GitHub.\nRecognition Won the Global Entrepreneurship Bootcamp (35+ countries), organised by MIT faculty and alumni, Indonesia. Selected for the IIT Startups Accelerator Cohort 8 out of 2000+ applications. Want me to speak at your event or run a workshop? Reach me on LinkedIn.\n","permalink":"https://www.shivamtiwari.si/research/","summary":"\u003ch2 id=\"research--writing\"\u003eResearch \u0026amp; writing\u003c/h2\u003e\n\u003cp\u003eI write about agentic AI systems, LLM evaluation and AI product strategy, mostly for an audience of D2C founders and product managers. My work focuses on the gap between an agent demo and a production system: memory design, evaluation methodology, inference cost and where autonomy should stop.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"/posts/\"\u003eReading notes and essays on the blog\u003c/a\u003e\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://scholar.google.com/citations?user=SHIVAM_SCHOLAR_ID\"\u003eGoogle Scholar\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eSelected themes:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eEvaluation as infrastructure\u003c/strong\u003e — building faithfulness and relevancy eval harnesses (RAGAS, regression suites) that gate every model swap.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eMemory architecture\u003c/strong\u003e — retrieval and summarisation layers that give stateless channels like WhatsApp durable multi-turn memory.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eModel economics\u003c/strong\u003e — treating foundation model selection as a margin decision, benchmarked on cost per resolution rather than raw quality.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"speaking\"\u003eSpeaking\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eGoogle for Startups\u003c/strong\u003e — Lead Instructor, teaching AI product and rapid prototyping to early-stage founders across India.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eIIT Bombay\u003c/strong\u003e — Visiting Faculty, guest sessions on generative AI and product.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eMasters\u0026rsquo; Union\u003c/strong\u003e and \u003cstrong\u003eMesa School of Business\u003c/strong\u003e — guest lectures on AI product management.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003e20+ events\u003c/strong\u003e across India on generative AI, agentic systems and product strategy.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"teaching\"\u003eTeaching\u003c/h2\u003e\n\u003cp\u003eI teach practitioners how to go from an AI prototype to something they can trust in production, covering prompting, retrieval, evaluation and the product decisions that surround them.\u003c/p\u003e","title":"Research, Speaking \u0026 Teaching"}]