“How to test a Test Planning AI Agent”
Let’s be honest: building a cool AI agent is pretty easy these days. But keeping it from breaking in production? That’s a different story. We know how to automate stuff. But with LLMs, we need to define what is “working”. Then we need to check nondeterministic answers automatically. We like AI, but we want to make sure it behaves. So, let’s do this to a real Test Planning Agent.
In this session, I’m skipping the high-level slides (but keeping the memes) to show you the thinking and tests for our non-deterministic agent. Here is what we’ll walk through:
- Sanity checks and guardrails - the first line of defense.
- Golden datasets - define what’s good, bad, and in between.
- Automatic scorecards - translate the quality definitions into automated tests
- Performance and security: A whole set of new things we need to think about in the age of testing AI
And a bit more. Maybe a couple of pirates. If you’re a QA or automation engineer trying to figure out how to apply your testing skills to AI applications, dip your toes, it’s a big ocean out there.
Bio
Gil Zilberfeld has been a software practitioner since the days of his Sinclair ZX81. With over 25 years of hands-on experience, he is a consultant and trainer who helps testing professionals and teams elevate their craft. Gil challenges outdated QA dogma in favor of pragmatic, high-impact testing strategies and believes quality is a team sport. From mastering web automation to developing robust test strategies for AI-powered systems, he provides actionable frameworks for today’s quality challenges. You can find his practical advice and courses at testingil.com.