Building an AI-Driven API Testing Framework
My team moved to AI-driven development — full specs fed to AI, entire epics delivered at once, no more incremental story releases. As a QA engineer on a backend payments team, I suddenly had a problem: everything landed on my desk complete, ready for validation, all at the same time.
Context
Traditionally, we'd break epics into stories. Developers would deliver one story, I'd validate it, we'd push it live, rinse and repeat. Testing was distributed across the release cycle.
Now? The entire epic arrives complete. Spec finalized, AI writes the plan, AI writes the code, feature delivered. No phased rollout, no incremental testing windows.
The testing load didn't grow linearly — it multiplied.
The Challenge
Work arrives in bulk. There's no spreading the testing load across sprints anymore. Manual testing or manually scripting automation at this pace wasn't an option
And the math makes it worse. I'm one QA supporting over five developers. Before, our team would tackle one, maybe two epics at a time. Now we're running three to four epics concurrently. The streams of work haven't just changed — they've multiplied.
The opportunity: I could build a testing framework driven by our domain knowledge — our specs, our requirements, our QA expertise. Keep quality at the forefront while enhancing the process to match the new pace. That became my focus.
What I Tried Till I Got It Right
I was already writing test cases upfront and supplying them to developers — embedded into the spec so the AI had the right context to generate code with good coverage. But as the pace increased and multiple epics started landing at once, I couldn't keep up. Writing test cases manually for every developer, every epic, in parallel — it was unsustainable. I was becoming the bottleneck I set out to avoid.
So I turned to AI chatbots. I used the chatbot in Cursor to try to generate test scenarios on demand. Some days it gave me exactly what I needed. Other days the same prompt produced something completely different. I couldn't tell if it was hallucinating context, losing history, or just interpreting me differently each time. For QA work that needs to be repeatable and reliable, that inconsistency was a dealbreaker.
So I moved on to explore AI agents — and that changed everything.
The Move to AI Agents
The AI agent system needed to be built with intention — clear rules, embedded domain knowledge, and predictable output. Something that could:
Generate test cases automatically from requirements and domain knowledge
Apply QA best practices — positive, negative, and boundary tests
Stay within project scope, no irrelevant edge cases
Generate executable API test scripts
Deliver consistent results every time
With agents, I could set the rules and guardrails upfront, give each agent explicit context, embed QA best practices directly into the system, and get consistent, predictable output every time.
Now when I invoke my testing agents, the accuracy is solid. I'm not fighting the tool — I'm guiding it with the same knowledge I'd use to mentor a new joiner within the team.
The Solution: An AI-Driven API Testing Framework
I built a coordinated system of AI agents — each with a specific role — powered by a central orchestrator and grounded in domain knowledge. You add in requirements, the pipeline runs, and a validated test report comes out the other end.
The pipeline: Requirements → Orchestrator + Payment Knowledge → 4 Agents → Validated Test Report
The Orchestrator
The Orchestrator is the brain of the system — called first on every stage, whether you run steps individually or trigger Run All. It reads your requirements, queries the Payment Knowledge file, and passes the full context to the right agent. It's not generating random tests; it's generating business-relevant tests every time.
Payment Knowledge
This is where things started to click. I created a file called Payment Knowledge — a structured document containing everything I know about our payments domain: every payment method we support, the validation rules for each one, the regulatory and compliance requirements per jurisdiction, and the edge cases that only surface in production. This file is queried by the Orchestrator before every agent delegation. It's what makes the output consistent and business-relevant rather than generic. The agents don't guess what a valid payload looks like for a specific country — they know, because the knowledge is baked in from the start.
Stage 01 — E2E Test Case Generator
The E2E Test Agent produces behaviour-driven test cases in GIVEN / WHEN / THEN format, covering happy path, negative, and boundary scenarios. After tuning the rules, accuracy sits at ~90%. These same test cases are also supplied to developers embedded in the spec — so when AI writes the code, it has clear coverage targets from the start.
Stage 02 — API Test Case Generator
The API Test Gen Agent strips out non-API steps and maps each scenario to an endpoint, method, and expected response. When a new API is implemented, the contract is passed as part of the requirements and the agent builds a brand new collection from scratch — not just extending an existing one.
Stage 03 — Data & Collections Generator
The Data Gen Agent draws from an existing dataset of payment endpoints — built from my Postman collections — to generate randomised payloads and a fully executable Newman-ready collection. For brand new APIs with no existing dataset, the contract in the requirements is enough to build the whole thing from scratch.
Stage 04 — Execution & Reports
Newman runs the full collection automatically — no manual triggering, no scripting per test. Every failure surfaces with the complete request, response, and assertion context. Two reports are generated per run: an HTML stakeholder report and a Newman debug report.
Results
The Bottom Line
Instead of spending time manually writing test cases or scripting API requests, I can now focus on what actually matters: exploratory testing, validating that the system behaves as expected, and catching the things a script never would. The repetitive work is automated. The judgment work stays with me.
That's the shift. QA expertise becomes more valuable than ever — we hold the domain knowledge, we know how things should work, and now we have AI to amplify it. Take that time back and focus on quality where it actually counts.