End-to-end acceptance testing with AI agents

Challenge

Acceptance testing a customisation on the client’s system was manual from start to finish. The developer remembered what had changed, wrote a script, hunted for real data that would work for the test, ran it screen by screen and wrote down the result. The script came out in the programmer’s vocabulary, hard for the client’s key user to follow, and edge cases fell off when the deadline got tight.

Solution

I built a tool with three commands that orchestrates four AI agents, each with a role and clear limits.

Context. The first agent reads the branch, the diff and the project notes, and returns the changed routines, the rules involved, the test scenarios and the data each scenario needs.

Real data, with no write access. The tool builds read-only query scripts, the developer runs them on the client’s database and hands back the result. No agent writes to the client’s database.

Script. The second agent writes the test script so that someone with no knowledge of the project can follow it, copying from the source code the exact messages and labels that will show up on screen. The script becomes a page the developer reviews before anything runs.

Execution. The third agent runs the script on the ERP’s web interface through a Playwright driver and records, in real time, the expected and actual result of each case.

Fixing. Failures go to the fourth agent, which classifies the cause as code, script, data or environment. When the defect is in the system, it fixes it in the development environment and logs the fix for the developer to ship. It never sends anything to the client and never commits.

Between stages there are checkpoints where a person approves the next step.

Outcome

The test script stopped being the developer’s private notes and became a document the key user can follow, with results recorded case by case. The tool grew out of Playwright scripts I was writing to test specific projects and became a shared tool for the team.

What this case shows

AI applied to the development process, with the limits designed before the capabilities: the agents read, propose and test; the one who approves and the one who ships to the client is a person.