What a skeptical developer should ask an AI testing tool

A passing demo is the easy part of evaluating an AI testing tool. The useful questions start when the application changes. If I were buying one, these are the questions I'd ask, including of us.

1. What happens when the product really breaks? Break a flow on purpose with the expected outcome unchanged. The test should fail, and you should be able to tell why from the result. A tool that adapts around a real regression is worse than no tool.

2. What happens when the UI changes harmlessly? Move a button and change the layout, keeping the requirement the same. Does the test still pass? Can you see what it did differently?

3. Could it have accepted the wrong thing? Two buttons can share a label, and a replaced component can mean something different. Ask how the tool decides that a new target is the same control, and whether you can inspect that decision.

4. Can you read what ran? You should be able to see the steps, what happened in each and what the test checked at the end. "The agent verified it" isn't an answer.

5. Who can change the expected outcome? The answer you want is that a person does, deliberately. An agent shouldn't weaken an assertion to get a pass.

6. What leaves your environment? Find out what page data goes to the vendor or a model provider, and whether that still happens when the browser runs on your machine.

Those questions tell you more than any claim about how intelligent an agent is. They show whether your team can understand and maintain the resulting suite.

Here's how Zerocheck answers the third one, since it's the one vendors tend to wave away. Healing history records the previous target, the replacement and why. A moved control can be found again; a renamed one isn't treated as the same target, and a failed outcome stays failed.

None of this helps if the expectation is vague. "The page looks right" is much harder to review than a specific visible value or URL, whichever tool you use.

For a way to run this evaluation on one of your own flows, see how to evaluate AI test automation.