A demo shows a test passing on the day it was written. A better evaluation uses one real flow from your own app, and looks at what happens after that day.
Choose the flow first
Pick a real checklist item with fixed test data, and write down the expected outcome before you open the tool. Otherwise it's easy to accept whatever the tool produced as what you wanted.
Run it three ways
| Run | What you do | What you want to see |
|---|---|---|
| Normal pass | Run the flow as it is | Steps you can read, and evidence of what happened |
| Harmless change | Move a button or change the layout, and keep the requirement the same | The test still passes, and you can see what changed |
| Real break | Break the flow on purpose, with the expected outcome unchanged | The test fails, and you can tell why from the result |
The third run matters most. An agent can produce plausible steps without the right assertion, and a pass can hide an incomplete expectation.
What to look at
- Could someone else understand the failure? Show the result to an engineer who didn't write the test.
- Did the expected outcome stay fixed? A failure should never be fixed by weakening the assertion.
- What does it take to maintain? Compare the effort after a real UI change, not just how fast the first test was generated.
- Where does your data go? Include the service the tool depends on at runtime, and what it sends there.
Use your own results rather than a vendor benchmark.
Trying this with Zerocheck
Describe the flow and expected outcome in the web app, and Zerocheck runs it in a hosted browser. For the failing run, open the failed step, its screenshots and the error. For the harmless change, the run's healing history shows the old and new target.
It doesn't crawl your app to find tests. It starts from what you describe, and your team supplies the app URL, test accounts and expected behavior.