Why AI test generation should ask, not guess

Ask a large language model for test cases and it will happily give you twenty. They'll look professional: numbered steps, expected results, priorities. And some of them will be quietly made up.

The model doesn't know the exact error message your app shows for an expired coupon, so it writes a reasonable-sounding one. It doesn't know how many failed logins lock an account, so it picks a number. It wasn't told what happens when two discounts collide, so it decides. Each guess turns into a test case that someone will run, fail, and then spend time investigating, only to find the test was wrong.

We think that's backwards. A wrong test is worse than no test, because people trust it.

The problem with "always generate more"

Most generation prompts optimise for volume and coverage: "generate at least N test cases", "cover security, concurrency, edge cases and boundaries". That sounds rigorous. But when the requirement says nothing about concurrency, the only way to "cover" it is to invent the expected behaviour.

The result is a test suite that looks comprehensive and is partly fiction. Reviewers have to read every case to find the invented parts, which defeats the purpose of generating them.

What we do instead

When sqale generates test cases from a requirement, it follows one rule: write tests only for what the source actually says, and ask about everything else.

In practice that means:

1. Every test case cites its source. Each generated case carries the exact sentence from the requirement it verifies. If the model can't point to a sentence, it shouldn't be writing the test.
2. No invented details. Error messages, button labels, field names, limits, URLs and business rules must come from the source. For arbitrary test data, the model uses an obvious placeholder like <a registered user's email> instead of a realistic-looking fake.
3. Missing information becomes a question. If a test would need a detail the requirement doesn't give, the model doesn't write it. It asks, for example: "What message is shown when the coupon has expired?"
4. No quotas. There's no minimum number of tests. Five correct tests beat twenty plausible ones.

Don't trust the model to follow the rules

Prompt instructions are a request, not a guarantee. So the rules are also checked in code, on the server, before anything is saved:

• The case's source quote must actually appear in the requirement (or its linked documents, the product context, or your clarifications). A fabricated citation is rejected.
• Every exact value the case quotes, such as an error message or a label, must appear in the source too. A test expecting the message "Invalid coupon code" is held back if that text appears nowhere in the requirement.
• A case that hedges ("needs clarification", "to be confirmed") is held back rather than saved half-finished.

One subtle point: the AI's own earlier analysis of the requirement (the risks and edge cases it suggested) is used as a hint, but it doesn't count as a source. Otherwise the model could ground a test in something it invented a step earlier.

Every case that fails a check is turned into a question for you.

Asking you, then trying again

After generation you see the tests that were written, plus a short list of questions, for example:

3 questions before more tests can be written.

1. What is the exact message shown for an invalid coupon code?
2. Can a coupon be used together with a gift card on the same order?
3. Is the coupon field case-sensitive?

Answer what you can. Your answers are saved to the requirement as a Clarifications section, so they become part of the source, and the next generation can write the tests that depend on them. If a requirement is too vague to test at all, you get only questions, which is itself a useful signal to send back to the product owner.

What this costs, and why it's worth it

You'll get fewer generated tests than you might from a tool that never says no. That's the point. The tests you do get are ones you can trust, and the questions surface gaps in the requirement before development, when they're cheapest to fix.

Generated cases are always saved as drafts, so a person reviews them before they're used in a run. AI should speed up a QA engineer's judgement, not replace it with confident guesses.

If you want to try it, start a free trial or read how requirements and AI generation work in sqale.