Skip to main content

AI Assertion

  • Updated

AI Assertion is a feature that allows you to simply describe "what results you expect" in natural language within your test case, and generative AI will understand the screen or text content and automatically perform the necessary checks.

*Note: For old Standard Plan subscribers, this feature is subject to AI credits. AI credits are required to use this feature.

Table of contents

Use cases

  • Checks that require understanding the meaning of data, such as "consistency between search conditions and results"
  • Checks for list screens and feed screens where content is complex and changes dynamically, making it traditionally difficult to create stable tests
  • Non-mechanical checks that require human judgment, such as "checking for design inconsistencies"

How to use

Prerequisites

To use AI Assertion, the organization owner account must enable the "Generative AI" feature. (See here for configuration instructions)

 

Usage Allowance

If you have subscribed to one of the eligible plans listed below, you can use this feature without requiring AI credits.

Eligible Plans

  • Standard Plan (2025) *Old Standard Plan is not eligible
  • Enterprise Plan

AI Assertion Usage Allowance: 3,000 times/month

  • This allowance is sufficient for typical usage without needing to worry about remaining quota
  • Usage is counted separately for each contracted platform (Browser Plan/Mobile Plan)
  • The usage count resets on the 1st of every month at 10:00 AM JST (1:00 AM UTC).

If you have subscribed to the old Standard Plan, AI credits are still required to use AI Assertions.

Please refer to the page below for information on how to migrate your plan.

Differences Between the New and Old Standard Plans

 

Steps

1. Add the Assert with AI command.

2. Describe what you want to check in English or Japanese.

You can also use variables like this:

How to check logs

You can view the AI's thought process in the logs when it makes pass/fail judgments.

For failures, you can check from the error message and execution log on the test results screen:

For successes, you can check from the log on the right side of the Execution Log tab on the test results screen:

 

Tips

  • Describe what you want to verify clearly and specifically
    • Examples:
      • Are detail buttons displayed in the list table? → Detail buttons should be displayed at the right end of all data rows in the list table
      • Is text displayed in a readable size? → Text A should be displayed in a larger size than Text B
  • Verify that it fails as expected in scenarios where it should fail
    • Examples:
      • Button color is blue -> Does it fail when the color is not blue?
      • Only home appliance products are displayed -> Does it fail when non-home appliance items are displayed?

Use case examples

Here are specific use case examples. 

Accuracy improves when you describe the following content more specifically. Please refer to the Tips for details.

  • When the "Home Appliances" category is selected, are only home appliance products displayed?
  • Search for "In Stock" → Are sold-out products not appearing in the results?
  • When filtered by "October 2025", are all the results October data?
  • Are sorts like "Most Popular" and "Highest Price" functioning correctly?
  • Are the results filtered by "¥10,000~¥50,000" within range?
  • Is the latest post displayed at the top?
  • Does each post have a like button and comment button?
  • Are posts not displayed in duplicate?
  • Is the graph drawn on the Canvas element displayed?
  • Does the store address match the icon display position on the map?
  • Is video playback occurring/Has playback completed?
  • Is the button color red?
  • Is the photo displayed in sepia tone?
  • Does the product image fit within the frame?
  • Is text displayed in a readable size?
  • Are there no contrast issues between the background and text color in dark mode?
  • Is the text written in Japanese?

Information passed to LLM(Generative AI)

  • Prompts entered by users（Text of what you want to check）
  • Test steps
    • Information on the steps executed before the target Assert with AI step
  • UI image
    • Only the UI displayed at the time the Assert with AI step is executed

FAQ

Q. How should I decide between AI Assertion and existing assertion commands (including the visual diff check)?

A. Use AI Assertion when assertion is difficult with existing assertion commands.
If you can test without issues using existing commands, there's no need to replace them with AI Assertion.

 

Q. The pass/fail judgment is inconsistent. What kind of prompt works best?

A. Start by checking the "AI judgment details" in the execution log for any failed runs.

If the judgment details differ with each failure, the prompt itself may be too vague. Try rephrasing it with more specific and precise language. You might also find it helpful to ask an AI assistant you have on hand to describe the target image — this can give you useful wording to work with.

If the failure reason is the same every time, the most effective approach is to explicitly add an exclusion condition to your prompt to rule out that false-positive pattern.

Example:

Confirm that a 👆 (pointing finger icon) is displayed on the screen along with the date {$today}. 
However, do not judge the following as a failure: 
・When the gesture appears to be a thumbs-up 👍

 

Q. Can I use AI Assertions beyond the monthly limit of 3,000 times?

You can configure in advance whether to use AI Credits to continue using AI Assertions or to skip AI Assertion steps when the monthly usage limit of 3,000 AI Assertions has been reached.

This setting can be configured by logging in with an organization administrator account and navigating to the Settings tab > Optional Features page for your organization.

If "Don't use AI credits" is selected, or if "Continue with AI credits" is selected but there are insufficient AI credits, the AI Assertion steps will be skipped. This will not affect the overall execution of the test case; however, the test result itself will be marked as "Failed". (Please note that this specification is subject to change.)

In addition, if you are on the legacy Standard Plan and AI credits are insufficient at the time of test execution, AI assertion steps will likewise be skipped.

 

Q. Which failure behavior will be applied: "Stop on failure" or "Mark as failure and continue"?

It will be "Mark as failure and continue." Currently, the option to stop a test at the point of failure is not provided.

Was this article helpful?
0 out of 0 found this helpful
If the problem persists, please contact us from here.
Contact
Return to top

Related articles