How to maintain a test when it fails

This section describes how to maintain tests when they fail on a batch run and the cause of the failure is not known.

Re-execute the failed test cases

If a batch run is executed and some test cases fail in the batch run,

Execute only the test cases that failed in a single run or a batch run.

For detailed instructions on how to batch run specific test cases, please click here.

• In case of failure
  • The test case may have failed because it depends on another test case(that succeeded during batch run but was not executed this time). Execute the tests again considering the dependencies.
  • The test case may have failed due to changes in the product. Check the error message on the test result screen and fix it.
• In case of success
  • The test case may be unstable. Open the test edit screen, fix the test case to make it stable, and confirm by a single run.

Fix an unstable test case

• Put a wait time before the failed step. (until the UI element exists/equals, fixed seconds, etc.)
  • FYI: Wait command
• Change the locator specified for the UI element of the failed step.
  • Change from a locator that changes at each execution.
    • Based on Amount/DateTime

      • e.g.) xpath=//output[text()='7,000 yen']

    • Based on value, id, etc. that may change

      • e.g.) xpath=//input[@value='mp62130']

  • Change the UI element with coordinate specification operation.
    • It is a UI element that is displayed in a test run log as follows. "[WARNING] As operation on the specified element failed, will retry by specifying coordinates. If the target element is covered by another element, the element above will be clicked."

Tips: Recommended number of steps for easy maintenance

The recommended number of steps per test case is less than 200. (In browser testing, If the website under test has many UI elements, we recommend less than 300.)

Too many steps will make it difficult to isolate errors. It can also cause a problem with subsequent steps not being executed.