If we use “Assert no visual diff” feature, we can check whether the image display differs from the expected value. Based on this, it will be possible to detect unexpected screen design corruption, etc., during test execution.
On this page, we will explain how to use "Assert no visual diff" function.
Table of contents
- Basic usage
- Adding “Visual diff patterns”
- Skipping Visual diff check
- Points to note when running from the local PC command line
Basic usage
1. Setting up expected images
First, add “Assert no visual diff” command.
Run the test. *Do not press "Edit" button of the command yet.
Once done, an expected image will be generated and a test result will be “Unresolved.”
So click "Detail."
When you are transferred to the test results screen, click the message.
A dialogue will then appear. If there are no problems with the image, press "Approve".
After approval, this image will be saved as an expected image and used for the next differential check.
1-1. Setting a diff threshold and exclusion areas
After that, return to the test edit screen and press "Edit" button of the command.
A dialogue appears. Then, you can specify a diff threshold, which specifies what percentage or more of the difference is an error, or exclusion areas that are not checked, by dragging.
2. Running a test
Once an expected image is generated, Visual diff checks will be performed on subsequent test executions.
If there is a difference, a test result will be "Unresolved."
Press the error message and a dialogue will appear. So check the content of the difference.
If checks are required for several visual diff check commands, it is recommended to check all steps from "Visual diffs" tab.
On this check screen, if you check “Show diff image,” you can display the area(s) where differences have occurred in red.
If there is no problem with the content of the differences, click “Approve” and use this image as the next expected image. If this difference is caused by a problem with the site or application being tested, treat this test result as an error by pressing "Mark as error" and run the test again after fixing the problem with the site or application.
Adding “Visual diff patterns”
In order to run the same test in various environments, you will want to prepare the correct image for each environment. This is the case even for the same step, because the expected value image for the Visual diff check will vary depending on the type of browser or device. In such cases, please add " Visual diff patterns" for each environment from the Test Target Specification Panel, and run the test after switching the pattern for each environment.
You can specify which pattern to use when running each test from the Test Target Specification Panel. (The Edit dialog for the expected value image also has a " Visual diff pattern" selection box, but this is used when switching the image displayed in the dialog box, rather than specifying the pattern to be used during test execution. )
Skipping Visual diff check
For example, for tests run in a normal CI environment run on your own local PC for debugging purposes, there will almost certainly be unexpected image differences due to environmental differences. In such a case, specify “Skip” for the “ Visual diff pattern” on the Test Target Specification Panel to omit execution of the Visual diff check.
Points to note when running from the local PC command line
To run the command line on your local PC using MagicPodDesktop, you need to add a visualDiffPattern entry to your settings file. See here for details.