By using the Assert no visual diff command, you can check whether the display on the screen deviates from the expected image. This enables the detection of unexpected screen design issues during the test execution.
On this page, we will explain how to use the Assert no visual diff feature.
Table of contents
- Recommended policy for Visual diff checks
- Basic usage
- Add Visual diff patterns
- Skip Visual diff check
- How to check vertically long pages
- Points to note when running tests from the local PC command line
- Troubleshooting
- Limitations
- Known issue
Recommended policy for Visual diff checks
The primary purpose of Visual diff checks is to detect major layout breakages across the screen. From the perspectives of execution time and maintenance overhead, we do not recommend inserting a large number of Visual diff checks.
For example, if there are 100 Visual diff check steps within a single test case, it becomes difficult to verify each difference, which often leads to neglected maintenance. In such cases, there is a tendency to approve test results without reviewing them, ultimately rendering the test meaningless.
For this reason, we recommend limiting Visual diff checks to approximately one per screen.
For asserting individual elements, please use assertion commands such as Assert UI element equals as much as possible, and only rely on Visual diff checks when such assertion commands are not feasible.
Basic usage
1. Set up expected images
First, add the Assert no visual diff command.
Ensure that the element selection mode is deactivated.
Run the test. *Do not click Edit for the command yet.
This will generate the expected image for confirmation, and the test run result will show as Unresolved. Click Details.
Once you are on the test result screen, click the message.
This will display a dialog, where you can check the content of the image and press Approve.
When approved, this image will be saved as the expected image and will be used for subsequent difference checks.
1-1. Set a diff threshold and exclusion areas
Then, return to the test editing screen and click the Edit button for the command.
In the dialog, you can use drag and drop to specify a diff threshold (the allowable mismatch rate for triggering an error) and specify areas to exclude from the diff check.
2. Run a test
Once the expected image is generated, the visual diff check will be executed in subsequent test runs.
If there is a difference, a test result status will show as Unresolved.
Clicking the error message will display a dialog, allowing you to view the details of the difference.
If there are several unresolved visual diff checks, it is recommended to check all of them from the Visual diffs tab.
In the dialog, checking the Show diff image checkbox highlights the areas where differences occurred in red.
If there are no issues with the difference, you can click Approve to use the new image as the expected image for subsequent executions. If differences are occurring as a result of issues with the site or app being tested, click Mark as an error to treat this test result as an error. Run the test again after fixing the problem with the site or app.
Add Visual diff patterns
When you want to run the same test in various environments, the expected images for the visual diff check will differ depending on the type of browser or device. In such cases, you want to prepare the correct image for each environment by adding a visual diff pattern from the test run settings dialog. You can then switch patterns for each environment and execute the test.
You can specify which pattern to use when running each test from the test run settings dialog. (The edit dialog for the expected image also has a visual diff pattern selection box, but this is used when switching the image displayed in the dialog box, rather than specifying the pattern to be used during test execution. )
Skip Visual diff check
For example, when you want to run tests that are usually executed in a CI environment on your local PC for debugging purposes, differences in the environment will almost certainly result in unexpected visual differences. In such cases, you can specify Skip for the Visual diff pattern in the test run settings dialog to avoid executing the visual diff check.
How to check vertically long pages
Currently, it is not possible to check vertically long pages with a single visual diff check. (Feature request is here)
As an alternative, you can test a vertically long page by repeating the following steps:
"Assert no visual diff" -> "Scroll (Swipe) until the specified element at the bottom of the page is visible" -> "Assert no visual diff".
However, with the “Scroll (Swipe) until visible” command, slight variations in the scroll (swipe) amount may occur each time it is executed. This can sometimes cause failures in the image comparison step.
If you encounter this issue, you can achieve more stable and consistent scrolling (swiping) by using the “Keyboard Input” command to send the “PageDown” key instead of using “Scroll (Swipe) until visible.”
When using the “Keyboard Input” command to send the “PageDown” key, you need to ensure that the target page is focused beforehand. To do this, add a step at the beginning to click on any blank area within the page.
Points to note when executing from the local PC command line
To run the command line on your local PC using MagicPodDesktop, you need to add a visualDiffPattern entry to your settings file. See here for details.
Troubleshooting
I want to understand why a difference was detected, even though there appears to be no visible change
Please check the area around where the difference is detected to see if there is any variation between the expected and actual images.
For example, suppose you have the following labels on the screen:
In the expected image on the left side, the label next to "MagicPod" is "3:15", but the actual image on the right side is "10:15".
In this case, the position of the "MagicPod" label shifts slightly to the right by the width of one digit, and this causes a difference to be detected.
This is a common situation where an adjacent element, such as a date or a monetary value, changes dynamically, causing slight positional shifts (a few pixels) in neighboring elements, which are then detected as differences.
Limitations
Usage with data patterns, loop commands, and shared steps
You can use the "Assert no visual diff" command in test cases with data patterns configured, as well as inside loop commands and shared steps, but you cannot set expected images for each iteration.
This enhancement request has been formally documented and prioritized within the MagicPod Product Portal. Should you require this feature enhancement, we encourage you to cast your vote on the corresponding feature request card to help us prioritize development efforts based on customer demand.
Check visual diff for each data pattern
Pass correct images from visual diff checks to the shared step
Known issue
Differences between similar colors may not be detected
Assert no visual diff command performs pixel-level image comparisons using an underlying image comparison library. We have confirmed an issue where differences may not be detected when the color variation between images is very small.
We are currently investigating this issue and evaluating possible improvements.
If you have encountered this issue, please contact MagicPod Support.