This page describes the failure analysis feature of MagicPod Autopilot.
*Note: This feature is subject to AI credits. AI credits are required to use this feature.
Table of Contents
Overview
The failure analysis feature enables MagicPod Autopilot to automatically investigate and analyze the root cause when a test fails.
It analyzes data such as screenshots, error logs, and test step details from failed tests, and provides insights into why the failure occurred and how to resolve it.
How to use
Prerequisites
To use the failure analysis feature, the "Generative AI" feature must be enabled by an organization administrator account. （Refs: How to Enable Generative AI Features）
Using the failure analysis feature with a batch run
-
Click the "Details" button on the "Batch runs" tab of your project.
-
Select the batch run settings where you want to enable the failure analysis feature, then under "Auto-maintenance of tests" on the "Common Settings" screen, select "Auto-analyze on failure".
You can also set a maximum number of AI credits the failure analysis feature may consume per batch run.
- Execute the batch run.
-
If any tests fail, the failure analysis results will be displayed on the test result details screen.
By clicking the "Show analysis process" button, you can review the analysis process and the number of AI credits consumed during the analysis.
Using the failure analysis feature with single test run
You can run a failure analysis from the details screen of a failed test result.
Failure analysis is also available for batch runs performed without the failure analysis feature enabled, as well as for single test runs and partial test runs.
Using the failure analysis feature with MagicPod Web API
The failure analysis feature is also available when running batch executions via the MagicPod Web API.
For details on the API, please refer to the following:
Auto-fix tests based on MagicPod Autopilot suggestions
Using the failure analysis results as input to MagicPod Autopilot allows suggested fixes to be applied automatically.
Click the "Fix test based on suggestions" button on the failure analysis results screen.
※ If MagicPod Autopilot determines from the failure analysis that the issue cannot be resolved by modifying the test case, the "Fix test based on suggestions" button may not appear.
The screen switches to the test case editing view, and MagicPod Autopilot launches. The suggestions from the failure analysis are automatically entered into the prompt, and the test case is updated accordingly.
Auto-filled prompt
Information passed to the LLM (Generative AI)
- Test steps
- UI images and UI tree information
- Shared steps
- Test run logs
- Includes past test run logs for the target test case
- Values of secret shared variables are masked
- Screen captures taken during test run
Notes
- Results from AI assertion commands are not subject to failure analysis. Please note that only failures occurring during the execution of test steps are analyzed — not all failures will be analyzed.
Q&A
Q. How many AI credits are consumed?
On average, 0.2 to 0.3 AI credits are consumed per failure analysis.
Q. Can the failure analysis feature be used when running tests from the test case editing screen?
The failure analysis feature can also be used on results from single test case runs and partial runs.
For details, please refer to the following:
Using the failure analysis feature with single test run
Q. Can the same test result be analyzed more than once?
You can re-analyze a test result that has already been analyzed.
Each analysis result can be viewed by using the arrow controls at the bottom of the analysis process details screen.
Q. Can analysis results be deleted?
You can delete an analysis result from the three-dot menu on the right side of the failure analysis results screen.
If there are multiple analysis results, use the arrow controls at the bottom of the analysis process details screen to select the result you wish to delete, then click the trash icon to delete it.