This page explains how to automatically register the results of tests executed on MagicPod to the test management tool TestRail using GitHub Actions. This is based on the blog article(Japanese) by TechMatrix Corporation. The sample code is published in the GitHub repository.
Table of Contents
1. Overview
On this page, we will demonstrate how to register a total of six test results in TestRail using two
test cases created in MagicPod, executed across three browsers: Chrome, Edge, and Firefox.
We use GitHub Actions as the CI tool, to execute everything from test execution to result
registration in a single job.
Flow to register results
- Create a test plan for registering test results via TestRail's Web API.
- Run tests via MagicPod's Web API.
- Register each test result in the test run via TestRail's Web API.
GitHub Actions execution results screen
Test result registered in TestRail
MagicPod test results (Chrome)
2. Preparations
This section describes the preparations required for TestRail, MagicPod, and GitHub Actions.
2-1. TestRail
2-1-1. Add configurations to be used in the test plan
Add configurations for Chrome, Edge, and Firefox. Please refer to the TestRail manual for instructions on how to add them.
- TestRail configuration settings screen
2-1-2. Add a custom field
To link TestRail test cases with MagicPod test cases, add a custom field named "MagicPod_URL".
Label: MagicPod_URL System Name: magicpod_url Type: URL (link)
Please check the TestRail manual for details on how to add it.
TestRail test cases screen
2-1-3. Create a test suite and test cases
Create a test suite and test cases in TestRail. In this example, two test cases (C9 and C11) are created within a test suite (S3:TestRail).
TestRail test suite screen
For each test case, register the MagicPod test case URL into the "MagicPod_URL" custom field that you added in 2-1-2. This URL links the TestRail test case to the MagicPod test case, so please be sure to register it.
Test case sample URL: https://app.magicpod.com/<org_name>/<project_name>/4/
Please check the TestRail manual for details on how to create test suites and test cases.
2-2. MagicPod
2-2-1. Create test cases
Create test cases in MagicPod. In this example, two test cases(#4, #5) are created.
MagicPod project screen
2-2-2. Create batch run settings and patterns
In this example, three device patterns "Chrome", "Firefox", and "Edge" are created under the "1. Default" settings, with the corresponding browsers assigned to each pattern.
Please refer to this help page on how to create patterns.
2-3. GitHub Actions
2-3-1. Register secrets
Register the following items in your Repository secrets (secret variables created in the repository environment). Please check the GitHub Actions document for details on how to register.
| Secret Name | Description |
|
MAGICPOD_API_TOKEN |
MagicPod API token |
|
TESTRAIL_PASSWORD |
TestRail password |
GitHub repository secrets screen
3. Create scripts
First, download the script from the GitHub Repository.
From here, we will explain the steps according to the GitHub Actions job definition file, ".github/workflows/magicpod_test.yml" (hereinafter referred to as the “yml file”).
GitHub Actions workflow run screen
3-1. Install Python
(No action required) Since part of this script is written in Python, steps to install Python and HTTP library "requests" are defined.
## .github/workflows/magicpod_test.yml
- name: Set up Python
uses: actions/setup-python@v4
- name: Install python dependencies
run: |
python -m pip install --upgrade pip
pip install requests
3-2. Create a test plan in TestRail
To register test results, you must create a Test Plan and Test Run(s) in TestRail in advance.
The "testrail_prepare.py" file describes the process of creating a TestRail test plan using TestRail's Web API. For that reason, please define all necessary information for creating the test plan within "testrail_prepare.py". For details on what information can be registered via the Web API, please refer to this test plan document.
Tip: How to check "config_ids"
The "config_id" refers to the "Configuration" of the test plan, and in this example, it corresponds
to the ID of browser names, such as Chrome. You can check the "config_id" by using the get_configs method of TestRail's Web API.
Define TESTRAIL_URL and TESTRAIL_USER in the yml file.
## .github/workflows/magicpod_test.yml
- name: Prepare testplan
env:
TESTRAIL_URL: <YOUR URL>
TESTRAIL_USER: <YOUR USER ACCOUNT>
TESTRAIL_PASSWORD: ${{ secrets.TESTRAIL_PASSWORD }}
run: python testrail_prepare.py
3-3. Install magicpod-api-client
(No action required) The yml file is configured to run "download_magicpod_api_client.sh" as shown below. For more information about the magicpod-api-client, please consult this help page.
## .github/workflows/magicpod_test.yml
- name: Install magicpod-api-client
env:
MAGICPOD_API_TOKEN: ${{ secrets.MAGICPOD_API_TOKEN }}
run: bash download_magicpod_api_client.sh
3-4. Run tests in MagicPod
Define MAGICPOD_ORGANIZATION_NAME, MAGICPOD_PROJECT_NAME, and MAGICPOD_TEST_SETTING_ID in the yml file. Tests will then be executed based on these specified batch run settings. In this example, two test cases are executed in three patterns, yielding a total of six tests.
## .github/workflows/magicpod_test.yml
- name: Run MagicPod
env:
MAGICPOD_API_TOKEN: ${{ secrets.MAGICPOD_API_TOKEN }}
MAGICPOD_ORGANIZATION_NAME: <YOUR ORGANIZATION_NAME>
MAGICPOD_PROJECT_NAME: <YOUR PROJECT_NAME>
MAGICPOD_TEST_SETTING_ID: <YOUR SETTING_ID>
run: python run_magicpod.py
3-5. Register MagicPod test results in TestRail
Define TESTRAIL_URL and TESTRAIL_USER in the yml file.
- name: Add test results to TestRail
env:
TESTRAIL_URL: <YOUR URL>
TESTRAIL_USER: <YOUR USER NAME>
TESTRAIL_PASSWORD: ${{ secrets.TESTRAIL_PASSWORD }}
run: python testrail_add_test_result.py
4. Run a job in GitHub Actions
This script will automatically run MagicPod and add test results to TestRail each time code is pushed to GitHub. In addition to pushes, other events such as pull_requests and deployments can also trigger these actions. For more details, please check the GitHub Actions documentation.