Integrate with Auto test stability report of QualityForward

This article explains how to register test results executed in MagicPod to the Auto test stability report in QualityForward, a test management tool.

This page is based on an article created by Veriserve Corporation(available in Japanese only).

Overview

QualityForward's Auto test stability report is a tool for analyzing and visualizing automated test results and execution times in chronological order. By integrating your MagicPod test results, you can view test success rates in graphs and identify failure-prone test cases for each batch run configuration.

QualityForward's Auto test stability report

Detailed view of batch runs in the Auto test stability report

This section provides an example of using GitHub Actions to integrate batch run results with the Auto test stability report, running everything from test execution to result registration in QualityForward as a single GitHub Actions job.

Prerequisites

QualityForward

Set up a project to associate with the Auto test stability report.

For detailed instructions, refer to this page(available in Japanese only).

MagicPod

Prepare the test cases and batch run configuration to be executed.

In this example, we created four test cases.

You can configure the batch run settings from the project page under the Batch runs tab > Details.

Add or modify settings as needed.

Note that the batch run setting name shown in the figure below will be registered as the "Auto test suite name" in QualityForward.

GitHub

Register the following items in your Repository secrets (secret variables created in the repository environment). 

Please check the GitHub Actions document for details on how to register.

GitHub Actions

Prepare the GitHub Actions job definition file at .github/workflows/magicpod_test.yml. Below is an example of the YAML file configuration.

on: [push]

jobs:
  magic_pod_job:
    runs-on: macos-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      # MagicPod batch run
      - name: Batch run test
        env:
          MAGICPOD_API_TOKEN: ${{ secrets.MAGICPOD_API_TOKEN }}
        run: |
          OS=mac
          FILENAME=magicpod-api-client

          # Download and unzip the latest version of magicpod-api-client to the current directory
          curl -L "https://app.magicpod.com/api/v1.0/magicpod-clients/api/${OS}/latest/" -H "Authorization: Token ${MAGICPOD_API_TOKEN}" --output ${FILENAME}.zip
          unzip -q ${FILENAME}.zip

          # Set the various environment variables used on MagicPod
          MAGICPOD_ORGANIZATION=<MagicPod organization name>
          MAGICPOD_PROJECT=<MagicPod project name>

          # Execute batch run
          TEST_SETTING_NUMBER=<The test setting number of MagicPod>
          ./magicpod-api-client batch-run -t ${MAGICPOD_API_TOKEN} -o ${MAGICPOD_ORGANIZATION} -p ${MAGICPOD_PROJECT} -S ${TEST_SETTING_NUMBER}

          # Get the batch run number from a batch run
          BATCH_RUN_NUMBER=$(./magicpod-api-client latest-batch-run-no -t ${MAGICPOD_API_TOKEN} -o ${MAGICPOD_ORGANIZATION} -p ${MAGICPOD_PROJECT})

          # Export batch run results to JSON
          ./magicpod-api-client get-batch-run -t ${MAGICPOD_API_TOKEN} -o ${MAGICPOD_ORGANIZATION} -p ${MAGICPOD_PROJECT} -b ${BATCH_RUN_NUMBER} > batch_run_results.json

      # Integrate test results using actions
      - name: Integrate with QualityForward
        uses: QualityForward/test-stability-report-sync@v1.2.0
        with:
          file-path: batch_run_results.json
          test-framework: magicpod
          api-key: ${{ secrets.QUALITYFORWARD_API_KEY }}

Run a job in GitHub Actions

This script will automatically run MagicPod and add test results to QualityForward each time code is pushed to GitHub. In addition to pushes, other events such as pull_requests and deployments can also trigger these actions. For more details, please check the GitHub Actions documentation. 

Note that if you have added multiple browser/device patterns to your MagicPod test run, only the results from the first pattern will be sent through the GitHub Actions workflow described above. Adjust the script as needed.