Run batch tests from the command line (Local PC environment)

On this page, we shall explain how to run batch tests on the local PC in MagicPod from the command line.

In a local PC environment, batch test execution uses MagicPodDesktop. For installation instructions, see Before performing a browser test on local PC.

Table of contents

1. Create a configuration file for batch runs
2. Run batch tests on Mac
3. Run batch tests on Windows
4. Run batch tests on Linux
5. Run batch tests in parallel

1. Create a configuration file for batch runs

To execute batch runs from the command line, you need to prepare a configuration file in JSON format. Although the file name can be arbitrary, the file automatically generated by MagicPod is named magic_pod_config.json. There are two configuration styles:

• Using a test settings number on the web page
• Specifying test settings directly

The former is the recommended approach because it offers the advantage of creating or modifying settings intuitively on the web page. On the other hand, some settings are supported only with the latter. We describe both methods on this page.

Using a test settings number on the web page

By writing magic_pod_config.json in the following format, it is possible to reference the batch run settings already created on the Batch runs page. Once you launch MagicPodDesktop, the magic_pod_config.json file should be found on the PC. For Windows, it's placed in C:\Users\<Username>\AppData\Roaming\MagicPodDesktop\magic_pod_config.json(for MagicPodDesktop version 1.50.0 or earlier, C:\Users\<Username>\AppData\Roaming\magic_pod_desktop\magic_pod_config.json). For Mac, it's located in /Users/<Username>/Library/Application Support/MagicPodDesktop/magic_pod_config.json(for MagicPodDesktop version 1.50.0 or earlier, /Users/<Username>/Library/Application Support/magic_pod_desktop/magic_pod_config.json). Copy the file to a suitable location and use it.

{
  "owner": <Organization name. This is not the display name, but the alphabet description in the URL>,
  "project": <Project name. This is not the display name, but the alphabet description in the URL>,
  "branch": <Branch name. If omitted, this is the default branch>,
  "testSettingsNumber": <Test Settings Number>,
  "testSettingsPatternName": <Test Settings Pattern Name. Required when there are multiple patterns>,
  "language": <display language for the test execution log. ja or en. If omitted, this is en>
}

For items that can be configured on the web page, the settings on the page are prioritized in principle. If you configure individual items with the measure described in the next section, they will not be applied during test execution. The only exception is "Test result language", which can be overwritten by "language" in the configuration file.

Specifying test settings directly

When you specify test settings directly, the contents are as follows. Please note that you should not specify testSettingsNumber in this case.

{
  "owner": <Organization name. This is not the display name, but the alphabet description in the URL>,
  "project": <Project name. This is not the display name, but the alphabet description in the URL>,
  "branch": <Branch name. If omitted, this is the default branch>,
  "capabilities": <Target OS and target app settings. Same as Appium Capabilities>,
  "sendMail": <whether to distribute a test results e-mail. If omitted, this is false>,
  "captureType": <test results screen capture setup. Any one of on_error, on_ui_transit, or on_each_step If omitted, this is on_each_step>,
  "xmlTestOutput": <output the test results XML in JUnit format. Mainly in coordination with Jenkins. If omitted, this is false>,
  "workDir": <full path to the work directory. If omitted, this is “magicPod” on the Desktop>,
  "authTokenFilePath": <full path to authentication token file. When omitted, set to ~/.magic_pod_token>
  "envVars": <list of environmental variable strings in KEY=VALUE format. If omitted, this is a blank list>,
  "testNumber": <describes the test number to be run delimited by commas. When blank or omitted, run all tests. If you want to run a specific test multiple times, you can use something like "1*100" (in this example, test number 1 is run 100 times)>,
  "includedTestCaseLabels": <specifies the label for the test in character string list format>,
  "excludedTestCaseLabels": <specifies the label for the test in character string list format>,
  "logLevel": <sets the volume of log to output. Any one of beginner, expert, or magic_pod_bug_investigation. If omitted, this is beginner>,
  "retryCount": <how many times to retry failed tests. If omitted, this is 0 (in other words, do not retry)>,
  "externalAppium": <Whether to use the MagicPodDesktop external Appium Server. If omitted, this is false>,
  "externalAppiumUrl": <Specifies the URL of the Appium Server to connect to if externalAppium is true. >,
  "language": <display language for the test execution log. ja or en. If omitted, this is en>,
  "visualDiffPattern": <Image difference pattern name. It is possible to specify skip. When omitted, this is handled as unspecified>
}

Refer to here when specifying shared variables.

Refer to here when using variables for values in the configuration file.

Based on the above, the actual configuration files will become as follows. As we mentioned above, using a testSettingsNumber is recommended for normal test execution. Since some settings we describe later don't support configuration by testSettingsNumber, direct configuration is required.

• Run batch tests in parallel
• Save the test results in a video for Linux environments

Example 1: iOS simulator

{
  "owner": "SampleCompany",
  "project": "SampleApp",
  "capabilities": {
    "platformName": "iOS",
    "platformVersion": "17.2",
    "deviceName": "iPhone 15",
    "app": "/Users/<yourUserName>/magicpod_demo_app.app",
    "automationName": "XCUITest"
  },
  "sendMail": true,
  "workDir": "/Users/<yourUserName>/magicpod-work"
}

If you are using the iOS 18.3.1 simulator, set "platformVersion": "18.3".

If you set 18.3.1, your test will fail with an error message such as: '18.3.1' does not exist in the list of simctl SDKs. Only the following Simulator SDK versions are available on your system: 18.2, 18.3, 17.5

Example 2: iOS real device

{
  "owner": "SampleCompany",
  "project": "SampleApp",
  "capabilities": {
    "platformName": "iOS",
    "platformVersion": "17.2",
    "xcodeSigningId": "SampleId",
    "xcodeOrgId": "SampleOrgId",
    "deviceName": "iPhone 15",
    "app": "/Users/<yourUserName>/magicpod_demo_app.app",
    "automationName": "XCUITest"
  },
  "sendMail": true,
  "workDir": "/Users/<yourUserName>/magicpod-work"
}

Example 3: Browser (Chrome)

{
  "owner": "MyCompany",
  "project": "SampleWebApp",
  "capabilities": {
    "browserName": "chrome"
  },
  "sendMail": true,
  "testCondition": {
    "browser": "chrome"
  }
}

• Similar content must be written for capabilities and testCondition, but we plan to improve this in the future. The test may be run even if there is no testCondition, but browser information may be missing when viewing the test results on the Web.

2. Run batch tests on Mac

Once the configuration file is complete, run the test according to the following procedure.

1. Close MagicPodDesktop if it has been launched.
2. On the Mac Terminal, navigate to the MagicPodDesktop directory.

3. Run the following command.

   # Update MagicPodDesktop to the latest version if available
   "MagicPodDesktop.app/Contents/MacOS/MagicPodDesktop" update --magic_pod_config="<Full path to magic_pod_config.json>"
   # Test batch execution
   "MagicPodDesktop.app/Contents/MacOS/MagicPodDesktop" run --magic_pod_config="<Full path to magic_pod_config.json>"

4. Run the test in accordance with the magic_pod_config.json configuration. If the test is successful, the value returned by the process will be 0, whereas if it fails, this will be 1. If it ends abnormally, this will be -1.

If you copy magic_pod_config.json, command line execution is possible with other Mac PCs in the same way.

However, since authentication information is not automatically saved in magic_pod_config.json, you need to log in to MagicPod once and press the "Connect" button on the test case edit page to save the authentication information on your PC.

3. Run batch tests on Windows

Test batch execution within the project can also be performed from Windows PowerShell.

Once the configuration file is complete, run the test according to the following procedure.

1. Close MagicPodDesktop if it has been launched.
2. Open Windows PowerShell. There are several ways of doing this, but the simplest way is to open Windows PowerShell from the start menu.

3. Run the following command. For the MagicPodDesktop version section, enter (0.36.0, etc.) according to the folder in which it actually exists. This can be completed on PowerShell. Note that an error will occur if there is no & at the start (when entering from 1 on PowerShell, this is inserted automatically)

   & "C:\Users\<Username>\AppData\Local\magic_pod_desktop\app-<MagicPodDesktop version>\MagicPodDesktop.exe" run --magic_pod_config="<magic_pod_config.json full path>"; Wait-Process -Name MagicPodDesktop -Timeout 3600

   The 3600 in the command is the maximum number of seconds to wait for the test to complete, and means to wait up to 3600 seconds. This wait time may be changed based on the length of the test.

4. Run the test in accordance with the magic_pod_config.json configuration.

If you copy magic_pod_config.json, it is possible to run the command line in the same way on other Windows PCs.

However, since authentication information is not automatically saved in magic_pod_config.json, you need to log in to MagicPod once and press the "Connect" button on the test case edit page to save the authentication information on your PC.

4. Run batch tests on Linux

It is possible to run a test batch on Linux, using a procedure similar to the Windows one.

Install the required libraries

First, install xvfb (required libraries for browser rendering) and imagemagick (required libraries to acquire screenshots for the entire browser window). For Ubuntu, you can install these using the following commands.

sudo apt-get update
sudo apt-get install -y unzip xvfb libxi6 libgconf-2-4 imagemagick

In certain CI services, such as CircleCI, xvfb and imagemagick are installed from the start, so this procedure is not necessary.

Next, if we are using Chrome to run the test, we must first install Chrome. For Ubuntu, you can install these using the following commands.

wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
sudo sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list'
sudo apt-get update
sudo apt-get install google-chrome-stable

Next, we need to install Japanese language fonts so that there will be no garbled characters in Japanese language screen captures. In the case of Ubuntu, you can install these using the following commands.

sudo apt -qqy --no-install-recommends install -y locales-all
sudo locale-gen ja_JP.UTF-8
sudo update-locale LANG=ja_JP.UTF-8
sudo update-locale LANGUAGE="ja_JP:ja"
sudo apt -qqy --no-install-recommends install -y fonts-takao-gothic fonts-takao-mincho
sudo dpkg-reconfigure --frontend noninteractive locales
sudo fc-cache -fv

Download magicpod-command-line

Execute the following command on the command line to download.

curl -L "https://app.magicpod.com/api/v1.0/magicpod-clients/local/linux/latest/" -H "Authorization: Token {apikey}" --output {filename}.zip

• {apikey}: API token
• {filename}: Any file name you like

You can also acquire the zip file for magicpod-command-line for Linux from the Download page unzip it, and place the resulting magicpod-command-line and node_modules directories in an appropriate location (both files need to be in the same place).

Write a configuration file

Create magic_pod_config.json in the same way as in Windows (There is no original file to copy, so you need to create it using a text editor, etc.).

Set authentication information

If you are using a PC on which you have successfully logged into MagicPodDesktop, below the Mac folder /Users/<Username> or for Windows, in the C:\Users\<Username> folder, there should be a file called .magic_pod_token. This file is the authentication information file for the login user, so copy this below “~/” on a Linux machine.

Alternatively, you can set the authentication information by running the following command and acquiring the token information called {"token":"****"}, and saving the contents of the **** part in ~/.magic_pod_token (only for users not configured for SAML authentication).

#Specify the e-mail address used for MagicPod in xxxx and the password in yyyy.
curl -X POST -L --data-urlencode "email=xxxx" --data-urlencode "password=yyyy" https://app.magicpod.com/api/v1.0/internal/token-auth/

The path for ~/.magic_pod_token can be changed using authTokenFilePath in magic_pod_config.json.

Launch Xvfb

You need to launch Xvfb before running a test, and close it when it is finished. As this process is carried out automatically by some Cloud CI services, such as CircleCI, this is not required. If you are using Jenkins, the Xvfb plug-in will automatically launch Xvfb when running a job. In other cases, by changing the command when running the test, it is possible to launch Xvfb and the test together (to be described later).

Run tests

Once you are ready, you can run the test using the following command. This is the command when Xvfb is already running on CircleCI, Jenkins, etc.

magicpod-command-line --magic_pod_config="<magic_pod-config.json full path>"

The command to run when launching Xvfb and the test at the same time is as follows.

xvfb-run magicpod-command-line --magic_pod_config="<magic_pod-config.json full path>"

In any case, it is necessary either to use the full path to magicpod-command-line on execution, or specify a relative path to magicpod-command-line.

Save the test results as a video

It is only possible to save a video of the test in progress when running local tests in a Linux environment.

Before running the test, install a tool called ffmpeg, which is required for saving the video.

sudo apt install ffmpeg

Next, add the option to save videos to the test configuration file. Note that this option cannot be added as a screen operation, so it is necessary to edit the configuration file manually. If you specify record_type as the key and always as the value, it will be saved as a video every time you run a test. If you specify only_failed as the value, it will only save if the test fails, and if you specify never, the setting is for it not to be saved.

{
  "owner": "MyCompany",
  "project": "SampleWebApp",
  "capabilities": {
    "browserName": "chrome"
  },
  "sendMail": true,
  "testCondition": {
    "browser": "chrome",
    "record_type": "always"
  }
}

Specify several environment variables as necessary when running the test.

# Mandatory in principle
export MAGIC_POD_RECORDING_SCREEN_SIZE=1280x1024
# Option
export MAGIC_POD_RECORDING_DISPLAY=:99.0
export MAGIC_POD_RECORDING_FRAME_RATE=8

xvfb-run magicpod-command-line --magic_pod_config=""

To find out what to specify for the above two environment variables, check the display numbers, etc., used, with Xvfb running.

# Example when launching Xvfb and Chrome manually. Launch these using method used for ordinary tests
xvfb-run google-chrome --no-sandbox &


# Check the running Xfvb processes and parameters
ps -ef | grep Xvfb


(Output example)
root       213   202  0 02:17 pts/1    00:00:00 Xvfb :99 -screen 0 640 x 480 x 16 -nolisten tcp -auth /tmp/xvfb-run.uyTR8o/Xauthority
root       810    13  0 02:18 pts/1    00:00:00 grep --color=auto Xvfb

640 x 480 x 16 is where the screen resolution is specified. In this case, the MAGIC_POD_RECORDING_SCREEN_SIZE is 640 x 480. The display number is :99.0, in accordance with the first parameter and screen option. As the default value is :99.0, it will work even if not specified, but you can specify this as an environment variable if the value is different. Be sure to close related processes (Xvfb, xvfb-run, google-chrome, etc.) after checking the variable value.

5. Run batch tests in parallel

For Browser Testing

To execute tests in parallel on a local PC, prepare multiple magic_pod_config.json files as shown below, and execute MagicPodDesktop by specifying each magic_pod_config.json file. In this case, it is not supported to use testSettingsNumber.

{
  "owner": "<Organization name. This is not the display name, but the alphabet description in the URL>",
  "project": "<Project name. This is not the display name, but the alphabet description in the URL>",
  "workDir": "<directory that does not duplicate any of the MagicPod settings in the PC>",
  "capabilities": {
    "browserName": "chrome",
    "chromeDriverPort":<port number that does not duplicate any of the MagicPod settings in the PC>
  },
  "internalAppiumPort": <port number that does not duplicate any of the MagicPod settings in the PC>,
  "internalMagicPodPort": <port number that does not duplicate any of the MagicPod settings in the PC>
}

• Ports and workDirs should not overlap within the same PC throughout the course of the browser test, Android test, and iOS test.

The following links are examples of configuration files.

• Config file1
• Config file2

For Mobile Testing

If you want to execute multiple tests on real devices connected to the local PC(Mac PC) in parallel, prepare multiple magic_pod_config.json files as follows, execute MagicPodDesktop with multiple actual devices connected to a single PC, and specify each of the magic_pod_config.json file separately. Note that when running batch tests on parallel real devices on Mac, it is not supported to use testSettingsNumber.

 iOS parallel execution settings

{
  ....
  "capabilities": {
    "udid": "<device UDID>",
    "wdaLocalPort": <port number that does not duplicate any of the MagicPod settings in the PC>,
    ....
  },
  "internalAppiumPort": <port number that does not duplicate any of the MagicPod settings in the PC>,
  "internalMagicPodPort": <port number that does not duplicate any of the MagicPod settings in the PC>,
  "workDir": "<directory that does not duplicate any of the MagicPod settings in the PC>",
  ....
}

Android parallel execution settings

{
  ....
  "capabilities": {
    "udid": "<device UDID>",
    "systemPort": <port number that does not duplicate any of the MagicPod settings in the PC>,
    "chromeDriverPort": <port number that does not duplicate any of the MagicPod settings in the PC>,
    ....
  },
  "internalAppiumPort": <port number that does not duplicate any of the MagicPod settings in the PC>,
  "internalMagicPodPort": <port number that does not duplicate any of the MagicPod settings in the PC>,
  "workDir": "<directory that does not duplicate any of the MagicPod settings in the PC>",
  ....
}

• Ports and workDirs should not overlap within the same PC throughout the course of the browser test, Android test, and iOS test.
• The device udid can be obtained by connecting the device to the machine via USB and using the command xctrace list devices in the case of iOS (note that a lot of information about the iOS simulator is displayed at the same time.) For Android, this can be obtained using the command ~/Library/Android/sdk/platform-tools/adb devices.

The following links are examples of configuration files.

• Config file1
• Config file2