Once the test is ready, set it to run daily. By doing this, you can detect problems that occur during the development period at an early stage. Even if they are not run every day, it is good if they are run every time there is a change to the source code. As the first step, though, set the test to run once per day.
Here, we shall explain how to periodically run a test using the CI (continuous integration) tool, Jenkins. A Mac PC is required to run MagicPod tests, so on this occasion we will explain how to simply set up Jenkins on a Mac PC and test.
- Install Java 9 in accordance with this procedure.
- Next, we shall set up Jenkins on a Mac PC in accordance with the article Let us try to install and use Jenkins.
- Follow the content of the article, and “Enter the administrator password,” “install the recommended plug-ins,” and “Register the initial users”. The screen shown in Figure 1 will be displayed.
- Next, following the “Let’s try running the test” section of the same article, “Register the Job” and “run the Job”. If the Job is successful, setup is complete (Figure 2).
Figure 1 Jenkins launch screen
Figure 2 Job success screen
Creating a MagicPod Test Execution Job
Once setup is complete, it is time to create the job to run the test.
Creating the Job
Next, in the same way as before, create a “Freestyle” job using an appropriate name.
Place MagicPodDesktop on the same Mac PC as Jenkins, right-click on it, and select “Open” to launch. Once it has launched, enter the authentication information and log in (once you log in, authentication information shall be saved on the PC, so you will not need to log in again).
When this task is complete, close MagicPodDesktop. If you do this while it is launched, execution of the test shall fail.
Editing the Settings File
Following the procedure here, edit the settings file, magic_pod_config.json, required for running the test.
- In “workDir”, you should specify <Jenkins worksplace>/magic-pod-work(example: /Users/user1/.jenkins/workspace/MagicPodJob/magic-pod-work).
- Set “xmlTestOutput# to true. This will output the test results to the working directory as an XML file to be read by Jenkins.
Adding the build procedure “Run Shell”
Click the “Settings” for the created Job, and move to the Settings screen, and select “Run Shell” from “Add Build Procedure” to add it (Figure 3).
Once added, the description should be written like "<path to MagicPodDesktop.app >/Contents/MacOS/MagicPodDesktop" run --magic_pod_config="<path to magic_pod_config.json>" in the “Shell Script” field (Figure 4).
Figure 3 Adding “run Shell”
Figure 4 Setting “Shell Scripts”
Adding the post-built processing “Collate JUnit test results”
In the same way, select and add “Collate JUnit test results” from “Post-Build Processing”, and specify the relative path <workDir location>/*.xml to the “Test results XML”, as shown in Figure 5. If you do not run any test, a warning message that the directory does not exist will be shown in red. However, once you run the test once, XML is generated and the error shall be resolved.
Figure 5 “Collate JUnit test results” setting
Save the settings and try to run JOB. Once JOB completes, a link to the results for each test shall be created on the results screen (Figure 6).
Figure 6 Link to test results
Email/chat notification of the test results
The test execution results notification can be provided by email or chat, such as Slack, etc.
To provide notification by email, follow the procedure in How to notify to Gmail when the build by Jenkins fails and run “Email Server settings” and “Job-side settings”. This will cause an email to be sent upon test failure.
To provide Slack notification, configure the settings on Slack and Jenkins, following the procedure in Notifying the Jenkins build results to Slack.
Running the test periodically
To run the test periodically, configure the settings in Running a job periodically with Jenkins.
For example, if you want to run the test at noon every day, you should enter H 12 * * * in “Scheduling”.
The settings introduced here are the most basic ones. For actual operations, the following additional improvements can be considered.
- Assign a fixed IP to the Jenkins server, so it can be seen by anyone within the company
- Use the Jenkins “Extended E-mail notification” to further specify the method by which emails are sent