To establish test automation habits in the organization as quickly as possible, the two questions of “What is the purpose of automation?” and “In which order should automation proceed?” are extremely important, in addition to how you use the tool.
If these two questions have been appropriately answered, the flow of automation will proceed in the organization in a smooth way, and yield immediate results. If these two questions have not been answered in an appropriate manner, it will take a long time to achieve results, and the risk of failure along the way will increase. Let us look at each in turn.
1. What is the purpose of automation?
There are two objectives
When automating tests, enter one (or both) of A or B that follows as an objective.
A. Increase the productivity of the development team
By running daily tests in the development environment, it is possible to promptly notice and fix defects in the developed product(s).
If the defect is only discovered after the development period is complete, it becomes a real chore to specify the cause among the multitude of changes added during the development period. Even if it is identified, as time will have passed since the developer made the changes in question, it will take time to remember the content and fix the issue.
Additionally, if there are defects in the code during development, many developers will encounter this problem during development, so additional time will be spent investigating the issue. If tests are run every day, the development and test environment can be kept clean and development work can take place without losing productivity.
B. Early detection of defects and problems
By running daily tests in the development environment, the QA team will be made immediately aware of defects during the development period.
If a defect is discovered during the post-development testing period, the test schedule will be delayed while waiting to fix this defect, which may lead to a delay in the release. Even if manual testing is carried out prior to release, it is important for the QA team to maintain a certain level of quality in the code by running automated tests every day.
Additionally, by constantly running the tests created in the production environment, it will be possible to quickly notice and deal with any server or other problems that may occur in the production environment.
While one objective can be to “reduce the costs required for carrying out tests before release,” it is important to make either A or B an objective as well.
Running the tests every day is the key to success
Why are A and B important? This is because, if A or B are included in the objectives, you will definitely run automated tests every day. As a result...
- it will be easier for team members to realize the benefits by experiencing the benefits of running automated tests every day, and this will become a habit.
- Compared to when tests are run only once per month or once per week, if tests can be utilized dozens of times or more, the cost vs benefit ratio radically improves.
- In the case of daily testing, as it is impossible to “run them manually,” even if maintenance is hard work, rather than saying “I will just run the tests manually,” you are more likely to have the mindset of “let’s persevere with the maintenance.”
- The incorporation of this into the daily process will increase general awareness among the team regarding the need for automated testing maintenance, and this will create a pathway for automation to continue even after the members who initially implemented it are no longer there.
On the contrary, if you only run automated tests before release...
- as you are running tests that include all the changes made during the development period, it will be difficult to determine the source of errors, and the number of tests that are abandoned when errors occur will increase.
- If error correction and investigation become necessary immediately before release, you may be busy and put it off until later, or the automated test results you have worked so hard to get may not be used in the release decision. The tests that are put off until later will be abandoned in a state in which errors are not dealt with, and the number of failed tests will increase.
In fact, many of the customers who have improved their results using MagicPod, as in
- “Making fixes before release causes a flurry of work, so it is advantageous from an engineering perspective to be made aware of problems earlier by running tests daily.”(note)
- “If the regular test the previous day ran without a hitch but failed today, the cause is highly likely to be due to the difference in that day, so the number of manhours before resolution will be reduced.”(Showcase Gig)
- “Any degradation can now be detected at an early stage, making it easier to facilitate corrections. Moreover, there will no longer be a rush to meet deadlines.”(LINE Fukuoka)
value the advantages of A and B, and are running tests every day.
If your only goal is to reduce costs before release, you should reconsider your objectives. For automated tests run every day, there is an extremely high risk that maintenance will not take place.
Finally, MagicPod has a fixed monthly charge, regardless of how many tests you run. In addition, this fixed charge is based on the assumption that the user will run the tests daily. Make sure you run the tests every day to ensure that you are not wasting the money you paid.
2. In what order should you proceed with automation?
On startup
If you set the objectives as “A. Increasing the productivity of the development team” and “B. Rapidly detecting defects and problems,” the first thing to do is “Run tests on the latest version of the program under development every day, and create a mechanism by which the team can be notified of test results.”
“Automating a lot of tests” is not the same as “Reorganizing all of the test cases.”
It does not matter which tests you automate first. You should first automate one test case that is easy to automate.
For the actual method of creating the test cases, see Start Guide 1 and 2, and for the settings to run them every day, refer to Start Guide 3.
Did you manage to automate it successfully?
Congratulations! Compared to when you did not have anything, you have increased your efficiency a little, and improved the speed at which you can detect defects.
There may not be much of an effect created with only one test case, but if you create 5–10 test cases, the automated test cases will be able to find a substantial number of bugs.
The best method of causing automation to permeate through the team is to obtain results quickly. If there are benefits, however small, it will be easier to persuade your superior or team members to continue.
On main deployment
If you put a system in place to run tests every day, you will finally be able to spread automation within your team in a genuine manner.
It is recommended that you do this in the following order.
- First, get to the point where you can run one test case every day.
- While paying attention to whether you can maintain the test correctly and whether your team members become interested in automation, gradually increase the number of tests that you run every day.
- First, automate the checks on whether the basic functions and screens work as intended. Rather than automating one function in detail, cover the full range of functions and screens in a broad manner.
- After covering the full range of functions, try running the tests, and confirm that you can perform continuous maintenance.
- Once you know that you are able to maintain them correctly, you can proceed with the automation of more complex test cases, such as checking the behavior of error messages and checking behavior under special situations. When automating these kind of tests, consider the use not only of MagicPod, but also API and unit testing tools.
We will now describe steps 1 to 5 in detail, one by one.
- First, run them every day This should have been achieved at the time of startup.
- Increase the number of tests little by little, while running them every day When you try running daily tests, you will come across various problems you would not have noticed before, such as test stability, etc. Increase the number of tests little by little, while resolving these issues. There may be some people who think that by setting aside a large amount of time, creating all the tests, and then running them every day, they can be more efficient. This, however, is delusional. If you suddenly add a whole host of new tests without verifying whether you can maintain them, nobody will be able to maintain them and they will be unusable. Validate them one by one and steadily spread their use out among the team.
- Cover all functions broadly without much depth To achieve the best results with the least effort, first focus your efforts on covering all functions broadly without much depth. You should not automate all manual test cases in order from the start. Attach a priority order, and start by automating only the basic test cases.
- Run them to see whether maintenance is possible Once you can automate the full range of functions, you will then want to automate more, so first you should observe the situation for at least 1–2 months, to see whether this automation state can be spread to the team and whether it can be properly maintained. If you quickly increase the number of tests even though they cannot be operated correctly, the end result will be that nothing can be maintained, and all of the effort expended to that point will be wasted.
- Increase the number of tests further Around the time that you are able to continuously maintain the basic test cases, the team’s automated test operability will greatly increase compared to what it was before. Once you have come this far, you can automate even more tests. However, the more that you increase E2E(UI) tests, the more the maintenance costs will increase, and more time will be required to run the tests. Famous companies in Silicon Valley automate many tests, and make multiple releases in one day. However, that is because they have automated many tests as unit tests and API tests. Utilize unit tests and API tests, and with the cooperation of the development team and QA team, increase the coverage of automation.
Finally, we shall introduce how to make inquiries when you have questions with the method of operation or you are facing problems with errors, as well as the Slack community.
→6. Introduction to the method of inquiry and the Slack community (Browser)