Why Do Testers Need CI/CD?
Competency in the TestOps field is now just as much of a basic requirement for QA engineers as the ability to write automated tests. This is because of the ongoing development of CI/CD and the increasing number of QA engineers who work with pipelines (or the sequence of stages in CI/CD) and implement their own. So why is CI/CD such a great tool for quality control? Let’s find out.
Running tests automatically
Automated tests haven’t been run locally in what feels like ages. These days, CI/CD pipelines run tests automatically as one of their main functions.
Pipeline configuration can be assigned to DevOps. But then we will be a long way from making use of CI/CD’s second function: quality control, or more precisely, “quality gates”.
Quality control using quality gates
But what are quality gates? Let’s say the product code is like a castle. Every day, developers write new code – which could weaken the foundations of our castle or even poke holes in it, if we are really unlucky. The purpose of a QA engineer is to test each feature and reduce the likelihood of bugs finding their way into product code. Lack of automation in the QA process could cause QA engineers to lose sleep, since there is nobody to watch over all the various metrics – especially at dangerous times, like Friday evenings when everyone wants to leave work and is hurrying to get everything finished. An ill-fated merge at that moment can cause a lot of problems down the road.
This problem can be solved by building in quality checks.
Each check deals with a different important metric. If the code doesn’t pass a check, the gates close and the feature is not allowed to enter. A feature will only be merged into the product when it passes all the checks and potential bugs have been prevented.
What quality checks can be included in CI/CD?
We need to put together a list of checks to ensure that the process is as automated as possible. They can be sequenced in “fail first” order. A feature needs to pass all the checks to get through the pipeline successfully. The initial checks are the ones that make sure the app is capable of working: build, code style check, and static analysis.
“Build” speaks for itself: if the app cannot be built, the feature does not progress. It is important to incorporate a code style check into your CI/CD pipeline to ensure the code meets unified requirements, as doing so allows you to avoid wasting time on this kind of bug during code reviews.
Static analysis is an extremely important tool for judging code quality. It can point out a huge number of critical errors that lead to bugs.
We then continue with stage-two checks: unit tests with coverage analysis and coverage quality control, as well as integration and systems tests. Next, we review detailed reports of the results to make sure nothing was missed. At this stage we may also perform a range of non-functional tests to check performance, convenience, and security, as well as screenshot tests.
When developing a pipeline, we need to pay attention to two competing requirements:
- The pipeline must guarantee the best possible feature quality in light of your needs.
- Time spent running the pipeline should not slow down your workflow. It should generally take no more than 20 minutes.
Examples of tools to incorporate in quality checks
Code style highlighting
A code style is a set of rules that should be followed in every line of code in a project, from alignment rules to rules like “never use global variables”.
You might be wondering what style has to do with testers. The answer is a lot, actually. A style check provides several benefits for QA experts, not to mention the rest of the team:
- A unified style helps developers work with the code and gives them more time to implement new features and fix bugs.
- A unified style allows you to dispense with manual code checks and use a CI/CD tool to run the checks instead.
Large companies usually have their own style guides that can be used as examples. For instance, Airbnb has a JavaScript style guide and Google has a number of guides. You can even write your own.
The choice of tools for code checking depends on the language. You can find a suitable tool on GitHub or find out which tools other teams are using. Linters use bodies of rules and highlight code that fails to abide by them. Some examples include ktlint for Kotlin, checkstyle for Java, eslint for JavaScript.
Static code analysis
Static code analysis is a method of debugging by examining source code without executing a program. There are many different static code analyzers on the market. We’ll take a look now at a platform we’re developing ourselves – Qodana. The major advantage of this code analyzer is that it includes a number of inspections that are available to us in JetBrains development environments when we’re writing code.
Many of you probably use an IDE-driven approach, where the IDE helps you write code and points out various bugs such as suboptimal code usage, NullPointerExceptions, duplicates, and so on.
But unfortunately, you can never be sure all the critical problems found by the IDE were fixed before the commit. One way you can guarantee that all the problems will be addressed is by incorporating Qodana into your CI/CD pipeline. If you can’t fix everything at once, you can select critical problems, add them to the baseline, and gradually work your way through the technical debt. This allows you to avoid slowing down the development process while keeping the problems that have been found under control.
Test coverage
Code coverage is a metric that helps you understand how well your code has been covered by your tests (generally unit tests).
Here you need to define the minimum coverage percentage that you want to support. The code won’t be able to go live until it has been covered sufficiently by the tests. The minimum percentage is established empirically, but you should remember that even 100% coverage may not completely save your code from bugs. According to this article from Atlassian, 80% is a good figure to aim for.
Different coverage analyzers are available for different languages, such as Jacoco for Java, Istanbul for JavaScript, or Coverage.py for Python. You can build all these analyzers into your CI/CD pipeline and track the metrics with ease.
Shaping the release process
In addition to automatically running tests and ensuring particular code quality requirements are satisfied, CI/CD lets testers organize the release process.
The release process can be complex and can depend on a lot of different manual actions. It is often a completely manual process: the artifact is created by a developer, then passed to the testers for checks, and finally comes to the person who knows how to roll it out for the go-live. Once again, there are a lot of potential choke points here. For instance, one of those people could fall ill or go on vacation.
An effective release process will look different for each team, but it will generally include the following steps:
- Each change in the Git branch triggers a build of the app.
- The build undergoes quality checks and does not become part of the main branch until it passes all the checks successfully.
- A release candidate is taken from the release branch or the main branch: this fixes the version and guarantees that nothing will go live unless it has been tested and has not been changed afterwards. This helps with tracking releases and all the changes they include. In addition, by storing artifacts of the stable version, you make it possible to revert to them quickly in the event of an unsuccessful release.
- The release candidate is tested and undergoes final checks.
- The release candidate goes live. This may be either a manual pipeline launch or an automated launch, if the release candidate passed all the checks at the preceding stage. The choice between an automatic release process and a manual one will depend on how frequent and how important the releases are, as well as the preferences among team members and the convenience of the rollout.
Any CI/CD system allows you to set up this kind of process. The process should be convenient for the whole team, including the testing team.
Given the factors outlined above, we believe following these basic rules will help ensure an easy and efficient release process:
- Artifacts must be ready for download and testing, ideally all in one place.
- As many checks and tests as possible must be automated. They should take place without human intervention.
- All complex operations with builds should be as automated as possible.
- All builds that will go live should be recorded and remain available for a certain period after release. This will help if you need to investigate errors in the production version, reproduce bugs, or just track the history.
We would also like to remind you that if quality metrics are not controlled automatically and do not affect anything, they are useless, as there’s no way to guarantee that these metrics will be adhered to.
Implement pipelines, automate processes, and use static code analysis!
Your Qodana team
Subscribe to Qodana Blog updates
