Digital transformation has gripped us all! Most of the businesses are moving towards a cloud-centric environment and leaning towards ‘Shift-left’ DevOps practices. This rapid paradigm shift in Software Development Life Cycle (SDLC) is creating a need to integrate continuous testing to build out resilient CI/CD pipelines. Balancing quality and speed is a major challenge and continuous testing in CI/CD can act as a catalyst in the whole process.
To ace the competition, businesses are looking for better ways to perform CI/CD Pipeline Automation Testing. One of the approaches adopted by many is an integration of continuous testing in the entire product development life cycle, right from the start until the end.
The basic goal for testing any software is to isolate errors or bugs that will impact the quality or will compromise the security framework. In the case of legacy waterfall development cycles, the release cycle was long, and the software tester / QA would spend time manually simulating and testing various scenarios. However, as organizations are adopting DevOps ways of working, the frequency of release is changing from months to every hour and it would be inefficient as well as a humongous task to manually test each time. Also, manual testing might not cover all the corner cases, leaving scope for production defects.
Continuous testing in DevOps is an uninterrupted process of testing throughout the SDLC where test cases are included in every stage in the CI/CD pipeline. Right from code commit, to build, to provisioning, to deployment, and everything that happens in between—all the steps are tested automatically. In case there are any bugs encountered in the process, instant feedback is provided to developers ensuring rapid fixes of failure before the final release.
Test automation and continuous testing are often used interchangeably. However, these two are not the same. Wikipedia defines Continuous Testing as “the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.” However, this definition is not accepted by many. Lisa Crispin, an influential Agile Testing Professional, highlights that automated test is an integral part of continuous testing framework but the terms cannot be used interchangeably as continuous testing in CI/CD is not just about test automation. Continuous testing in DevOps encompasses and is a collaborative effort involving processes, teams, and tools along with automation. It is an evolution of test automation and aims to provide validation and feedback at every step of the software delivery pipeline - right from design to production.
Continuous testing is beneficial in more than one way. Following are some of the benefits of CI/CD testing:
Now that we’ve looked into what continuous testing is, and how it is aligned with the test automation, let’s focus on how we can bring it all together and fit in the DevOps pipeline.
In a typical scenario, a developer writes a piece of code and commits it. The code then goes into version control. This is the first step in DevOps. In the next phase, continuous integration, the code is merged into the main branch and then Jenkins / Git is used to build and test. Following this, as part of continuous testing in CI/CD, QA writes automated test scripts and executes them.
As part of the continuous testing in DevOps, test cases are introduced even before the developer starts writing the piece of code, i.e. during the design phase, and until delivery. If the test fails, the failure notification is sent to the developers and if the test passes, the code is pushed for Continuous Deployment. All the aspects of CI/CD testing like Unit, functional, security, and performance which were manual before should now be automated and made as atomic as possible to help reduce refactoring effort. Having said that, we need to ensure we try and automate enough to be able to push something from the design and development stage into production automatically or with minimal effort possible.
Let’s understand how these tests should be selected, written, and executed.
Multiple types of CI/CD testing can be integrated into the pipeline as it is important to have coverage on all the software aspects. Some of them are listed below:
The list is not comprehensive however, it touches on key testing types. Apart from the ones mentioned above you will hear API Testing, Performance / Load Testing, Stress testing, regression testing, security testing, UI testing, etc. Each of these testing variations can be performed using multiple CI/CD Test Automation Tools which allow concurrent test execution like Selenium and Cypress.
For the tests to run smoothly in a continuous integration (CI) environment, it is important to run them independently. Let’s consider that you are building an application to help find restaurants in the city. Some of the important workflows for this application would be, users should be able to search the restaurants, login into their accounts, and check out the different restaurants based on selected criteria. So while writing the test workflow focus on creating independent test cases to run the test on each of the features. This will help reduce the CI/CD testing time and will also help identify failures at a feature level.
In serial CI/CD testing, the tests are run in a sequence one after the other. However, in parallel testing, multiple tests run side by side, and hence the overall execution time is shorter. For a parallel test execution suite to run smoothly, factors like the number of processors and the capability of the processors play a very important role.
Identifying the types of tests that should be included and automated in the CI/CD pipeline of the software is a difficult task. Agile Testing Quadrants is helpful for the teams to identify and plan the types of tests that should be included, and the resources that would be required for implementation. This model was introduced by Brian Marick, an agile testing consultant, in the year 2003. It combines both the aspect of testing, business, and technology.
There are four quadrants as depicted in the picture below. The bottom of this matrix is tech-focussed, while the top is business-facing. The left-hand side focuses on tests that guide the development of the product whereas the right-hand side critiques the product.
Quadrant 1 (Lower left corner) - All the technology-facing tests that guide the programming team are part of this quadrant like unit testing, integration testing, and component testing. These tests are automated and provide regular feedback to the developers on the code quality and if the software is behaving as expected. The test cases for CI/CD Pipeline Automation Testing are written alongside development to confirm that the functionality being built is moving in the right direction.
Quadrant 2 (Upper left corner) - Partially automated and partially manual, this quadrant is business focussed and the tests are performed to validate product specifications, simulate or build a prototype. Automated functional testing which is done to ensure the product is aligned with the business specification is part of this bracket.
Quadrant 3 (Upper right corner) - This quadrant is focused on business-facing tests that critique the product. It primarily includes manual testing like exploratory testing, A/B Testing, User Acceptance Testing (UAT), and usability testing. These tests are more user-oriented and are helpful to assess aspects like ease of use, flaws in application workflow, etc.
Quadrant 4 (Lower right corner) - This corner is for technology-facing tests like performance testing, load testing, and security testing of the software application. Several tools are available today which can help automate this type of testing like Apache JMeter, LoadNinja, etc. The tests in this quadrant should be prioritized based on the application-specific need. For example, for reporting software containing non-sensitive data type information, fast refresh and loading of the page is of utmost importance so implementation of performance testing would add value.
Following the order in which quadrants are listed or ensuring at least one test from each quadrant must be included is not necessary. Choice of type of CI/CD Testing and number of tests to be included in the software is circumstantial. So, assess the need and leverage the test or the quadrant which is relevant for your specific use case. These quadrants should be used more like a guiding framework so that the whole team understands the importance of testing and starts thinking about it right from the start.
Coined by Mike Cohn, the testing pyramid is a test automation framework that gives a graphical representation of testing types and the ideal composition of each of them for better coverage. However, it’s not a complete guide and there are other tests too that should be included apart from the ones part of the pyramid.
The above pyramid calls for automation testing at three different levels. Unit testing is the foundation of automation test strategy followed by service-level testing and user interface testing. Service here refers to testing the application’s response based on the set of inputs.
CI/CD Test Automation Tools
It is important to involve the complete team in the test automation process. Business stakeholders can help classify high-priority features, tests can help the developers identify the edge cases, and the developers can help write the test scripts. Keep the automation suite lean and create a strategy on how much automation is possible and what tools will help achieve that.
After defining your automation strategy and deciding what automated tests to incorporate, the next step is to select the automation tools. As a first step in selecting the tool, focusing on defining two things:
1. What tests need to be implemented?
2. Which is the preferred language in which test cases should be written?
Once you have clarity on the above two, start looking for automated tools that best suit the requirement and promote cross-functional collaboration between testers, developers, and business stakeholders.
Some of the popular test automation tools are:
Selenium is an open-source end-to-end testing framework with a user-friendly UI. It supports parallel testing and the test cases prepared can be tested using any OS. The framework also allows the testers to write complex test cases in RUBY or HTML.
There are few limitations of Selenium too like It does not support CI/CD automation Testing for desktop applications/windows applications. Also, the Selenium testing framework depends on other third-party frameworks like cucumber for reporting.
QuerySurge is an automated data testing and validation tool. It is used for testing Big Data warehouses, Data migration products, Extract Transform Load (ETL) solutions and Business Intelligence (BI) reports.
CI/CD test automation for data solutions can be done using QuerySurge’s Restful API or Command line. The process can be automated to execute unattended and post-execution, it shares all the reports of pass/fail results and data anomalies detected over email.
One of the limitations of QuerySurge is that it does not point to the exact location/cell where data discrepancy exists. However, this can be overcome easily by writing a small utility on top of the tool to automatically detect the exact location where the data mismatch is.
Apache JMeter™ is an open-source application that is used primarily for load and performance testing. It can be used on not only static but even dynamic resources, including Web dynamic applications.
This DevOps load testing tool can also be used to perform automated functional testing. The stress testing helps identify the point where the software application can fail. JMeter supports multiple plugins like basic graph and Response Times Over Time Listener.
The application is designed on a multithreading framework, allowing concurrent sampling on multiple threads and simultaneous testing of varied functions on separate threads. You can also visualize the test results in multiple different ways like tree views, XML, graphs, JSON, etc.
Apart from the ones listed above, there are many other continuous CI/CD testing tools available. For example, there is Nunit for C#, JUnit and TestNG for Java, and Embunit for C++ development. Choose the tool that best supports your continuous testing requirements.
Apart from including the multiple types of CI/CD Testing and enhancing the testing coverage, when designing the continuous testing framework focus on some of the key points listed below:
DevOps is also cross-referenced as agile on steroids. To keep up with the pace, testing needs to be automated more than ever now. The scope of testing is not limited to re-running unit test cases, functional tests, workflow, bug finding only but also includes security testing, load, and performance testing. Hence it is crucial for the testing teams to:
The trick to successful continuous integration and continuous delivery is “Continuous Testing”. Continuous testing in CI/CD helps isolate failures and provide a critical feedback loop to move fast and build effective software solutions. Most importantly, it helps build a quality CI/CD pipeline and reduces the risk and wastage in the software development lifecycle.