Ensuring High Quality Software at EfficientIP with QA Digital Factory

This week’s blog comes from QA automation expert Nabil Bourenane.

There are domains where software requires a high level of quality since any major bug can lead to high service impact. At EfficientIP we are developing such a strategic solution – to put in place a digital testing factory that helps us release stable versions with minimal regression impact.

Why do we need a digital testing factory?

Our factory is embedded in the development process and serves the following objective: controlling quality and testing for non regression on very important infrastructure software running at the heart of the Internet and corporate IP networks.

We are convinced that testing software is mandatory in any modern development process, developing tests during or even before the functional code is a must. Even if test driven defined development is not always easy to set-up, performing continuous testing with a fully automated digital factory allows us to save time in our dev cycle. It also provides efficiency in the global release process by allowing more time for global test coverage.

It is well known that bug fixing costs increase further along the development phase:

What is our digital testing factory?

Developing software embedded in an appliance requires large spectral testing coverage, especially for end-to-end testing.

The automated test bench consists of multiple SOLIDserver appliances installed and configured in various topologies, as close as possible to general use cases of our customers. Both management plane and service plane are covered for this specific part of the system that can be considered as SUT (System Under Test). In addition, a set of standard stubs (not only mocks) is added with the embedded services (eg: ISC Bind daemon or Microsoft DNS server).

For efficient testing, we do also require some clients that will use the proposed service like DHCP to get their IP configuration and the DNS to get access to their applications. Many scenarios for clients are defined in order to fit with specifications but also with special use cases and bugs that customers have encountered.

Intensive testing of the configuration and administration interfaces are performed using each topology set with its corresponding network topology. This requires specific simulated user interface, in our case mostly browser based but also through standard API.

Test frequency matches the changes on the software code and serves the high quality objective. Each commit performed on specific branches of the code triggers a dedicated test run. The coupling of the digital testing factory and the source control system is very tight in order to guarantee that software changes will be tested as soon as possible. This helps the development team to fix small errors early rather than very complex ones later on.

We also perform night builds of various supported versions of the solution in order to obtain a clear status of the software stock in the source repository. In addition, we can of course launch a specific and dedicated quality test at any time during the day, allowing teams to validate the full solution when issuing a specific fix that may be blocking for a customer in production.

How is our digital testing factory built?

The digital testing factory is a full product that requires a dedicated team and resources. It requires any phases of a development process driven by very knowledgeable people. User experience on the digital testing factory is a real subject, it is highly technical and requires deep understanding from infrastructure up to the client. Abstraction process should be pushed very deep in order to have the ability to quickly add tests on any aspects of the solution.

We decided to adopt a keyword-driven testing approach for easy maintenance of all the test scenarios and better coverage of the complete SOLIDserver solution. It is composed of 4 macro scenarios orchestrated in pipelines and grouping test conditions as reusable resources, around 15 test suites and 10.000 main assertions or control points. A history of each test is kept for evolution and some easy to manipulate reports are used to quickly identify the steps failing in the pipeline.

The factory performs unit testing with very simple features up to advanced module testing reproducing typical topologies of our customers and features like DNS engine switching mechanism used in hybrid DNS.

The principal tests are chained together early in the pipeline in order to fail fast if any mandatory feature of the solution is broken. Although this is not the way we generally perform coverage of unit tests that are run continuously on the software development side, it optimizes the time required for the whole testing process. The smoke tests are mandatory and cover installation, basic parameters, licence installation, logging, service control, internal security or binaries release control. If those are not working, there is no need to continue non-regression testing since the solution will not be in a good environment to run and may produce false positives.

The TAF (Test Automation Framework) mainly uses virtualization framework for SUTs, Stubs and Drivers but also cloud solutions for some features that target multi-cloud environments. Highly triggered by the central source control system, the main tools composing our factory are:

• Jenkins: used to orchestrate and schedule defined tasks, using standard Groovy pipelines
• Hashicorp Packer & Vagrant: used to build specific system images and deploy them with ease, enhanced with various Ruby scripts
• Selenium in containers: scalable hub and nodes for web browsers mainly to test the user interface interactions
• Nightwatch.js and node.js: as the end-to-end Testing Framework

And we have added a lot of secret sauce around all these recipe ingredients in order to set up the various infrastructures, networking and usage situations in order to fit as close as possible to field scenarios.

Why are we so proud of our testing factory?

A dedicated and rich digital testing factory is an investment and always a cost center. But without this factory the produced software will not build a solution which serves over a thousand clients around the world and is capable of supporting only specific industries and IT situations. Part of the Internet is running EfficientIP solutions, this is precious enough to be handled with a lot of care from our dedicated teams. The EfficientIP solutions are intricate and serve so many situations that need to be deeply tested at each code line change, at each fix installation and each new feature integration.

A lot of time and effort concerning basic, repetitive manual tests has been invested to migrate towards the digital testing factory, allowing time to be freed up for performing more complex manual tests. The order of magnitude between manual and automatic testing in our infrastructure case is about 50 times.

There will always be bugs in solutions developed, in some industries we can use canary testing, blue/green deployments and let the end users test and provide feedback. When proposing an essential solution such as enabling user to application communication, deeply embedded in the infrastructure with services like high performance DNS, security or GSLB, the bugs need to be found as soon as possible. This is the mission of our testing factory, including wonderful tooling but also people dedicated to increase the overall quality of the product and consequently the satisfaction of all our customers.