Advanced Testing in an Agile DevOps Environment


In the past 10 years, the Software Development Lifecycle (SDLC), especially as implemented in government environments, has undergone major changes. The migration from waterfall project management approaches to those more closely related to Agile methodologies has decreased development time and costs. By enabling development teams to also manage all aspects of operations (and security), organizations also reduce support costs by eliminating redundancies. Finally, advances in technology, and the advent of a repeatable approach to production releases (Continuous Integration / Continuous Delivery, aka CI/CD), has allowed development teams to focus more on functional and business changes to applications and reduced the amount of time necessary to troubleshoot release issues in all environments.

These exciting improvements to the SDLC are accompanied by parallel changes for Independent Verification and Validation (IV&V). Agile development techniques are supported by the concepts of Test-Driven Development (TDD) and Minimum Viable Product (MVP), helping development teams define success from the perspective of successful IV&V. The wall between Development and Operations team members has been broken down, allowing a seamless integration when it makes the most sense to support the product. New, integrated CI/CD testing tools are inserted into the pipeline build process resulting in higher quality in higher environment. In this post, we will discuss how Innosoft has taken advantage of these advances to improve Independent and Integrated testing for our customers.

Integration of Testing into an Agile Process

Agile software development works along with the concept that development of novel business requirements into applications is best accomplished with an approach that can rapidly absorb results and changes into a successful delivery. This works best for rapid development, where a full set of requirements for a project is either not available or is otherwise unreliable. This happens very often in business and government, where a large overall goal for a project may be available, but the details may not have been fleshed out. Rather than spend months on a detailed specification, the development team, and business start work right away, identifying the aspects that are known, and building them, rapidly deciding together on what to do next. As the team works, it establishes epic stories of large-scale items to complete and breaks them down into discrete user stories, end goals written from the perspective of a user.

In a regular waterfall development scenario, testing might be more trivial: the requirements are known at the outset, so testing against those requirements is simply a matter of developing the test harness and running the tests that can be shown to satisfy those requirements. In Agile development, however, the requirements change with every Sprint. To address these changing requirements, we flip the script and use a TDD approach. User stories written for each Sprint are accompanied by a test script written for the user story. The developer has completed the user story when the test script runs successfully. Each epic is also accompanied by a definition of an MVP, describing what the product would look like in order to achieve the team’s goals. With this approach, we ensure that incremental enhancements are always tested and implementable. The process naturally injects quality and accountability, while increasing development efficiency.

Testing in a DevOps environment

DevOps has been around for a long time and actually is most similar to the way that software engineering naturally functions: The development/engineering team will always be the most knowledgeable about any application they build, and only through knowledge sharing (documentation), does the operations team really learn anything. The gap between Operations and Development can be huge, especially in Agile environments where there might be less emphasis on documentation. Simply assigning the development team to operations tasks, however, is bad practice, because it takes time away from addressing new user stories in the backlog. The solution, rather, is to encourage the development and operations teams to work hand-in-hand.

Excellent tools exist for this, specifically the Confluence and Jira tools from Atlassian, and communications tools such as Slack. Confluence can be used to create dynamic hyperlinked operations and engineering documentation, accessible by all teams. Even more important to testing is the Jira tool. Jira makes the Agile development process happen and can be directly integrated into testing, by tracking test results directly in the application. Slack may be the most important, taking team communication out of the heavy-overhead email tools, and giving direct access to operations and engineering counterparts. The result is that work simply gets done.

It must be remembered, however, that DevOps isn’t a toolset, it’s an approach to sustainably supporting an application by an engineering and operations team that is fully integrated. The result is different for every environment, but the goal is getting work done by breaking down arbitrary barriers that organizations naturally create. At Innosoft, we recognize this, and always put the success of our customers first by focusing on personal relationships.

Testing Integrated into CI/CD

On the technology front, the CI/CD stack provides for repeatable migration of new code from environment to environment. In the past, this was done with a hodge-podge of approaches, documented and undocumented. The result was unsuccessful releases and hours of troubleshooting. “Why does it work in DEV and not TEST? What’s the difference? Check the code versions? Maybe the difference is in the data?”

New tools are available, however, that allows an engineering team to stand up a blank slate that matches enterprise requirements for security and architecture, as well as the requirements for the applications, at the push of a button. At Innosoft, we use CloudFormation scripts, executed by tools such as Jenkins and Ansible Tower to orchestrate infrastructure automation, application builds, and automated testing. The resulting CI/CD stack decreases the development/test cycle and improves sprint throughput. Our testing tools are integrated into this CI/CD stack. Along with each build completed, there is a step to execute scripts written in Java and Selenium using the Cucumber Framework. Depending on the customer, we have anywhere from 76% to 88% coverage with automated testing. We also provide automated testing that is de-coupled from the CI/CD stack; for example, scripted functional testing that executes manually against a web browser, using Selenium software. Over three years for one customer, we have increased automated testing from 0% to 88%, and the average time to test has decreased from five to four hours.

Overall, the benefit of integrated IV&V into CI/CD is that developers and engineers no longer have to spend time researching environmental differences that may have caused a release issue and can focus more on the business and functional requirements. These improvements mean more effective work is completed with each Sprint, and the product is advanced more quickly to the benefit of the users.


Innosoft has 20 years of experience in software development, engineering, analysis, and data management. In that time, the processes, organizations, and tools supporting testing have increasingly changed and evolved. As these changes have taken place, Innosoft has adopted them for the benefit of our customers. We stand prepared to continue to provide an Agile approach, DevSecOps staff and organization, and expertise in CI/CD tools to support integrated and independent testing services. Please send any questions or comments to