In our previous post, I discussed the Traditional I&IV approach to Independent Validation and Verification (IV&V). Agile doesn’t explicitly consider IV&V, because an emphasis on quality is included in the values and principles contained in the Manifesto for Agile Software Development. If we need to include Agile IV&V, we’re assuming that some principle related to […]
In our previous post, I discussed the Traditional I&IV approach to Independent Validation and Verification (IV&V). Agile doesn’t explicitly consider IV&V, because an emphasis on quality is included in the values and principles contained in the Manifesto for Agile Software Development. If we need to include Agile IV&V, we’re assuming that some principle related to technical excellence is likely to be met sub-optimally by the development team(s). An external IV&V function can lead to lengthy feedback loops and hinder agility.
Before I describe how to overcome this, I want to discuss the need for Agile IV&V. When organizations start using IV&V for their critical development efforts, they begin to feel that they should use them for all development efforts. This is more common when all the efforts are large. IV&V should be reserved for only those efforts of critical importance. When applications and systems are smaller, they tend to be less critical and reduce the need. A good decoupled architecture can help organizations focus and apply IV&V only where it is absolutely necessary.
There are two primary ways to approach IV&V with Agile teams, an embedded approach or a quality facilitation approach, which we call facilitative.
An embedded approach takes Agile IV&V staff and embeds them into each team. They take on the primary testing functions for the team in terms of acceptance, performance, and other higher-level tests. Lower level tests, like unit tests, are the responsibility of the development team. Team members may pair with developers to smooth the work flow and create shared understanding. They become full team members, even though they are from an independent organization. Figure 3 shows how this might look.
To be effective, the Agile IV&V staff must be considered valued team members. This can be difficult because of their specialization and role. A test engineer who could fix a piece of problematic code most likely will not because that would violate their independence. These challenges can be alleviated by an effective team launch that creates working agreements and common understanding.
Another important consideration is that the embedded IV&V team members have to be dedicated to their team. If they are spread across more than one team, then prioritization becomes problematic and the bonds of team membership will be weakened.
The great advantage of this approach is that IV&V will work in close collaboration with development on the same cadence, and feedback will be provided within the iteration as rapidly as possible. Testing will not create an inventory of fixes that must be addressed in a following iteration.
A second advantage is that the IV&V team members become part of the team. They embrace the shared identity of the team and work together with development each and every day. This helps to break down the “us and them” cultural challenge fostered by a traditional approach.
A facilitative approach maintains the independence of the Agile IV&V group. They sit outside the development teams, establish a relationship with them, and provide tools that help teams improve their quality. Figure 4 shows what this would look like.
Rather than actually running tests, Agile IV&V provides mechanisms to help teams perform this work on their own, and may offer software and necessary training. IV&V personnel may establish a set of metrics (usually through a dashboard like SONAR) to help the team monitor and continually improve the application’s quality. These metrics should be contextually adapted to help the team track what’s important so they receive continual feedback throughout each iteration.
This group uses these metrics to highlight concerns and to provide feedback from their independent perspective. They also periodically review the development and testing processes and techniques, giving feedback to the team(s) on how they can improve. Specific decisions for how to improve are left to each team, although they may have to meet required thresholds in order to deploy. Such thresholds are made public and continuously radiated to the team(s).
A disadvantage to this approach is that the IV&V group are still outsiders. An ‘us vs them’ feeling could emerge, particularly if quality thresholds sideline a deployment. A team launch can proactively address this by working with IV&V during their launch and creating external working agreements. This embeds IV&V—and its needs—within the context of the team. Said group can help through cultivating a positive relationship and asking how their information will be used. If they know what is helpful and what is not before a threshold ever gets crossed, the team will recognize they are all on the same side.
This brings us back to outcomes. How do you know you are fulfilling Agile IV&V’s purpose? This isn’t an easy proposition. There are three elements to it:
The organization’s mission is captured in its value stream, so more critical applications will be closer to the value stream. Creating some way to identify that closeness is probably the easiest way to determine criticality; it will likely be a bit abstract. One way to validate the method would be to see whether, as an application improves, the value stream’s outcomes also improve. If there is little to no correlation, then the application or system probably doesn’t require IV&V.
To know if the effort is successful, examine escaped defects (those that make it into production) and keep track of trends. A ratio that balances the criticality of defects with the number that escape can help determine IV&V effectiveness.
To see if the IV&V feedback loop is timely, you can track two durations: the time it takes for a ‘developed’ item to complete testing and the time required to repair escaped defects. If it is part of the development iterations, then both should be low for easy testing, identifying where problems occur, and getting them corrected.
As you can see, IV&V can be performed in a more responsive and Agile manner. It just takes some work to structure these activities so that the feedback lead-time is kept short. This requires a change from a “test afterwards” to a “test during” mindset along with new tools and skills to make it effective.
In summary, if you want to create a more Agile IV&V environment, consider the following:
If you have a need for IV&V, hopefully this post gives you a starting point for how to include it as part of your Agile journey. The approaches described here can be blended in various ways to help your organization. If you are already using these approaches or trying to get started with them, we’d love to hear about your experience. Tweet @excellaco and/or @paul_boos and let us know!
Last fall, Excella participated in the Department of Defense’s (DoD) Eye in the Sky Challenge....
At Excella, we’ve helped a wide variety of clients get the most out of their Agile adoption. It’s never easy, but as long...