In our previous post, I discussed the Traditional I&IV approach to Independent Validation and Verification (IV&V). Agile doesn’t explicitly consider IV&V, because an emphasis on quality is included in the values and principles contained in the Manifesto for Agile Software Development. If we need to include Agile IV&V, we’re assuming that some principle related to technical excellence is likely to be met sub-optimally by the development team(s). An external IV&V function can lead to lengthy feedback loops and hinder agility.
Before I describe how to overcome this, I want to discuss the need for Agile IV&V. When organizations start using IV&V for their critical development efforts, they begin to feel that they should use them for all development efforts. This is more common when all the efforts are large. IV&V should be reserved for only those efforts of critical importance. When applications and systems are smaller, they tend to be less critical and reduce the need. A good decoupled architecture can help organizations focus and apply IV&V only where it is absolutely necessary.
There are two primary ways to approach IV&V with Agile teams, an embedded approach or a quality facilitation approach, which we call facilitative.
The Embedded Approach
An embedded approach takes Agile IV&V staff and embeds them into each team. They take on the primary testing functions for the team in terms of acceptance, performance, and other higher-level tests. Lower level tests, like unit tests, are the responsibility of the development team. Team members may pair with developers to smooth the work flow and create shared understanding. They become full team members, even though they are from an independent organization. Figure 3 shows how this might look.
Figure 3
To be effective, the Agile IV&V staff must be considered valued team members. This can be difficult because of their specialization and role. A test engineer who could fix a piece of problematic code most likely will not because that would violate their independence. These challenges can be alleviated by an effective team launch that creates working agreements and common understanding.
Another important consideration is that the embedded IV&V team members have to be dedicated to their team. If they are spread across more than one team, then prioritization becomes problematic and the bonds of team membership will be weakened.
The great advantage of this approach is that IV&V will work in close collaboration with development on the same cadence, and feedback will be provided within the iteration as rapidly as possible. Testing will not create an inventory of fixes that must be addressed in a following iteration.
A second advantage is that the IV&V team members become part of the team. They embrace the shared identity of the team and work together with development each and every day. This helps to break down the “us and them” cultural challenge fostered by a traditional approach.
The Facilitative Approach
A facilitative approach maintains the independence of the Agile IV&V group. They sit outside the development teams, establish a relationship with them, and provide tools that help teams improve their quality. Figure 4 shows what this would look like.
Figure 4
Rather than actually running tests, Agile IV&V provides mechanisms to help teams perform this work on their own, and may offer software and necessary training. IV&V personnel may establish a set of metrics (usually through a dashboard like SONAR) to help the team monitor and continually improve the application’s quality. These metrics should be contextually adapted to help the team track what’s important so they receive continual feedback throughout each iteration.
This group uses these metrics to highlight concerns and to provide feedback from their independent perspective. They also periodically review the development and testing processes and techniques, giving feedback to the team(s) on how they can improve. Specific decisions for how to improve are left to each team, although they may have to meet required thresholds in order to deploy. Such thresholds are made public and continuously radiated to the team(s).
A disadvantage to this approach is that the IV&V group are still outsiders. An ‘us vs them’ feeling could emerge, particularly if quality thresholds sideline a deployment. A team launch can proactively address this by working with IV&V during their launch and creating external working agreements. This embeds IV&V—and its needs—within the context of the team. Said group can help through cultivating a positive relationship and asking how their information will be used. If they know what is helpful and what is not before a threshold ever gets crossed, the team will recognize they are all on the same side.
Understanding the Outcomes
This brings us back to outcomes. How do you know you are fulfilling Agile IV&V’s purpose? This isn’t an easy proposition. There are three elements to it:
- Alignment to the organization’s mission.
- Improvement in product quality.
- Timeliness of IV&V feedback.
The organization’s mission is captured in its value stream, so more critical applications will be closer to the value stream. Creating some way to identify that closeness is probably the easiest way to determine criticality; it will likely be a bit abstract. One way to validate the method would be to see whether, as an application improves, the value stream’s outcomes also improve. If there is little to no correlation, then the application or system probably doesn’t require IV&V.
To know if the effort is successful, examine escaped defects (those that make it into production) and keep track of trends. A ratio that balances the criticality of defects with the number that escape can help determine IV&V effectiveness.
To see if the IV&V feedback loop is timely, you can track two durations: the time it takes for a ‘developed’ item to complete testing and the time required to repair escaped defects. If it is part of the development iterations, then both should be low for easy testing, identifying where problems occur, and getting them corrected.
Conclusion
As you can see, IV&V can be performed in a more responsive and Agile manner. It just takes some work to structure these activities so that the feedback lead-time is kept short. This requires a change from a “test afterwards” to a “test during” mindset along with new tools and skills to make it effective.
In summary, if you want to create a more Agile IV&V environment, consider the following:
- Is IV&V absolutely necessary for this development effort? Is the application or system critical to the survival of the organization?
- Which approach will work best for you and your organization? While you can blend the facilitative and embedded approaches, you will probably find that one resonates more strongly. Do you need testing performed by an independent group, or can you achieve the results through a disciplined approach to tools and metrics?
- Structure your IV&V group and develop their skills based on the approach you decided to take.
- Regularly review your process and the length of your feedback loops, and try to continuously shorten these loops.
If you have a need for IV&V, hopefully this post gives you a starting point for how to include it as part of your Agile journey. The approaches described here can be blended in various ways to help your organization. If you are already using these approaches or trying to get started with them, we’d love to hear about your experience. Tweet @excellaco and/or @paul_boos and let us know!