Many people think that Agile approaches don’t work for AI efforts; that there is a fundamental mismatch between Agile practices and the work required for advanced analytics, Machine Learning, and AI development. This could not be further from the truth. Agile approaches are specifically intended for complex problems with high uncertainty and a need for emergent learning and validation—key characteristics of AI efforts.
Too often teams think “Agile” is a single set of very specific practices, usually centered around Scrum, but often involving detailed processes and artifacts that go far beyond what even Scrum prescribes. Applying Agile this way ignores the significant differences between advanced analytics and the typical software development projects that shaped standard Agile practices. Successful application of Agile for AI requires that practitioners focus on the principles of Agile rather than specific methodologies and practices.
What is Agile for AI?
Agile for AI is in many ways the same thing that “Agile for anything” should be. It’s a mindset that emphasizes outcomes, embodied by practices focused on establishing rapid feedback loops based on intense collaboration around small increments of work. High-value products increasingly require working in environments of significant uncertainty and volatility. Agile approaches allow us to explore high-risk solution approaches, validate (or disprove) relevant progress, and adjust our AI product designs early if needed, quickly and iteratively.
“Successful application of Agile for AI requires that practitioners focus on the principles of Agile rather than specific methodologies and practices.”
Benefits of an Agile Approach
The biggest risk to any IT project is whether the deliverable will produce the desired value for internal or external customers. In other words, are we building the right thing? For AI efforts, this is compounded by the fact that our customers’ needs may not always be feasible, because of limitations in the underlying data.
Agile principles like “responding to change over following a plan” and “customer collaboration over contract negotiation” help us manage the inherent uncertainty of AI efforts and reduce the resulting risk. Progressing in small increments with systematic, frequent validation allows us to learn much more quickly. We can discover whether a product will create value and whether the chosen solution approach is effective, and to what degree. This avoids the all too common risk of burning through our budget only to discover that we’ve built a poor solution based on unproven assumptions of value.
An additional benefit is that Agile makes it much easier to identify and deliver the earliest usable product and thus deliver the solution faster—whether for internal customers or an external market.
An Agile approach for AI provides the following benefits:
- Manages the inherent uncertainty of AI efforts and reduces the resulting risk
- Rather than exhausting our budget chasing unviable solutions, we can frequently validate, and if necessary, rapidly abandon one concept and pursue another
- Faster time to delivery and to market
Why Traditional Agile Methodologies Fail in AI Projects
Agile is a proven approach for delivering software solutions. With traditional agile methodologies, software teams work in time-boxed iterations to develop and deliver incremental portions of functionality in the form of vertical slices across the entire tech stack. This makes it easier to see progress and validate that the solution is developing as intended. Unfortunately, this approach is less effective for data-science solutions.
One reason traditional Agile methodologies fail in AI projects is that when it comes to advanced analytics products, the development stack often looks more like a pyramid with a broad base supporting fewer user-visible outputs.
Data science requires a detailed examination of available data, thorough analysis of solution alternatives, and repeated hypothesis testing to determine the best approach. As a result, data efforts tend to emphasize research and learning, work that does not naturally fit into a software development time-box.
Agile as traditionally applied to software also makes implicit assumptions about what is known and certain. Two specific ones can cause problems for AI efforts. First, software efforts tend to assume that the right problem to solve has been identified. Second, they also assume that the overall solution design is effective and will result in a cohesive, valuable solution. With AI, we need to validate both assumptions as we progress.
What Makes AI Projects Different from Software Development?
In addition to the usual uncertainties of software development, AI projects usually have many additional layers of unpredictability and unknowns. These are just some of the questions AI teams need to examine:
- Is there enough relevant content in the data to derive valuable, actionable outputs?
- Do we have enough training data to generate predictive results?
- How good is the data quality?
- What biases are embedded in our source data that could skew predictions towards unethical impacts?
- What biases are in our algorithms that could similarly skew predictions?
- How much entity resolution or other effort is required to cleanse and integrate different data sources?
- Which combination of algorithms is going to give us enough accuracy?
- Can we get the model to process the data fast enough, so the outputs are timely and useful?
- Is what the data tells us stable enough over time that the usefulness of the model holds up long enough to get ROI?
As a result, AI projects do not follow a linear, predictable path. Instead, they require constant exploration and experimentation, building off validated knowledge and proven (or disproven) assumptions. Much of the work is far removed from end-users, so feedback must be gathered in many forms, not just sprint demos. User stories often feel awkward and may need to be replaced by testable hypotheses and tech stories. Sprints may be too short to work out a messy modeling algorithm but too long if exploration quickly indicates a need to pivot.
“At its core, Agile very much supports the kind of iterative learning loops that AI development requires.”
How Agile for AI is Possible
Here are three key considerations to make Agile work for AI:
1. Build on the principles of Agile – not specific implementation frameworks.
“Established” Agile processes or frameworks quickly clash with the reality of AI development for all the reasons described above. However, at its core, Agile very much supports the kind of iterative learning loops that AI development requires. Much of what is considered mandatory “best practices” for Agile software development is not specified in the Agile Manifesto or even the Scrum Guide. Teams can position themselves well if they start with an Agile mindset before jumping into specific practices.
The right “Agile” is always context-dependent, so AI teams should let go of things that don’t work for them, adopt blended practices from multiple frameworks, and create their own where needed. Some successful AI teams work in three-week sprints with plenty of opportunities for internal learning. Others don’t use sprints at all. Many have abandoned the classic user story format. Some are comfortable accruing significant technical debt during early exploration, only gradually tightening their definition of “done” as they get closer to a production release. What stays constant across these effective teams is a consistent focus on collaboration, learning, and adaptability.
2. Approach AI data projects from the top-down and bottom-up.
Outcomes of AI efforts depend on finding the overlap between …
- what is possible based on the information that can be extracted from the underlying data
- and what is desirable, based on identifying actionable, timely decision support for business stakeholders.
The first part requires bottom-up discovery – data analysis, model exploration. The second is focused on top-down User-Centered Design or similar approaches such as value definition, prototyping, etc. Initially, teams often benefit from decoupling these distinct problems to reduce dependencies and delays. They can maintain alignment through shared vision and direction. Since the insights from both aspects inform and constrain each other, it is important to ensure feedback and learning between them.
3. Include and adapt related concepts like Lean Startup and DevOps.
For advanced analytics efforts, there is much more uncertainty around first selecting the right problem and then the right solution approach to maximize value. Combining agile delivery with elements of Lean Startup helps address both by providing early and ongoing feedback that goes beyond validating the correct implementation of known requirements.
AI development needs an approach that converges from lots of potential questions to address, through exploration, to refine the set of potential questions, and to a systematic selection and solutioning process. Often, the fastest way to an effective solution is by exploring multiple options at the same time in parallel.
Once code is developed, however, a culture of solid engineering practices is required to ensure repeatable, reliable, and sustainable configuration management, deployments, and monitoring. DataOps and MLOps have adapted and expanded the technical approaches of DevSecOps to address the additional complexities of automating the processing and maintenance of both data sets and algorithmic models.
Ready to Explore Agile for AI? Reach Out to Our Team Today
Ready to explore Agile for AI? Excella’s cross-functional approach encourages iterative delivery, addresses cultural changes, and enables artificial intelligence at scale. If your federal agency, commercial business, or nonprofit organization would like to learn more about Agile for AI and how this approach can help your product get to market faster, contact us today for a no-obligation conversation, 703-840-8600.