Agile Data Conversion: 5 Keys to Success

Imagine that you are a Product Owner on the replacement of a legacy operations system using an Agile approach.  You may have experienced a mad scramble to complete the conversion of the data from one system to the other.  This scramble may have been the culmination of all the design changes between the original design and the final build, or even worse that the migration of data wasn’t considered until the end of the project.  Ultimately, this feeds into the statistic that 40% of all IT projects fail due to data concerns.

While Agile methodologies can lead to reduced risk and faster results, they also create a new twist for data conversion because of the inherent Agile premise that change is welcomed throughout the project lifecycle.  So, what are the key components that should be watched when converting data in a constantly changing Agile environment?

Key Component #1

It is important to recognize when either the source or target environments have changed.  This may be due to ongoing maintenance in the legacy system, or the development of new user stories in the new environment.  This check for changes may be performed manually using a schema mapping tool, like ER Design Studio, or could be achieved by querying the system tables and comparing the results to a snapshot of the schema after a successful run.

Key Component #2

Integrate data quality checks into the migration process.  Since the data will change in the legacy system over time, minimize the manual adjustments to data in the conversion process and focus on automating the process through creating rules to identify and remedy data issues.

Key Component #3

Build the data conversion process into an Automated Acceptance Test (AAT) Pipeline.  Just as you would never deploy untested code into TEST or PRODUCTION environments, you should always run your full regression test suite against the converted data before migrating it to a higher environment.

Key Component #4

Require Product Owner sign-off on story functionality with converted data.  In order to ensure that the system will function as designed, it is critical that the software and data are validated as a whole.  If possible, once approved the acceptance criteria should be included in the automated test suite for regression testing future runs.

Key Component #5

Welcome change – any findings can highlight exceptions or new requirements.  Every script failure, ticket about unexpected results, or AAT event will get your system closer to ready.  The key to success is to use each of these as an opportunity to strengthen the conversion process.

To Sum It All Up

While it may be possible to pick and choose the components listed as part of your Agile data conversion, the elimination of them will lead to greater cumulative costs as the project progresses.  This may be due to performing the same manual tasks with each sprint, or it may be the result of increased debugging time as items are not caught by the AAT pipeline until the build has moved into higher environments.  As a Product Owner, it is worth considering tools and processes that enhance the team’s technical capabilities to address automating repetitive tasks and ensuring that your acceptance is based on the new data as well as the software.