My Data Migration focal point for this week moves us from prepping for migrating data to actually executing the Data Migration process. To summarize our progress so far, at this point you have:
- selected the data you want to migrate; and
- cleaned the data so that it is ready to move to your new system.
There are three steps remaining before you load your data into your ‘production’ environment:
- decide the method by which you will move each data-set;
- confirm the order in which you will move them; and
- establish the time it will take to do so.
MIGRATION METHODOLOGY
It’s safe to assume that when preparing to move to a new system, you will have many ‘data-sets’ (e.g. purchase orders, GL balances, invoices, etc.) that need to be migrated in order to make the new system functional. And it’s unlikely that you will use the same methodology to move all them. Some of the more common tools you can use to migrate data-sets are:
- a custom program (typically developed in-house);
- an application designed specifically to migrate data (typically purchased externally);
- your hands…for manually entering data.
Each of these is more appropriate to specific situations, usually dependent on the amount of data to be
transferred and its complexity (i.e. your custom programs would typically be used to move large amounts of data, whereas it might be more efficient to enter low-volume data-sets manually). I will not get into which strategy works best in which circumstances here, but I will say that it is essential to decide which of these ‘tools’ will be used with what data-sets well in advance of executing your data migration plan.
MIGRATION SEQUENCE
Now that you have confirmed the methodology you will use to move each data-set, the next step is to confirm the sequence in which the data-sets will be moved.
This is far from a simple process, but let’s start with the basics. As a general rule of thumb, you should migrate your master data before you move your transactional data. (e.g. you should move your customer master records into your new system before you move your customer orders. Otherwise, you will have no customers to which to assign the orders when the orders are migrated)
But determining the migration sequence is not always that simple. As an example, let’s assume you want to load a Purchase Order that was originally for 100 widgits, but you’ve already received 60 of them. Therefore, the PO now has an open balance for 40 widgits. If you migrate all the POs, followed by all the Goods Receipts, you may encounter a situation where the PO for 40 widgits is in the new system when the Goods Receipt for 60 widgits is migrated. Obviously, this will cause a problem.
You need to consider all these types of situations when establishing the migration sequence, which is why it takes a significant amount of effort to make the sequence correct.
ESTABLISHING THE DATA MIGRATION SCHEDULE
The last step in your data migration plan (prior to actually moving the data into your ‘live’ environment) is to confirm how long it will take to complete the full migration. This is essential, as most data migrations occur immediately prior to your new system ‘going live’. Therefore, you need to be able to ensure that you can complete the data migration in time for the ‘go-live’ to occur on schedule.
Once again, confirming this timeline can be a very onerous exercise. Simply put – despite all your planning and preparation – there is a ‘trial-end-error’ element to data migration. For this reason, I strongly recommend that you dedicate one of your pre-production environments to data migration activities. (I normally propose that – if budget permits – there be separate pre-prod environments for testing, training and data migration.)
The reason I feel so strongly about this is simple. If you undertake a ‘trial data load’ into the same environment you are using for testing the rest of your solution (i.e. ‘functional testing’), and it doesn’t go well (which is very likely for the first few trials), you may corrupt that environment. This could mean that much of the functional testing is de-railed and the functional testing team will have to start over. (Trust me, if this happens, the functional testers will NOT be happy!!)
However, if there is a dedicated ‘data migration’ environment, you can carry out as many trial loads as you would like. If a particular load doesn’t go well, you simply erase all the data from the environment and try again. Eventually, you will get things right (you should expect to carry out at least 3 or 4 trial loads) and you will then have a solid understanding of the time it will take to load all your data into your production environment.
SUMMARY
It’s difficult to express in a few paragraphs just how complex carrying out the steps outlined above can be. However, I think I can make my point by stating that – on a project of any significant size – creating and assigning the data migration methodologies, confirming the migration sequence and establishing the migration schedule typically takes 2 – 3 months.
