General advice

The following steps are recommended when migrating data:

• Step 1: Ask “Do I need to migrate data?”

• Step 2: Understand export and import options offered by each system: we can write directly to the database, but recommend you use our existing Selenium scripts

• Step 3: Consider the activities and skills required to migrate the data, including costs; you can do migration or you can ask us to do it. Most of our scripts work directly ‘out of the box’.

• Step 4: Evaluate the data size and composition; once a script is created it can migrate literally thousands of records, the only downside is the time it takes.

• Step 5: Prioritise the data using a data value sheet; old, legacy data may have different validation rules, eg. before 1/1/2015 student’s didn’t need a USI, but do now.

• Step 6: Migrate a subset of data as a ‘proof of concept’. Validate this through common exports eg. AVETMISS, Selenium scripts can be improved and added to during this phase.

• Step 7: Complete the import, doing manual entries where necessary, then re-validate the data; this should be done regardless of the data method of entry.

Do I need to migrate data?

Migrating data is only useful if there is a need for the information in the future or it has an intrinsic value in the new system. If you can maintain access to the data through other archival methods this may avoid migrating it. There are requirements from different jurisdictions regarding the storage of data. You may need to store files and prove that you have a backup system in place. CourseSales.com allows files to be uploaded and stored against a student file, this management is under our Bulk File Management Storage add-on, where you pay a monthly fee for a particular amount of storage eg. as at 17/3/2015 100 Gb cost $70/month, this price is dropping as our storage provider (Amazon) lower their prices, contact us for an upt-to-date price.

Options available to import and export

Importing data is usually a manual activity by the technical architects of the SMS. For most systems exports (or reports) can be made in a CVS, Excel or flat file format. Exports usually lack the data relationships produced during manual entry and are grouped as student, registration and course data with appropriate mappings. At CourseSales.com we provide you with a spreadsheet, it includes both course data eg. course categories, emails, course modules, course masters etc and documents eg. registrations, files, locations AVETMISS information, contact information.

While we offer direct database additions we recommend you use the Selenium scripts in the first instance as these ensure validation is correctly applied.

Activities & costs

Consider the following major activities to migrate from one SMS to another. Use this pie graph as a starting point then break down the work required to be done.

• Export each data type that requires a separate mapping

• Seek advice for exporting to ease the import process

• Cleansing data from errors and inconsistencies

• Data verification

• Period of running systems in parallel including double entry (if planned)

• Seek advice for importing data

• Import activity

• Manual data relationship adding

• Manual data relationship checking

• Retirement of legacy system

How much data to migrate?

Consider the number of fields and the different mappings for each type of record. You may wish to include mappings for locations, customer notes, financial data, price methods & types, exports, course formats, workflow and other templates.

Determining if data should be migrated

Creating a data value sheet will help you answer the question is this data essential or optional? As an Australian RTO remember to ask yourself:

• Do I have data for the next AVETMISS report period?

• If I don’t need all customer and registration records do I need past, unused course data as well?

Manual data migration

Not all data can be automatically added to a database, manual intervention will be required. Ascertain how much manual work is required during the ‘proof of concept’. It will require expertise in data management and your training organisation. Here are some examples of data requiring manual intervention:

• Course templates eg. payment details/course outlines etc.

• Business rules eg. Customer account relationships

• Email content eg. automated email merges

• Pricing structures or rules

Risks associated with data migration

• Due to the mismatch of data definitions between systems because of new business rules or unclear definitions there is a threat that data or data relationships are lost.

• Due to the relationships usually determined during manual entry that do not exist in an exported format there is a threat that there will be data errors in the new system.

• Due to the system being a new environment the data and business rules may not be fully understood meaning that there is a threat the new system is not configured to match the data resulting in manual entry of additional data.

General tips

• AVETMISS export files are usually not helpful for importing into a new system because the files contain only a subset of the information required.

• Create a project plan to outline activities required to migrate to the new system. Include in the plan the cost and time it will take and what resources/skills are required.

• Ensure data security by having the necessary agreements in place to recognise the sensitive nature of personal and financial data.

• Create a visual diagram of the data flow to help understand where the data comes from, goes to and is change or used.

• Technical skills and business knowledge should be on hand to assist. eg. Project managers, business process experts, business managers, testers, data architects.

• Other affected applications need to be considered eg. accounting, CRM, website, backup system, file storage, learning management system, spreadsheets.

• Do backups before, during and after migration. Confirm you can ‘roll back’ to the legacy system if necessary.

• Retire legacy systems to ensure that the old data is no longer edited after migration which may cause confusion about which data is to be trusted.

• Check data accuracy. The more data, the more manual error correction required. The more records you have the more validation required and likely errors are going to occur.

• Cleanse data before migration, to ensure that errors are reduced, better future reporting, reducing the number or records by removing duplicates and correcting addresses.

Questions:

What if my student/registration data is not complete?

You may find for some reason your data does not include all the necessary fields that might usually be required for AVETMISS submission. For example, some courses may use paper forms, where it is easier to not complete or to forget fields like their Sex or reason for attending the course.

During the import you can change a field’s type to ‘text’ this allows normal text to be entered instead of a drop down or date selection, to be changed back when appropriate. The data you import must use the AVETMISS standard field values that appear in the AVETMISS specification, ie instead of the word ‘Male’ you must enter ’m'. Using the standard AVETMISS values means that changing the field back again results in the correct data being shown. If however you choose to use a value that indicates information has not been supplied, eg @@@@@@@@ to represent the Date of Birth, then that value will not be carried through if you save a document related to the contact. Here is a list of values to enter when information has not been supplied:

The following fields are required but can be submitted with ‘placeholder’ Not specified values. You may not need to enter this however - as CourseSales.com will submit these placeholder values for you.

Details about the table

• Prior educational achievement flag - this is inferred by the selection or not of the list of prior educational achievements, so no field required.

• Disability flag - this is inferred by the selection or not of the list of disabilities, so no field is required.

• Statistical area level 2 identifier - this is for states to define

• *The document referred to is available from NCVER, Edition 2.2 April 2013, AVETMISS 7.0, AVETMISS Data Element Definitions

• *1 Field must not be blank, but can be not specified according to AVETMISS Release 7.0, VET Provider Collection specifications

• *2 Field must not be blank and must match Postcode list, but can be ‘not specified’ if Postcode is @@@@

When you import enrolments you are also importing client data. AVETMISS reports on Units of Competency (UoC) or Modules - this means that often one enrolment is equal to multiple items to be exported for AVETMISS, ie

During the import you may like to specific defaults for the following fields (which are required to submit AVETMISS data):

• Funding Source National eg. Domestic full fee-paying client

• Commencing Program Identifier eg. Unit of competency or module enrolment only

• Outcome Identifier - National eg. Competency achieved/pass

• VET in Schools Flag eg. Not a VET in Schools program

• Client Identifier - need to auto generate in spreadsheet.

Remember that after modifying the fields you republish the registration forms.

Tips when migrating document data

• Any field that CourseSales.com is setup to validate, should be scrutinized in advance where possible

• Strongly consider installing the add-on Power Debugger (linked below) as this will stop the script when an error occurs.

• It is important to validate when a form is submitted succesfully by checking for an element on the successful completion page (this should already be in the script, but worth considering if you are customising the scripts)

• Email addresses in particular may not have been correctly validated, with spaces, wrong domain names etc this will halt the import until it is corrected

• If you are importing AVETMISS or other data that will require a third party to validate the data some areas that you may want to check include:

  • Unique Student Identifier - has it been validated and does the name/date of birth match what is expected?

  • Date of Birth (both format and limitations such as minimum age

  • If importing passwords confirm that the encryption is the same and minimum password requirements are met or deal with these appropriate for resets (we recommend upon migration all passwords are reset with strong minimum requirements)

  • Post code or zip code data matches any external database from a postal service etc.

  • Unique identifiers, ensure that these are in fact unique and valid for the database you may be interfacing to eg. Customer Relationship Management System or Learning Management System

CourseSales.com includes integrated import options, and some non-integrated import options. What follows is how to import using non-integrated import options, used for initial configuration rather than importing existing data from another system. Think of this as a configuration import rather than student/enolment import.

You may have existing configuration data that needs to be migrated into CourseSales.com. For student or enrolment data please see our integrated import options. Keep in mind that usually stduent management systems do not supply the option to export the configuration of the named system. That is, you can export the student data but you can’t export the configuration of workflow nor existing units/qualifications and their related data. However we are here to help - with tools that you can use and advice to assist in the migration effort. If you’d prefer not to do the configuration migration then we are happy to take over that job as well.

CourseSales.com provide tools using the Selenium web automation tool. What this does is use the CourseSales.com web interface to ensure that data integrity is maintained to the same level as if you entered if by hand or customers entered the data as part of their registration. It is efficient, you can see what is happening and it tests the integrity of possible scenarios making your configuration robust for future use. You also have control and the ability to modify the scripts to suit your purposes.