Updating and migrating overview
Matillion regularly releases updates to Matillion ETL that include security enhancements, new features, new components, quality-of-life improvements, bug fixes, and more. The Matillion ETL client will alert you via the Notices tab when a new update is available.
Matillion ETL integrates with over 100 external services, many of which are also updated regularly. Improvements and fixes included in the latest updates often risk causing issues with older versions of systems and/or services, therefore there is both a need to update and a risk in doing so.
We recommend updating immediately; however, this isn't mandatory. All software updates are inherently risky, and for operational reasons you may elect to delay or refuse an upgrade; however, you should be aware that Matillion ETL releases have a finite lifecycle and will no longer be supported after a specified date. Read Matillion ETL support lifecycle for details of this.
Matillion ETL offers two main methods of updating your instance, migration and in-place update. In most scenarios, the recommended approach is migration, which is considered the safest method. It's your choice which method you use, however. Some pros and cons of each method are listed below.
The instructions for updating and migration assume you will be moving to the newest release version. If you want to update to a specific previous version, read Updating to a specific release.
Note
If your Matillion ETL instance runs in an AWS high availability (HA) cluster, special considerations apply when updating or migrating the instance. For such instances, see Updating a Matillion HA Cluster.
Migration
With the migration method, data is copied from an existing instance (the source instance) and migrated to another instance (the target instance), where the target instance is a new installation of the new version.
Pros
- The risk of causing disruption to a production instance is vastly reduced. By migrating to a new instance you can verify that your workloads run as expected on the new version of Matillion ETL, and only enable the scheduling on the new instance when you're happy (and then remove/retire the existing infrastructure).
- You can test the new instance over a period of time while the old software is still running production jobs.
Cons
- Some elements, such as system modifications, aren't automatically migrated, as detailed in Migration. Some of these items can be managed via the API, but not all.
- Requires additional effort from users to maintain a pre-production environment and perform testing (although this is good practice).
- On rare occasions, Matillion ETL components may be deprecated and replaced by one with a newer driver. This is known as "versioned" components in the release notes. In these cases, a migration update will render the old components (and thus any jobs they are a part of) inoperable as old drivers are not included in new instances. It is up to the user to replace the deprecated components with their updated counterparts.
Recommendations
- You should manually maintain a record of any configuration changes made to your instances, so that they can be manually recreated on a new instance. Where possible, those changes should be scripted so they're repeatable.
For details of how to use this method, read Migration.
In-place update
With the in-place update method, the updates are applied to the existing instance.
Pros
- An update can be performed via the Matillion ETL user interface (UI) by a non-specialist.
- The RPM update is performed in the background and the service restarted automatically.
- Doesn't require additional effort to set up a new instance and manage the connectivity between the current and new instance.
Cons
- While Matillion takes every precaution to ensure there are no problems, it's possible that an issue could arise which causes your instance to stop working. If this happens, there is no "undo" or revert option, which means you will need to restore from a backup.
- In-place updates can't always be used. For example, moving from Amazon Linux to CentOS is an operating system change that can never be offered as an in-place update and must instead be resolved via a migration.|
- Only the Matillion ETL application is updated. Postgres, Tomcat, and OS updates/upgrades aren't applied.
Recommendations
- You should take backups (from within Matillion ETL or using another method) of the virtual machine (VM) and ensure that you have tested that a working system can be restored from that backup, so you can recreate the instance if the update causes issues.
For details of how to use this method, read In-place update.
Updating server utilities and dependencies
We recommend that you apply critical security updates using the command yum update --security.
Applying only the latest security updates in this way shouldn't cause Matillion ETL features or functionality to break. However, the Matillion ETL application isn't tested with every minor update to all dependencies, so we don't suggest updating all packages via yum update. If you do so, it's at your own risk.
Prior to updating the Matillion ETL application or other yum packages, or applying any custom configuration, we always strongly suggest creating a root volume backup and being prepared to restore it to your virtual machine in the event of any loss of service or impact to Matillion ETL features and functionality.