Matillion ETL for Redshift release notes

!!! note Starting from version 1.63 and onwards, we will be deprecating and removing the visual lineage tab from Matillion ETL. For more information, and to find out what the future of lineage in Matillion ETL looks like, read [Matillion ETL Supports Data Lineage Tools with Metadata API Integration](https://www.matillion.com/resources/blog/matillion-etl-supports-data-lineage-tools-with-metadata-api-integrations) from the Matillion blog.

Resources

Releases Overview is an index for all resources related to Matillion ETL releases.

Release Notes Advisories contains a library of notices that users should consider if they are running certain versions of Matillion ETL.

Release Notes Archive is an index of release notes for past versions of Matillion ETL. Each version's page contains a major release and any adjunct interim releases. Each page provides a changelog for all cloud data warehouses.

Matillion ETL for Redshift 1.62.8

(13th April 2022)

Bug fixes

Fixed an issue where clustered sessions would timeout and the Matillion ETL canvas would cease updating.
Fixed an issue where the AWS class WebIdentityCredentialsProvider would fail silently.

Matillion ETL for Redshift 1.62.7

(4th April 2022)

New Features & Improvements

New LinkedIn Ads Query component. The LinkedIn Ads Query component uses the LinkedIn Ads API to retrieve data and load it into a table. Certain data sources are no longer available in LinkedIn Query because they are available in this component instead. Please read the component documentation for more information.
Linkedin Query has been deprecated and replaced with a new version, still called LinkedIn Query.
- Existing jobs using the "old" LinkedIn Query component should continue to work as expected.
- The "old" LinkedIn Query component is highlighted as deprecated in the extant jobs.
- Dragging the LinkedIn Query component onto the canvas will use the "new" version of the component.
- A number of data sources are no longer available in LinkedIn Query but are available instead in LinkedIn Ads Query. These data sources are listed in the component reference documentation.
As of 23rd February 2022, the v11.0 Marketing API is no longer available. As such, Matillion ETL uses v12.0 as the default Marketing API for Facebook Ads Query. Read Available Marketing API Versions in Changelog for more information.
A new Twitter Ads Query component is available. This component uses the Twitter Ads API to retrieve data and load it into a table.
The Google AdWords Query component has been deprecated. This deprecation is in line with Google sunsetting the AdWords API on 27th April, 2022. On this date, any Google AdWords jobs in your Matillion ETL instance will fail. Read our tech note for more information.
A new Google Ads Query component is available, replacing the Google AdWords Query component.
The Rename component has been updated and now includes an Include Input Columns property.
The Dynamics 365 Business Central Query component has been updated. A new property has been added: Authentication. Read Dynamics 365 Business Central Query Authentication guide to learn how to create and authorize an OAuth entry. Additionally, the following properties have been removed:
- Organization URL (now required in the OAuth entry setup)
- User
- Access Key
The Salesforce Output component has been updated:
- A new On Warnings parameter lets you choose to continue loading files after a record errors, or otherwise fail the job.
- A new Records Per Ingest Job lets you specify an integer for the number of records to load per bulk job.
Data staging components now have an Export Sample feature, located in the Sample tab. Clicking Export Sample will download the sample data as a CSV file.
Enhancements to the Migrate wizard, including:
- Page 1 (Connection) will display an error message in the Test Connection box if the source and target instances are the same instance.
- Removed an unnecessary whitespace line from page 2 (Users).
- Improved the wording on page 3 (Project Content) regarding projects that have Git enabled.
- Fixed an issue where the wording related to projects that have Git enabled would load after the transition from page 2 to page 3 of the wizard.
- Projects with Git enabled are clearly identifiable on page 3 (Project Content). The Git icon is displayed next to a project that has Git enabled.
- Projects are filtered based on an access control list. Users must be able to access the project on both the source and the target instance, otherwise the project cannot be migrated.
- Hover tooltips have been added to the options on page 5 (Migration Options).
- Page 5 (Migration Options) will display a warning if no resources have been selected for migration.
- Removed a block of unnecessary whitespace from the Migration completed successfully dialog that can be found at the end of the Migrate wizard.
The navigation menu of Matillion ETL now displays the current project group, project, version, and environment.

Bug fixes

Fixed an issue where, when cloning a remote repository in the Matillion ETL Git integration, the first commit would not checkout successfully.
Fixed an issue where cloning a remote repository using SSH would fail.
Fixed an issue where the Salesforce Output component was behaving unexpectedly and while used in Bulk mode, jobs were not updating in status to Closed in Salesforce.

Driver upgrades

Couchbase API driver updated. 21.0.7867.0 → 21.0.8011.0
Dynamics 365 Business Central API driver updated. 19.0.7045.0 → 21.0.8011.0
Facebook Ads API driver updated. 19.0.7045.0 → 21.0.8011.0
Gmail API driver updated. 17.0.6634.0 → 21.0.8062.0
Google Ads API driver updated. 20.0.7647.0 → 21.0.8011.0
LinkedIn API driver updated. 19.0.7369.0 → 21.0.8011.0
Oracle Eloqua API driver updated. 18.0.6919.0 → 21.0.8077.0
SharePoint API driver updated. 19.0.7194.0 → 21.0.8080.0

Matillion ETL for Redshift 1.61.7

(13th April 2022)

Bug fixes

Fixed an issue where clustered sessions would timeout and the Matillion ETL canvas would cease updating.

Matillion ETL for Redshift 1.61.6

(18th February 2022)

Updated Matillion ETL from Apache Tomcat version 8.5.72 to version 8.5.75 after a vulnerability was discovered that affected versions 8.5.55 – 8.5.73.
Note: This issue is only exploitable when Tomcat is configured to persist sessions using the FileStore. This is not enabled by default in Matillion ETL.

Matillion ETL for Redshift 1.61.5

(15th February 2022)

New Features & Improvements

A new environment variable type is available: the Data Structure. This variable type accepts a JSON data format. Components that produce semi-structured data, such as Extract components, will have a new export variable, which will generate the structure based on the endpoint used in the Extract component.
Migrate has been updated and now allows users to migrate their Git integration projects. A Matillion ETL instance must be initialized with Git on at least one project for the Migrate wizard to ask users whether to migrate projects that use Git in the Project Content page of the wizard.
The Git Merge Conflict Resolution feature has been updated and now includes three selectable views: Conflicts view (the default), Differences view and Full view. This view selector can be accessed in the overlay's upper-left area. The upper-right area shows the current number of conflicts.
User credentials can be migrated when migrating a Git project.
The Sample tab now shows NULL values when sampling data.
The Migrate wizard has been updated to include a completion message. If not all resources migrated successfully, the message will indicate errors that the user should review.
Updated the First/Last component to include an Ignore Nulls setting. By default, this setting is set to No.
Extract components now have a Type parameter to allow users to select between an External or Standard schema. An external schema will stage the semi-structured data to an external RedShift Spectrum table, and a standard schema will stage the semi-structured data to an internal RedShift SUPER table.

Bug fixes

Fixed an issue where the Resolve Conflicts overlay did not behave as expected when resized.
Fixed an issue where creating Azure credentials in Manage Credentials would not complete.
Fixed an issue where the action of switching to a different Matillion ETL was not behaving as expected.
Fixed an issue where the Job Description value could cause a Git merge to fail despite not being highlighted as a conflict.
Fixed an issue where selecting a Matillion ETL project would lead to an infinite "Loading..." status.
Fixed an issue where the Merge overlay would not populate the Merge To field.
Fixed an issue where the password manager failed to authenticate with certain key:value pairs.
Updated the Jira driver to fix an issue where the IssueCustomFieldOptions data source was not returning any date.
Fixed an issue where the cascading functionality of the Resolve Conflicts overlay would occasionally freeze.
Fixed an issue where passing a default timestamp or time type value in the Create Table component would result in an SQL compilation error and an error at runtime.
Fixed an issue where running several concurrent jobs with many versions could result in a premature out-of-memory condition.
Removed the restriction on the total number of user accounts searchable via LDAP/External user configuration.
Fixed an issue where unhandled exceptions may leak resource handles, leading to user interface freezes.

Vulnerability fixes/updates

Updated Postgres driver to 42.3.3 (CVE-2020-13692, CVE-2018-10936)
Updated commons-beanutils to 1.9.4 (CVE-2019-10086)
Updated Google tink to 1.6.1 (CVE-2020-8929)
Updated Apache HTTP client to 4.5.13 (CVE-2020-13956)
Updated commons-codec to 1.15 (CWE-200)
Updated commons-io to 2.11.0 (CVE-2021-29425)
Updated protobuf-java to 3.17.3 (CVE-2015-5237)
Updated junit to 4.13.2 (CVE-2020-15250)

Driver upgrades

Google BigQuery API driver updated (19.0.7285.0) → (21.0.7930.0)
Jira API driver updated (20.0.7790.0) → (21.0.7954.0)
QuickBooks Online API driver updated (19.0.7354.0) → (21.0.7930.0)
Xero API driver updated (19.0.7332.0) → (21.0.7930.0)

Matillion ETL for Redshift 1.59.12 AND 1.60.2

(Updated 8th February 2022)

If you initially upgraded to 1.60.1, please update to 1.60.2 or higher to receive a bug fix.

Version 1.60.1. was adversely affected by a bug that was causing the following components to fail to locate their driver:

Google BigQuery
Jira Query
QuickBooks Online Query
Xero Query

Versions 1.60.2 and higher include a bugfix for this behaviour.

These two updates have been released together and contain specific, sequential package-based changes that will allow Matillion ETL products to be updated to versions 1.60 and higher. Full details of the changes, and the available upgrade options, can be found at Matillion ETL — Upgrading to version 1.60 and above.
As with all Matillion ETL updates, we encourage all customers to consume the updates as soon as they are available.

Matillion ETL for Redshift 1.59.11

(20th January 2022)

Updated the Apache Spark JDBC driver to version 2.6.22.1040 provided by Databricks. This driver version does not contain any versions of Log4j 1.x.

Matillion ETL for Redshift 1.59.10

(12th January 2022)

Bug Fixes

Removed all Log4j versions below version 2.17 from Matillion ETL.

Matillion ETL does still include sb_slf4j-log4j. This includes a version of log4j 1.x, and is brought in by the Apache Spark JDBC driver used by Matillion ETL for Delta Lake on Databricks. Databricks is in the process of removing it completely from the driver. Once it is ready, we will release a new version of Matillion ETL with the driver upgrade.

Please rest assured that:

The Log4j 1.x inside the driver is unused by the driver itself, and cannot be used by anything else in Matillion ETL.
The driver is unused by all products except Matillion ETL for Delta Lake on Databricks.
Log4j 1.x has no known vulnerabilities unless the customer manually adds XML configurations onto the Matillion ETL server to allow exploits.

Matillion ETL for Redshift 1.59.9

(24th December 2021)

Bug Fixes

The Apache Spark JDBC driver has been updated in relation the to Log4j vulnerability. Please see the Databricks release notes for more information.

Matillion ETL for Redshift 1.59.8

(16th December 2021)

Bug Fixes

Fixed an issue where AWS credentials were not always decrypted in the root location.

Matillion ETL for Redshift 1.59.7

(15th December 2021)

New Features & Improvements

The merge conflict resolution feature in Matillion ETL's Git integration has been updated. Both the Resolve Conflicts dialog and the Merge dialog have received updates:
- The Choice column is now enabled in Resolve Conflicts, and users can select Ours or Theirs on a granular, per-conflict level to settle a merge conflict.
- Conflicts are now highlighted red for added visibility in Resolve Conflicts. This red highlight will disappear after a change has been committed, not simply when a selection has been made in the Choice column.
- The Base column is no longer visible by default in Resolve Conflicts. This is to enhance readability when deciding between Ours and Theirs.
- Descriptions of jobs are also included in the Resolve Conflicts dialog.
- Since merge conflicts are now settled at a cherry-picked level of specificity in the Resolve Conflicts dialog, the Merge dialog no longer offers users the single choice of Ours or Theirs. Instead, the Merge dialog acts as a staging area before and after resolving individual conflicts.
The Migrate feature is now accessed by clicking Admin → Migrate in the upper-right of the user interface (previously, Migrate was located in the Project menu).
Migrate has also been updated to include user configuration including users, user groups, usernames, passwords, roles, and permissions as a migration resource.
Users can now select whether scheduled jobs that missed their trigger time are run as soon as possible by selecting the Ignore misfires checkbox in the Create Schedule dialog.
Improved the functionality of Matillion ETL when operating behind a proxy.
Connection options are now added to the Add Data Source dialog for SAP NetWeaver Query.
The Task API now has endpoints for filtering task history by endtime.
- GET/by/end/after/date/<date>
- GET/by/end/after/date/<date>/time/<time>
- GET/by/end/range/date/<startDate>/time/<startTime>/to/date/<endDate>/time/<endTime>
- The date format should be [yyyy-mm-dd].
- The time format should be [HH:mm].

Bug Fixes

Fixed an issue where large numbers of scheduled jobs could have a misleading queue order in the tasks view.
Fixed an issue where users were unable to transfer data via the Data Transfer component through a proxy server.
Maintainability improvements to the billing service codebase.
Fixed an issue where renaming a component that is joined to the Join component was causing the join to fail.

Driver Updates

Google Ad Manager API driver updated (20.0.7737.0) → 21.0.7930)
Marketo API driver updated (20.0.7767.0 → 21.0.7930)
ServiceNow API driver updated (18.0.6857.0 → 21.0.7930)
Snowflake JDBC driver updated (3.13.1 → 3.13.11)

Matillion ETL for Redshift 1.58.6

(14th March 2022)

Fixed an issue where unhandled exceptions may leak resource handles, leading to user interface freezes.

Matillion ETL for Redshift 1.58.5

(15th December 2021)

Bug Fixes

Maintainability improvements to the billing service codebase

Matillion ETL for Redshift 1.58.4

(10th November 2021)

New Features & Improvements

Apache Tomcat version updated to 8.5.72
A new Git Merge Conflict Resolution tool empowers users to observe in depth the narrative differences between "Ours", "Base", and "Theirs" before making merge conflict resolution decisions.
The new References lets users see where a job or Shared Job is being used or referenced by another job. The References tab lists all referents and includes a URL path so that users can quickly navigate to a job. To learn more, readJob References.
The References tab also applies to components.
Renaming an orchestration job or transformation job now includes the functionality to select whether to also rename all other references to that job. To learn more, read Job Reference Renaming.
The Oracle Output component has been updated. The component now has a Port property, where users must specify a port number for their Oracle endpoint. The default port number is 1521.
The File Iterator component has been updated. The Variables property now offers Last Modified as a selectable file attribute. When selected, this last modified date is formatted as ISO8601, with a UTC indicator. For example, 2021-01-04T10:45:15.123Z
The Table Iterator component has been updated. Users can now interact with a Mode property and select either Basic or Advanced. Subsequent component properties will be available depending on the selection. Similarly to how Query components work, Basic Mode requires users to select settings in additional properties; whereas Advanced Mode offers a single Query property, where users can manually add SQL.
All five iterator components have been updated to include a Stop on Condition property when running in Sequential concurrency (and not Concurrent). If Stop on Condition is set to Yes, additional properties related to composing the condition(s) will be available. Please refer to the component documentation for help with these properties.
The Google Analytics Query component has been updated with an API property. This property has two selectable options, each of which will determine the available data sources that users can pull data from. Data models are linked below:
1. Google Universal Analytics
2. Google Analytics 4
Increased the scope of the matillion-sudo file to allow for universal builds.

Bug Fixes

Fixed an issue where the Stripe Query component would report success and 0 rows staged, despite using an expired API key.
Fixed an issue with the Oracle Output component where if a user added a primary key, the temporary table name was longer than the 30 allowed characters. Now, the unique ID in the table name is denoted as six characters out of those 30.
Fixed an issue where the UseSandbox connection option was not pulled from the OAuth in the Salesforce Query component.
The Facebook Ads Query component now uses the v11 API by default (replacing v10). For users who want to use v12, set the APIVersion connection option to 12; the version must also be set to 12 in your Facebook application. If you do choose to set the API to v12, you will need to reconfigure any Facebook Ads OAuth entries, as v12 requests additional scopes.
Fixed an issue where attempting to use Windows File Share resulted in an exception.

Driver Updates

Dynamics 365 Sales API driver updated (20.0.7662 → 21.0.7930)
DynamoDB API driver updated (20.0.7662 → 21.0.7934)
Facebook Ads API driver updated (21.0.7810 → 21.0.7930)
Google Analytics API driver updated (20.0.7662 → 21.0.7898)
OData API driver updated (20.0.7662 → 21.0.7958)
Stripe API driver updated (20.0.7654 → 21.0.7954)

Matillion ETL for Redshift 1.56.10

(20th October 2021)

Fixed an issue where the Oracle Output component would fail without an error message when the output was a numeric data type and the data was NULL.

Matillion ETL for Redshift 1.56.9

(6th October 2021)

New Features & Improvements

The new Send Email component empowers users to send emails from within an Orchestration Job.
Matillion ETL Secrets Manager now supports two additional secret managers:
1. Azure Key Vault
2. GCP Secret Manager
Increased the maximum number of environments in Matillion ETL to 999. This only applies to Matillion ETL instances created via the Hub.

Bug Fixes

Fixed an issue where user defined credentials were malformed when accessing S3 buckets.
Fixed an issue where users were unable to create the true maximum number of environments in Matillion ETL instances created via the Hub.
Fixed an issue where file paths were not persisting as expected in components with a file browser property (such as S3 Object Prefix in S3 Load; Google Storage URL Location in Cloud Storage Load, Azure Storage Location in Azure Blob Storage Load; and the Source URL and Target URL properties in Data Transfer) and added validation when the / key is pressed by the user, to dynamically update a filepath as it is being typed.
Fixed an issue where the webserver would not restart cleanly if a login took place during the shutdown sequence.
Fixed an issue with Microsoft SQL Server Output where temporary tables were being dropped when an update occurred for drivers prior to 2016.

Driver Updates

Couchbase API driver updated (19.0.7354.0 TO 21.0.7867.0)
HubSpot API driver updated (20.0.7732.0 TO 21.0.7907.0)
Square API driver updated (20.0.7697.0 TO 21.0.7867.0)
Twilio API driver updated (18.0.6904.0 TO 21.0.7892.0)

Matillion ETL for Redshift 1.55.5

(9th August 2021)

New Features

Matillion ETL Secrets Manager lets you create a secret manager that will directly reference passwords, API tokens, keys, and secrets stored in third party secrets manager platforms. Currently, this feature only supports AWS Secrets Manager. Users can create new secret managers via the dialog, and see a list of their existing secret managers.
Matillion ETL server admins can now control the session timeout expiration of their Matillion ETL instances. The default value is 180 minutes, but server admins can reduce or increase this as their organizational needs require.
Query components using an Amazon S3 bucket as the staging location now offer an additional property, Use Accelerated Endpoint, which harnesses the capabilities of Amazon S3 Transfer Acceleration.
Please note: Amazon S3 Transfer Acceleration may not always be quicker. Use this speed comparison tool to estimate whether S3 Transfer Acceleration will benefit your use case.

Improvements

The Migrate tool has been redesigned into a four-page wizard. Users must now validate their connection from their source to their target instances. Validation errors will therefore be discovered earlier. The four-page wizard has been designed to improve the user friendliness and clarity of the migration process within Matillion ETL.

Driver Updates

Dynamics 365 Sales API Driver updated (18.0.6949.0 to 20.0.7662.0)
ElasticSearch API Driver updated (19.0.7354.0 to 20.0.7662.0)
Excel API Driver updated (19.0.7234.0 to 21.0.7829.0)
Facebook API Driver updated (20.0.7752.0 to 21.0.7810.0)
Google Analytics API Driver updated (19.0.7354.0 to 20.0.7662.0)
Google Search API Driver updated (17.0.6505.0 to 20.0.7662.0)
Salesforce API Driver updated (20.0.7662.0 to 21.0.7807.0)
SurveyMonkey API Driver updated (19.0.7121.0 to 20.0.7662.0)

Bug Fixes

Fixed an issue in Matillion ETL for Redshift where the Audit Log was failing to load.
Fixed an issue where the Updates Available prompt was appearing when no update was available.
Improved Matillion ETL deployability by automatically configuring environment variables using platform metadata.

Matillion ETL for Redshift 1.54.10

(5th July 2021)

Fixed an issue where the Audit Log was not tracking actions for Orchestration Jobs and Transformation Jobs imported/exported in Matillion ETL.
Fixed an issue where multiple collaboration bubbles* were shown if the Matillion ETL browser tab did not close cleanly.
Fixed an issue where upgrading from certain Matillion ETL CentOS versions to the "latest" available version would prevent a Tomcat restart.
Enabled Matillion ETL to access Hub billing features when a proxy is configured.
Fixed an issue where forward slash / characters used in Header parameters were causing the Configure Query Connector wizard (Manage Query Profiles) to fail.

*A collaboration bubble is the square with an initial inside it in the upper-right of the UI, visible when you have at least one collaborator on your Matillion ETL instance. See Project Collaboration for more information.

Matillion ETL for Redshift 1.54.9

(17th June 2021)

This update contains bug fixes and improvements.

If you are on 1.46.10 or older, please do not update your Matillion ETL instance to 1.54.9. Instead, please launch a new instance of Matillion ETL. This recommendation is also found at Tech Note 15th June 2021.

Matillion ETL for Redshift 1.54.7

(8th June 2021)

New Features

Apache Tomcat version updated to 8.5.65
Matillion ETL's API v1 - Metadata endpoint lets users determine where data in Matillion ETL jobs has originated from. Please consult our documentation for a full explanation of this endpoint. Please also note that this feature is an Enterprise Mode only feature.
Facebook Ads query component enables users to query the Facebook Ads API to retrieve data and load it into a table. Note: please use this component for any queries on Facebook Ads data sources—these data sources are no longer available in the Facebook Query component.
The Extract Nested Data component is now available for Matillion ETL for Redshift. This component lets users flatten nested data into rows, typically in the form of key:value pairs (such as JSON dictionary) and using those keys as column names.
Use Matillion ETL's API Connector Wizard if you need help deciding whether to use an API Query Profile or an API Extract Profile.
Users can now add column-level metadata comments in the Create Table component. Navigate to the Table Metadata property when the component is clicked, and add any comments.
If you're a Matillion ETL server administrator and you wish to control session timeout expiration, you can now set an environment variable to ensure that the timeout period is in line with your organization's policies.

Improvements

Users can configure the Python Script component to be unavailable on your Matillion ETL instance if desired—for example, for security reasons.
Users can configure the Bash Script component to be unavailable on your Matillion ETL instance if desired—for example, for security reasons.
SharePoint Query component now has two data models:
1. REST.
2. SOAP.
The Facebook Query component has been deprecated and replaced with a new version. The new component has the same name (Facebook Query), the same configuration properties, and the same property defaults as the deprecated component. The deprecated version of the component will persist for any of your existing Orchestration Jobs.
New NetSuite Query replaces the original component. From version 1.54 of Matillion ETL, the NetSuite Query component only accepts OAuth as an authentication method. The previous version of the component will persist for your existing Orchestration Jobs. Users can cycle back to a previous version of the component's documentation if they wish. The new version of the component does not include the following properties:
- Authentication Method
- Account ID
- User
- Password
Names of components will now wrap to a new line when a certain width is exceeded on the Matillion ETL canvas.
Guidance has been added to the help text in the Matillion ETL Updates dialog, explaining the preferred and safest way to update Matillion ETL.
Users can manually decided whether to track manual or scheduled (or both) job runs using Manage Error Reporting.
Users can now navigate directly to the task history of an Orchestration or Transformation Job. The string History* will be appended, enabling users to predictably determine the task history URL. When switching back to the job, the URL will drop the appended string.

Bug Fixes and Driver Updates

BingAds API Driver Updated (19.0.7216.0 to 20.0.7662.0)
Bing Search API Driver Updated (19.0.7354.0 to 20.0.7662.0)
Facebook API Driver Updated (19.0.7285.0 to 20.0.7752.0)
Google Ads Manager API Driver Updated (20.0.7662.0 to 20.0.7737.0)
Hubspot API Driver Updated (19.0.7354.0 to 20.0.7732.0)
Jira API Driver Updated (20.0.7646.0 to 20.0.7790.0)
Mailchimp API Driver Updated (19.0.7285.0 to 20.0.7760.0)
Marketo API Driver Updated (18.0.7018.0 to 20.0.7767.0)
Netsuite API Driver Updated (20.0.7513.0 to 20.0.7662.0)
OData API Driver Updated (19.0.7052.0 to 20.0.7662.0)
Zoho CRM API Driver Updated (19.0.7424.0 to 20.0.7662.0)