Detect Changes
The Detect Changes transformation component lets you identify the differences between two datasets. Use this component to compare two separate tables that contain similar data, and then insert a new column that indicates whether the data in each row has been inserted, deleted, changed, or left unchanged.
Any rows with key columns that contain NULL values will be ignored. NULL comparison values are considered equal.
Use case
This component can be used to keep your data up-to-date efficiently, without needing to reload all your data repeatedly. For example, you can use it to:
- Track changes to financial or healthcare data for compliance reasons.
- Identify rows that have been changed, making it easier to see only the most recent updates.
Properties
Name = string
A human-readable name for the component.
Master Table = drop-down
Select a master table from the two inputs. This table is the one treated as default in the comparison with the second table.
Match Keys = dual listbox
Select the key columns to join the two tables on. These columns must appear in both tables. NULL values are ignored.
Compare Columns = dual listbox
Select the columns that will be checked for changes. Just like the keys, these columns must appear in both tables; however, the two lists shouldn't overlap.
Output Column Mapping = column editor (optional)
- Input Column: Select input columns to map to output names. Defaults are provided automatically but can be changed.
- Output Column: Name each output columns that the input columns will map to.
Click the Add all button to automatically populate the Input Column and Output Column with all the columns selected in the Compare Columns property. You can then selectively edit the list if required, but if you click Add all again it will return the list to the default fully populated state, removing any changes you have made.
Indicator Column = string
Enter a name for the new column in the output. By default, this column is named "Indicator". This column contains an indicator that shows the status of each record:
- C the record has been changed.
- D the record has been deleted.
- I the record is identical.
- N the record is new.
Switching the master table in the Master Table property will reverse the meaning of new (N) and deleted (D).
Indicators
Indicators are single-letter codes that indicate what the state of a row is regarding Detect Changes. The following indicators are used:
- C Changed: the record is present in both tables, with different values, but with the same ID.
- D Deleted: the record is present in the master table, but not in the second table.
- I Identical: the same record is present in both tables with no changes.
- N New: the record isn't present in the master table, but is present in the second table.