Window Calculation
The Window Calculation transformation component lets you use SQL window functions to analyze a subset of the input datastream. Window functions operate across a "window" of rows relative to the current row, without collapsing them (unlike aggregation).
A window function operates on a group of related rows known as a partition. A partition is usually a logical grouping of rows. You specify how the upper and lower bounds of the partition are determined, expressed as an offset from the current row. Window function results are calculated for each row within each partition, with the calculation taking into account all the rows within the specified offset from the current row.
This is equivalent to an SQL function using the OVER and PARTITION BY clauses.
The available window functions depend on your cloud data warehouse. For more information, read the following:
- Snowflake window functions documentation
- Databricks window functions documentation
- Amazon Redshift window functions documentation
The full list of supported functions is given under the Function property, below.
Note
Window functions share some similarities with aggregate functions, and for some use cases you may find that the Aggregate component will serve your needs better. Bear the following in mind when evaluating which component best suits your use case:
- For an aggregate function, the input is a group of rows from the dataset, and the output is one row (so the aggregated group is collapsed into a single row).
- For a window function, the input is every row within the dataset, and the output is one row per input row.
For example, the SUM aggregate function returns a single total for all of the input rows, whereas the SUM window function returns one total for each input row, calculated from all the rows in the partition.
Use case
Window functions enable a range of advanced analytics such as ranking, time-series processing, and data comparison. Some typical uses for window functions include:
- Calculating cumulative totals across a sequence. For example, cumulative sales per customer. This would use the SUM window function.
- Calculating a moving average across a set of data. For example, to smooth out fluctuations in time-series data. This would use the Average window function.
- Identifying the first or last value in a set of ordered rows. This would use the First Value or Last Value window function.
Properties
Name = string
A human-readable name for the component.
Include Input Columns = boolean
Defines whether the component passes all input columns into the output. The default is Yes.
Partition Data = dual listbox
Select the columns that will define how the input data is partitioned. The window calculation will be performed on each partition.
Ordering Within Partitions = column editor
Select the columns that will be used to sort the partitioned data. For each column, select the sort order: Asc (sort ascending), Desc (sort descending), Nulls First (sort null values first), or Nulls Last (sort null values last). You can select multiple columns to create a complex sort. You can drag the selected columns to reorder the sort level if required.
Functions = column editor
Select a function to be performed on the rows contained in the window:
Refer to the list of supported functions below. Multiple functions can be selected. For each function, select the Input Column that the function will act on, and the Output Column that the result will be written to.
Supported Window functions:
- Any Value: Snowflake only. Returns some value of the expression from the group. For full details, read the documentation for Snowflake.
- Approximate Count Distinct: Snowflake only. Uses HyperLogLog to return an approximation of the distinct cardinality of the input. For full details, read the documentation for Snowflake.
- Array Aggregate Distinct: Snowflake only. Returns the input values, pivoted into an array. For full details, read the documentation for Snowflake.
- Average: Returns the average (arithmetic mean) of the input column values in the window. For full details, read the following documentation:
- Bit AND Aggregate: Snowflake only. Returns the bitwise AND value of all non-Null numeric records in a group. For full details, read the documentation for Snowflake.
- Bit OR Aggregate: Snowflake only. Returns the bitwise OR value of all non-Null numeric records in a group. For full details, read the documentation for Snowflake.
- Bit XOR Aggregate: Snowflake only. Returns the bitwise XOR value of all non-Null numeric records in a group. For full details, read the documentation for Snowflake.
- Conditional Change Event: Snowflake only. Returns a window event number for each row where the value of an argument is different from the value of the argument in the previous row. For full details, read the documentation for Snowflake.
- Conditional True Event: Snowflake only. Returns a window event number for each row within a window partition based on the result of a boolean argument. For full details, read the documentation for Snowflake.
- Count: Returns a count of the non-Null values for the specified field. For full details, read the following documentation:
- First Value: Given an ordered set of rows, returns the specified column value with respect to the first row in the window frame. For full details, read the following documentation:
- Hash Aggregate: Snowflake only. Returns an aggregate signed 64-bit hash value over the (unordered) set of input rows. For full details, read the documentation for Snowflake.
- Kurtosis: Snowflake only. Returns the population excess kurtosis of non-Null records. For full details, read the documentation for Snowflake.
- Last Value: Given an ordered set of rows, returns the specified column value with respect to the last row in the window frame. For full details, read the following documentation:
- List Aggregate: Snowflake and Redshift only. Returns the concatenated input values, separated by a delimiter string. Redshift currently doesn't support ordering within partitions for the List Aggregate function, therefore that option won't be applied to the results if selected. For more information, read the following documentation:
- List Aggregate Distinct: Snowflake only. Returns the concatenated input values, separated by a delimiter string. Duplicate values are eliminated before concatenating. For more information, read the Snowflake documentation.
- Maximum: Returns the maximum of the input expression values. The MAX function works with numeric values and ignores Null values. For full details, read the following documentation:
- Median: Snowflake and Redshift only. Calculate the median value for the range of values in a window or partition. Null values in the range are ignored. Redshift currently doesn't support ordering within partitions for the Median function, therefore that option won't be applied to the results if selected. For full details, read the following documentation:
- Minimum: Returns the minimum of the input expression values. The MIN function works with numeric values and ignores Null values. For full details, read the following documentation:
- Population Variance: Returns the sample variance of a set of numeric columns. For full details, read the following documentation:
- Sample Variance: Returns the sample variance of a set of numeric columns. For full details, read the following documentation:
- Standard Deviation: Returns the standard deviation of a set of numeric values. For full details, read the following documentation:
- Standard Deviation Population: Returns the population standard deviation of a set of numeric values. For full details, read the following documentation:
- Sum: Returns the sum of the input column in the window. For full details, read the following documentation:
Lower Bound = drop-down
This property is visible after Ordering Within Partitions is set. Select which row of the partition the window calculation will start on. Options are:
- unbounded preceding: The window starts at the first row of the partition.
- current row: The window starts at the current row.
- offset preceding: The window starts a number of rows (offset) before the current row. This requires you to set the Lower Bound Offset property.
Upper Bound = drop-down
This property is visible after Ordering Within Partitions is set. Select which row of the partition the window calculation will end on. Options are:
- unbounded following: The window ends at the last row of the partition.
- current row: The window ends at the current row.
- offset following: The window ends a number of rows (offset) after the current row. This requires you to set the Upper Bound Offset property.
Lower Bound Offset = integer
If the Lower Bound property is set to offset preceding, enter the number of rows before the current row that the window will start on.
Upper Bound Offset = integer
If the Upper Bound property is set to offset following, enter the number of rows after the current row that the window will end on.