Fixed Flow
The Fixed Flow transformation component lets you generate a table of data for use downstream in the pipeline. This data can be hard-coded into the component, or it can be generated from variables at runtime.
When you configure the Fixed Flow component, you first define the columns that the table will have, and then you create as many rows of data as you require in the table. The output of the Fixed Flow component is a table containing the specified data. The output can be joined to any number of downstream components that can then use that data.
Use case
The Fixed Flow component gives you a versatile tool for introducing data into your Data Productivity Cloud pipelines. Some common uses for this include:
- Testing and debugging downstream components. During pipeline development, you may want to test downstream components (like filters, joins, and transformations) without using live data. The Fixed Flow will allow you to simulate a dataset with a few rows and columns to run your tests before connecting to actual data sources.
- Creating static reference or lookup data. Your downstream components may need reference data that is static or rarely changes. For example, a lookup table of currency codes and country names. You could create this lookup table in a Fixed Flow component, and then it would be available to your pipeline without needing to connect to a separate external data source.
Properties
Name = string
A human-readable name for the component.
Columns = column editor
Name: The name of the input column.
Type: Select from:
- VARCHAR: This type is suitable for strings and text.
- NUMBER: This type is suitable for whole-number types (no decimals).
- FLOAT: This type is suitable for numeric types, with or without decimals.
- BOOLEAN: This type is suitable for data that is either 'true' or 'false'.
- DATE: This type is suitable for dates without times.
- TIMESTAMP: This type is a timestamp left unformatted (exists as Unix/Epoch Time).
- TIME: This type is a formatted time object without date.
- VARIANT: This is a flexible type that can be used for any purpose.
Size: For Text types, this is the maximum length. This is a limit on the number of bytes, not characters. For Numeric types, this is the total number of digits allowed, whether before or after the decimal point.
Scale: Scale is the number of digits to the right of the decimal point in a Float. For example, the number 123.45 has a scale of 2.
Values = column editor
A dynamic table of values with one column for each column created above.
Name = string
A human-readable name for the component.
Columns = column editor
- Name: The name of the input column.
-
Type:
- Integer: This type is a whole number (not a fractional number) that can be positive, negative, or zero.
- Number: This type is suitable for numeric types, with or without decimals.
- Float: For this type, values are approximate numeric values with fractional components.
- Text: This type contains alphanumeric and character strings.
- Timestamp: This type represents values comprising values of fields year, month, day, hour, minute, and second, with the session local time-zone. The timestamp value represents an absolute point in time.
- Date: This type is suitable for dates without times.
- Boolean: This type is suitable for data that is either "true" or "false".
- Binary: This type is byte sequence values.
-
Size: For text types, this is the maximum length. This is a limit on the number of bytes, not characters. For numeric types, this is the total number of digits allowed, whether before or after the decimal point.
-
Scale: Scale is the number of digits to the right of the decimal point in a float. For example, the number 123.45 has a scale of 2.
Values = column editor
A dynamic table of values with one column for each column created above.
Name = string
A human-readable name for the component.
Columns = column editor
- Name: The name of the input column.
-
Type:
- Text: This type can hold any type of data, subject to a maximum size.
- Integer: This type is suitable for whole-number types (no decimals).
- Numeric: This type is suitable for numeric types, with or without decimals.
- Real: This type is suitable for real-number types (no decimals).
- Double Precision: This type is suitable for double-precision numeric types, with or without decimals.
- Boolean: This type is suitable for data that is either 'true' or 'false'.
- Date: This type is suitable for dates without times.
- DateTime: This type is suitable for dates, times, or timestamps (both date and time).
- SUPER: Use the SUPER data type to store semi-structured data or documents as values.
-
Size: For Text types, this is the maximum length. This is a limit on the number of bytes, not characters. For Numeric types, this is the total number of digits allowed, whether before or after the decimal point.
-
Scale: Scale is the number of digits to the right of the decimal point in a Float. For example, the number 123.45 has a scale of 2.
Values = column editor
A dynamic table of values with one column for each column created above.
Date, DateTime, Time, and Timestamp validation
A variety of Date, Time, DateTime, and Timestamp values can be used. Some of the most common values accepted are listed below. The list of examples is not exhaustive, but covers many of the most commonly accepted formats.
Date values:
- 13/01/2021
- 01/13/2021
- 2021/1/13
- 13/JAN/2021
- 2021
- JAN 13 2021
- 13 January, 2021
- January 13, 2021
Time values:
- 23:12
- 23:12:59
- 23:12:59.123456 (milli/micro/nano are all supported)
Time values with offset:
- 23:12 +01:00
- 23:12:59 +01:00
- 23:12:59.123456 +01:00 (milli/micro/nano are all supported)
DateTime and Timestamp values:
- 13-01-2021 23:12:59.123456 +01:00
- 2021-01-13 23:12:59.123456 +01:00
Note
- To avoid confusion between day and month values, the number 13 is used as the value for day in these examples.
- The backslash character can be replaced by a dash, underscore, full stop, or colon.
- Dates cannot contain time values.
- Times cannot contain date values.
- The DateTime/Timestamp values can be just the date, but not just the time.
- The DateTime/Timestamp values accepted combine certain date patterns with all the time patterns, split by either a whitespace character or a T character.
- There are a number of exceptions that are accepted, such as BC dates, negative dates, and also month values greater than 12. None of these are listed here, and we advise caution when using these values.