S3 Manifest Builder
Build and upload a manifest file which describes a set of files (S3 objects) to load based upon a pattern using a regular expression.
If the pattern is simply a prefix, this is not required as the S3 Load can use a prefix directly. However, for more complicated file-naming schemes, use this component to build a manifest file first, and then use an S3 Load with the manifest option and a path pointing to the generated manifest file.
Name = string
A human-readable name for the component.
Source S3 Path = string
The S3 bucket (and optionally the path within it) to search.
Manifest Path = string
The S3 location to write the manifest file to.
Manifest File name = string
The file name for the manifest.
Filter Regular Expression = string
This is a regular expression. All names found from Source S3 Path are tested against this expression, and if they match they are included in the manifest.
This requires an exact match, so if you are searching for text anywhere within the name you will need to include wildcards before and after the text.
Mandatory = drop-down
Yes writes the Mandatory option into the manifest, meaning the subsequent S3 load will fail if the file is not present. No makes the file optional, and no error will be raised if the file is missing.
This component makes the following values available to export into variables:
|Matched Objects||The number of objects matching the filter pattern and written to the Manifest.|
|Snowflake||Delta Lake on Databricks||Amazon Redshift||Google BigQuery||Azure Synapse Analytics|