S3 Load Generator (Snowflake)

Overview

S3 Load Generator is a wizard that helps you load delimited data from public objects in an S3 Bucket (Amazon Simple Storage Service).

Unlike common components, the Load Generator does not appear as a standalone component when dragged onto the job canvas. Instead, the Load Generator takes the form of a wizard that lets you load and view files on the fly, altering load component properties and observing their effects without the need for a separate Transformation job. The generator can also guess the schema of a table, relieving much of your work.

Please Note

This component requires working AWS Credentials with "read" access to the bucket that contains the source data's file(s). This is easily achieved by attaching an IAM role to the instance when launching Matillion ETL for Snowflake. However, it can also be managed by editing an Environment. See the example at the bottom of this article.

Furthermore, Matillion requires use of a policy that contains the s3:ListBucket action, such as the policy provided in the Managing Credentials documentation.

The following section explains how to configure the S3 Load Generator in Matillion ETL for Snowflake:

---

Configuring the S3 Load Generator Wizard

1. Begin by loading Matillion ETL for Snowflake, and create an Orchestration Job in the Project.

Create an Orchestration job

2. In the Components tab, search for the S3 Load Generator, and drag it on to the canvas.

S3 Load Generator component

Please Note

The three-page S3 Load Generator wizard will immediately open in the Get Sample page, after the component has been dropped onto the canvas.

3. In the Amazon S3 Storage field, use to select a file from an existing S3 bucket.

Please Note

This file must be delimited (including .csv) and you must have permission to access the file.

4. In the Compression field use the drop-down menu provided. If the file is compressed, the correct compression method must be selected; supported compression methods are gzip and BZip2.

5. In the Row Limit field, select the number of rows to be returned from the sample file, then click Get Sample.

Please Note

The wizard will load the intended file's data and attempt to guess the schema. The raw data will be displayed in the sample box, situated underneath the aforementioned fields.

6. Click Next to progress to the next page of the wizard.

Get Sample

7. In the Guess Schema page, the input file's data will be automatically displayed in the section at the top of the dialog. You may want to scroll through the list to view the entirety of the data.

8. Column data is displayed in the lower-right panel, and is available for editing. By default, columns are named "Column1", "Column2", and so on. For the purpose of this document, "Column1" and "Column2" will be called something more descriptive, like States and State Codes, respectively.

9. Schema Configuration properties are displayed in the lower-left panel. Modify the fields relevant to the file type. The available properties are detailed in the next section of the document, see Properties.

10. Click Next to progress to the final page of the wizard.

Please Note

Clicking Guess Schema will order the S3 Load Generator to attempt to guess as many of these properties as possible. However, all of the properties are available for you to manually edit, as described above.

Guess Schema

11. On the Validation page, the input file's data will be displayed in the panel at the top of the dialog. To view the resulting table's output, click Test, situated to the bottom-left hand side.

Please Note

Running a test will create the requested table on the Snowflake cluster, and show a sample for you to inspect.

12. To modify the output data, click Back, situated at the button in the centre of the dialog, to return to previous steps within the wizard, and make property and column alterations.

13. Click Create & Run, situated to the right of the Back button, to complete the wizard.

Validation

The wizard will close, returning you to the job interface, and create two linked components; Create/Replace Table and Load. Each component is parameterized by the S3 Load Generator, and can be run as a job by linking to a Start component.

Job canvas

---

Properties

Property	Setting	Description
For more information on all the settings in this component, see the Snowflake COPY syntax for more information.
Table Name	Text	The descriptive name for the table.
Field Delimiter	Text	The Field Delimiter is a character that separates fields in an input file. The default is a Comma (,). A [TAB] character can be specified as "\\ ".
Field Optionally Enclosed By	Text	Character used to enclose strings. Value can be: NONE single quote character ('); or a double quote character ("). When a field contains this character, escape it using the same character.
Ignore Header Rows	Text	The number of rows at the top of the file to ignore - defaults to 0.
Date Format	Text	Defaults to 'auto' - this can be used to manually specify a date format.
Time Format	Text	Defaults to 'auto' - this can be used to manually specify a time format.
Time Stamp Format	Text	Defaults to 'auto' - defines the format of timestamp values in the data files to be loaded.
Escape Unenclosed	Text	Single character string used as the escape character for unenclosed field values only. Default is a backslash.
Escape	Text	Single character used as the escape character for any field values.
Null As	Text	This option replaces the specified string with null in the output table. Use this if your data has a particular representation of missing data.
Trim Blanks	Checkbox	If this is enabled, Matillion ETL removes trailing and leading whitespace from the input data.
Error on Column Count	Checkbox	If enabled, a parsing error is generated when the number of delimited columns (fields) in the input data file does not match the number of columns in the corresponding table.
Empty As Null	Checkbox	If this is set, empty columns in the input file will become NULL.

---

Video Example