S3 Load Generator (Delta Lake)
Overview
S3 Load Generator is a wizard that helps users load delimited data from public objects in an S3 Bucket (Amazon Simple Storage Service).
Unlike common components, the Load Generator does not appear as a standalone component when dragged onto the job canvas. Instead, the Load Generator takes the form of a wizard that lets you load and view files on the fly, altering load component properties and observing their effects without the need for a separate Transformation Job. The generator can also guess the Schema of a table, relieving much of the end user's work.
Please Note
This component requires working AWS Credentials with "read" access to the bucket that contains the source data's file(s). This is easily achieved by attaching an IAM role to the instance when launching Matillion ETL for Delta Lake; however, it can also be managed by editing an Environment. See the example at the bottom of this article.
Furthermore, Matillion requires use of a policy that contains the s3:ListBucket action, such as the policy provided in the Managing Credentials documentation.
The following section explains how to configure the S3 Load Generator in Matillion ETL for Delta Lake:
Configuring the S3 Load Generator Wizard
1. Begin by loading Matillion ETL for Delta Lake, and create an Orchestration job in the Project.
_0.1.png)
Create an Orchestration job
2. In the Components tab, search for the S3 Load Generator, and drag it on to the canvas.
_0.2.png)
S3 Load Generator component
Please Note
The three-page S3 Load Generator wizard will immediately open in the Get Sample page, after the component has been dropped onto the canvas.
3. In the Amazon S3 Storage field, use to select a file from an existing S3 bucket.
Please Note
This file must be delimited (including .csv) and the user must have permission to access the file.
4. In the Compression field use the drop-down menu provided. If the file is compressed, the correct compression method must be selected; supported compression methods are gzip and BZip2.
5. In the Row Limit field, select the number of rows to be returned from the sample file, then click Get Sample.
Please Note
The wizard will load the intended file's data and attempt to guess the schema. The raw data will be displayed in the sample box, situated underneath the aforementioned fields.
6. Click Next to progress to the next page of the wizard.
_0.3.png)
Get Sample
7. In the Guess Schema page, the input file's data will be automatically displayed in the section at the top of the dialog. You might want to scroll through the list to view the entirety of the data.
8. Column data is displayed in the lower-right panel, and is available for editing. By default, columns are named "Column1", "Column2", and so on. For the purpose of this document, "Column1" and "Column2" will be called something more descriptive, like States and State Codes, respectively.
9. Schema Configuration properties are displayed in the lower-left panel. Modify the fields relevant to the file type. The available properties are detailed in the next section of the document, see Properties.
10. Click Next to progress to the final page of the wizard.
Please Note
Clicking Guess Schema will order the S3 Load Generator to attempt to guess as many of these properties as possible. However, all of the properties are available for manual editing by the user, as described above.
_1.png)
Guess Schema
11. On the Validation page, the input file's data will be displayed in the panel at the top of the dialog. To view the resulting table's output, click Test at the bottom-left hand side.
Please Note
Running a test will create the requested table on the Delta Lake cluster, and show a sample for the user to inspect.
12. To modify the output data, click Back in the centre of the dialog, to return to previous steps within the wizard, and make property and column alterations.
13. Click Create & Run to complete the wizard.
_2.1.png)
Validation
The wizard will close, return users to the job interface, and create two linked components; Create/Replace Table and Load. Each component is parameterised by the S3 Load Generator, and can be run as a job by linking to a Start component.
_8.1.png)
Job canvas
Properties
| Property | Setting | Description |
|---|---|---|
| For more information on all the settings in this component, see the Delta Lake COPY syntax for more information. | ||
| Table Name | Text | The descriptive name for the table. |
| Field Delimiter | Text | The Field Delimiter is a character that separates fields in an input file. The default is a Comma (,). A [TAB] character can be specified as "\\ ". |
| Date Format | Text | Defaults to 'auto' - this can be used to manually specify a date format. |
| Time Stamp Format | Text | Defaults to 'auto' - defines the format of timestamp values in the data files to be loaded. |
| Null As | Text | This option replaces the specified string with null in the output table. Use this if your data has a particular representation of missing data. |
| Ignore Header Rows | Checkbox | Row at the top of the file to ignore. |