Cloud Storage Load Generator (BigQuery)

Overview

The Cloud Storage Load Generator is a wizard that helps users load delimited data from public objects in Google Cloud Storage.

Unlike common components, the Cloud Storage Load Generator does not appear as a standalone component when added to the Matillion ETL job canvas. Instead, the load generator takes the form of a wizard that allows users to load and view files on the fly, altering load component properties and seeing their effects without the need for a separate Transformation Job. The Load Generator can also guess the schema of a table, relieving much of the end user's work.

Please Note

This component requires working Google Cloud Platform credentials with "read" access to the bucket that contains the source data's file(s). You can do this by attaching an IAM role to the instance when launching Matillion ETL for BigQuery; however, it can also be managed by editing a Matillion ETL environment.
Matillion requires use of a policy that contains the gs:ListBucket action, such as the policy provided in Manage Credentials.
This guide presumes that you have launched a Matillion ETL for BigQuery instance. To begin with this step, read Launching Matillion ETL for BigQuery - GCP.

The following section explains how to configure the Cloud Storage Load Generator in Matillion ETL for BigQuery.

Configuring the Cloud Storage Load Generator wizard

1. Create an Orchestration Job in your chosen Project.

2. In the Components tab, search for the Cloud Storage Load Generator, and drag it on to the canvas.

Please Note

The three-page Cloud Storage Load Generator wizard will open in the Get Sample page after the component has been added to the canvas.

3. In the Google Cloud Storage field, use to select a file from an existing Google Cloud Storage bucket.

Please Note

This file must be delimited (including .csv) and the user must have permission to access the file.

4. In the Compression field, use the drop-down menu provided. If the file is compressed, the correct compression method must be selected; the supported compression method is gzip.

5. In the Encoding field use the drop-down menu provided. Supported data encoding types are UTF-8 (default) and ISO-8859-1.

6. In the Row Limit field, select the number of rows to be returned from the sample file, then click Get Sample.

Please Note

The wizard will load the intended file's data and attempt to guess the schema. The raw data will be displayed in the sample box, situated underneath the aforementioned fields.

7. Click Next to progress to the next page of the wizard.

8. In the Guess Schema page, the input file's data will be automatically displayed in the section at the top of the dialog. You may want to scroll through the list to view the entirety of the data.

9. Column data is displayed in the lower-right panel, and is available for editing.

10. Schema Configuration properties are displayed in the lower-left panel. Modify the fields relevant to the file type. The available properties are detailed in the next section of the document, see Properties.

11. Click Next to progress to the final page of the wizard.

Please Note

Clicking Guess Schema will order the Cloud Storage Load Generator to attempt to guess as many of these properties as possible. However, all of the properties are available for manual editing by the user, as described above.

12. On the Validation page, the input file's data will be displayed in the panel at the top of the dialog. To view the resulting table's output, click Test.

Please Note

Running a test will create the requested table on the BigQuery cluster, and show a sample for the user to inspect.

13. To modify the output data, click Back to return to previous steps within the wizard, and make property and column alterations.

14. Click Create & Run to complete the wizard.

The wizard will close, return users to the job canvas, and create two linked components: Create Table and Cloud Storage Load. Both of these components have been parameterized by the Cloud Storage Load Generator, and can be run as a job by linking to a Start component.

Properties

Property	Setting	Description
For more information on all the settings in this component, see the Loading CSV data from Cloud Storage .
Table Name	Text	A human-readable name for the table.
Max Error	Text	Set the maximum number of individual parsing errors that cause the whole load to fail. Values up to this will be substitued as null values.
Delimiter	Text	Add a delimiting character. The default is a comma ,. A TAB character can be specified using the backslash key and the TAB key.
Quote Characters	Text	Set the quote character to use to enclose records. The default value is a double quote. To indicate no quote character at all, use an empty string.
Skip Header Rows	Text	An integer number of Header rows at the top the file to skip.
Ignore Unknown	Checkbox	Ticked: Accept rows that contain values that do not match the schema. Unknown values are ignored. Note: Will ignore extra values at the end of a line for CSV files. Unticked: Omit any rows with invalid values.
Quoted New Lines	Checkbox	Tick to allow a CSV value to contain a new line character when the value is encased in quotation marks. Otherwise, a new line character, regardless of quotations, is always considered a new row.
Jagged Rows	Checkbox	Tick to ensure missing values are treated as null but accepted. Otherwise, rows with missing data are treated as bad records. Note: A bad record will count toward the Max Error count.