The ETL process creates some transient staging tables during a pipeline run that are deleted after a run, regardless of success. Tables are also created permanently to support the pipeline process, including the target table that contains your source data.
Properties
Property
Duration
Purpose
Target Table
Lifetime of a pipeline
Stores the flattened data from the agent for your source.
Metadata Table
Lifetime of a pipeline
Create/Append to a metadata table (named <target_table>_VERSION_METADATA) which holds the schema for each version supplied by the agent into cloud storage. This facilitates schema drift management. This table also contains metadata regarding data typing for each column, for each source, for a version. This can be used to translate data types if required.
External Table
Lifetime of a pipeline
A Snowflake external table (named by the user as per the shared job parameter of the same name). This table is a view of all files in cloud storage for the pipeline.
Property
Duration
Purpose
Target Table
Lifetime of a pipeline
Stores the flattened data from the agent for your source.
Metadata Table
Lifetime of a pipeline
Create/Append to a metadata table (named <target_table>_VERSION_METADATA) which holds the schema for each version supplied by the agent into cloud storage. This facilitates schema drift management. This table also contains metadata regarding data typing for each column, for each source, for a version. This can be used to translate data types if required.
Stage Tables (N)
Lifetime of a pipeline
Stage Tables (named <stage-prefix>, <source_database>, <source_schema>, <source_table>, <source_version>). These are external tables that provide a view of all files in cloud storage for the pipeline for a given version.
Property
Duration
Purpose
Target Table
Lifetime of a pipeline
Stores the flattened data from the agent for your source.
Metadata Table
Lifetime of a pipeline
Create/Append to a metadata table (named <target_table>_VERSION_METADATA) which holds the schema for each version supplied by the agent into cloud storage. This facilitates schema drift management. This table also contains metadata regarding data typing for each column, for each source, for a version. This can be used to translate data types if required.
Stage Table
Temprorary for a pipeline run
A temporary stage table (named <target_table>_<uuid>). Copies in all the relevant files for all the versions, which are cleaned up after each run.
Property
Duration
Purpose
Target Table
Lifetime of a pipeline
A native table. Stores the flattened data from the agent for your source. The table is named: <target_prefix>_<source_database>_<source_schema>_<source_table>.
Metadata Table
Lifetime of a pipeline
A BigQuery external table. Create/Append to a metadata table which holds the schema for each version supplied by the agent into cloud storage. This facilitates schema drift management. This table also contains metadata regarding data typing for each column, for each source, for a version. This can be used to translate data types if required. The table is named: <stage_prefix><source_database>_<source_schema>_<source_table>_<version_metadata>.
External Table
Lifetime of a pipeline
A BigQuery external table. This table is a view of all files in cloud storage for the pipeline. The table is named <stage_prefix><source_database>_<source_schema>_<source_table>_<source_version>.