SAP ODP Extract
The SAP ODP Extract component enables Matillion ETL users to connect directly through SAP ODP to access available data sources in SAPI and ABAP CDS views.
This component includes many connection options that you may need to set. Read SAP ODP Connection Options for more information about these advanced connection options.
Given the nature of this component, pages for FAQs and for troubleshooting SAP ODP Extract are available and will be continuously updated.
Note
This component is generally available as of version 1.71.
Add SAP libraries
Using the SAP ODP Extract component requires three additional files—two from SAP, and one from Matillion. These files must be added to the location /usr/share/java/sap/, which you can access via SSH.
Matillion files
Download the file matillion-jco-wrapper.jar, using the following command:
wget https://matillion-docs.s3.eu-west-1.amazonaws.com/Attachments/matillion-jco-wrapper.jar
SAP files
- Log in to SAP and access the JCo download software. If necessary, select the Tools and Services page to display the download page.
- Download the following files and ensure they are named as below:
libsapjco3.so— This is a "JCo" connection library.sapjco3.jar— This is a Java wrapper.
- Download the most recent version of the SAP JCo 3.x for Linux.
Next steps
- Create a directory
sapunder the directory/usr/share/java. - Once you have downloaded the three files, add them to
/usr/share/java/sap/. -
Restart Tomcat:
systemctl restart tomcat
Warning
- Without these additional files, you may get errors such as: "SAP JCo library not found: verify the correct jar file is present."
- With an incorrect password, or other authentication problem, you may get errors such as: "Initialization of repository destination INERP_JCO_DESTINATION_NAME3 failed."
Properties
Name = string
A human-readable name for the component.
Connection = drop-down
The connection method. Choose either Direct or Load Balancer. The below parameters will change depending on the connection method.
Host = string
SAP ABAP application server host DNS. This is the host name of your target system. Direct connection only.
System Number = integer
The system number of the SAP ABAP app server. Your system number should have been provided at the point of installation. For ECC, it is usually 10. For S/4H, it is usually 00. Direct connection only.
Message Server Host = string
SAP message server. This is a DNS or IP address. Load balancer only.
Message Server Service = string
An SAP message server port number, for example 3600. To resolve service names sapmsXXX, a lookup in etc/services is performed by the "network" layer of the operating system. If you're using port numbers instead of symbolic service names, no look-ups will be performed and no additional entries are needed. Load balancer only.
Group Server = string
A group of SAP application servers. This parameter is used to identify the application servers used (the "logon group"). This parameter is optional for connecting via a load balancer and is only required if used on your SAP system. The default value is a single SPACE.
Authentication = drop-down
Select the authentication method. Options include Username/Password and SNC. The following parameters will change depending on the connection method.
Username = string
Your SAP username. Only needed if Authentication is Username/Password.
Password = string
Your SAP login password. You can store the password in the component, or use Manage Passwords to create a masked entry for the password. Using Manage Passwords is recommended. Only needed if Authentication is SNC.
SNC Partner Name = string
SNC name of the communication partner server. Must be an LDAP name. For example, p:CD=SID, O=ACompany, C=EN. Find the app server SNC name in the profile parameter snc/identity/as on SAP NetWeaver application server for ABAP. This parameter is required for SNC. The default length is 256 characters. Only needed if Authentication is SNC.
SNC Library Path = string
An SNC library path. This is the path to the SNC library on the client machine (SAP GUI). The default setting is the system-defined library as defined in the environment variable SNC_LIB. Destination service uses the property snc/gssapi_lib on SAP NetWeaver AS for Java. This parameter is optional and only used if Authentication is SNC.
SNC MyName = string
Specify the SNC name. Must be an LDAP name. For example, p:CD=SID, O=ACompany, C=EN. Destination services require it be set in the property snc/identity/as on SAP NetWeaver application servers for Java. The default length is 256 characters. This parameter is optional, but it is recommended to set it to ensure that the correct SNC name is used for the connection if Authentication is SNC. When left blank and SNC is enabled, JCo derives the value.
SNC Quality of Protection = integer
SNC "quality of protection". This parameter is optional. The values available are: 1 for authentication only. 2 for authentication and integrity protection. 3 for authentication, integrity, and privacy protection (encryption). 8 for the global default configuration (equivalent to 3). 9 for maximum protection. Only needed if Authentication is SNC.
Client = integer
SAP client. The default is 100. From SAP help: "A SAP client is defined as a self-contained commercial, organizational, and technical unit within an SAP system. All business data within a client are protected from other clients." Read more about this at Customer Data and System Data.
Language = string
Defaults to en. If not defined, the SAP system returns to JCo the default user language, and JCo will use that default value. Language codes adhere to ISO 639-1 codes.
Connection Options = column editor
Additional connection options for SAP ODP. Add a new row with +; remove a row with -; or click Add All. For each connection option, select the Parameter from the drop-down list and enter its Value as a string. Documentation for each of the available connection options can be found here.
Context = drop down
List of contexts available in the SAP system for the subscriber type SAP_BW. The context acts as a "data provider" for the data source. In SAP, a data provider permits configuration of data for extraction when targeting a specific use.
Search = string
Search string used to identify and reduce the quantity of data sources returned by SAP. For example, in this parameter you may wish to use the string TEXT. The maximum search string length is 30 characters.
Data Source = drop-down
In the context of SAP ODP, a data source is an "extractor". In SAP, an extractor is a data extraction through a specified context. A data source can be the result of aggregation of one or more tables or views in SAP. Data sources are listed where a match with the Search parameter has been ascertained. A semantic character is also provided. Additionally, a contextual description is given where a data source has been provided with description metadata in SAP.
The data sources available show a concatenation of:
- The data source technical name.
- A semantic (the SAP type of data source).
- The SAP data source description, in the language specified at login.
Semantics glossary:
H = Hierarchy F = Transaction Data/Facts P = Master Data/Attributes T = Texts V = View
Data Selection = dual listbox
Columns to include in the extraction. Columns will also have a contextual description where applicable.
Data Source filter = column editor
- Input Column: Select an input column. The available input columns vary depending upon the data source.
- Qualifier:
- Is: Compares the column to the value using the comparator.
- Not: Reverses the effect of the comparison, so "Equals" becomes "Not equals", "Less than" becomes "Greater than or equal to", etc.
- Comparator: Choose a method of comparing the column to the value. Possible comparators include: "Equal to", "Greater than", "Less than", "Greater than or equal to", "Less than or equal to", "Like", "Null". "Equal to" can match exact strings and numeric values, while other comparators, such as "Greater than" and "Less than", will work only with numerics. The "Like" operator allows the wildcard character [%] to be used at the start and end of a string value to match a column. The Null operator matches only null values, ignoring whatever the value is set to. Not all data sources support all comparators, meaning that it is likely that only a subset of the above comparators will be available to choose from.
- Value: The value to be compared.
Load Type = drop-down
Select a load type to manage the quantity of data sent via the network. This is especially useful when dealing with large RFC table parameters. Available options depend on your chosen data source. Possible types are F (full load), D (delta load), and R (recovery, available only if the data source supports deltas). The default is F.
When full load is selected, the whole table is sent back to the caller. When delta is selected, only appended, deleted, and updated table rows are transferred back to the caller.
Max Package Size = integer
Byte integer representing the package size of the data in SAP. The default setting is 5,000,000 bytes.
Max Package Size is based on the size of the package in SAP, measured in compressed bytes. This size may need to be tweaked depending on the content of the package and the size constraint of the cloud data platform.
Delta Initialisation = string
Only used if Load Type is D (delta). To only extract changes (creates, modifications, and deletes) from now onwards, enter X. Leave blank (the default) to also return historical changes.
Recovery Pointer = string
Only available if the data source supports deltas, and the Load Type is R (recovery).
SAP ODP allows you to repeat the extraction of data that may have been lost or corrupted due to an interruption of service, to restore and guarantee the data integrity of the extracted data. A recovery pointer is set by SAP ODP at a specific point-in-time in the delta queue, and you can use this to recover any data change recorded from that point onward. When the recovery pointer is set and the job repeated, all the data changes that happened from that recovery pointer onward are extracted, while all data changes that happened before that recovery pointer are not extracted.
You must obtain the value of the recovery pointer from the Replication Pointers table in SAP. After a data extraction that you believe to have been interrupted, refresh the SAP Replication Pointers table and make a note of the pointer. The pointer is a string of digits in the following format: 20220921082113.000021000. This is a year-month-day-hour-minute-second timestamp, and must be in this exact format; any format other than this will lead to an error. This string should be entered in the Recovery Pointer property to tell the component where in the delta queue to begin the recovery run.
Subscriber Name = string
The subscriber is the consumer of the data. Subscriber name is assigned by the user to allow the identification of which app is consuming the data. The maximum string size is 32 characters.
Subscriber Process = string
The identifier for the extraction process of the subscriber. The maximum string size is 64 characters.
Subscriber Run = string
The run ID of the subscriber. For example, ${dt.now()} The maximum string size is 64 characters. This parameter is optional.
Location = storage location path
Select a storage location. The cloud storage buckets available for selection depend on the credentials you have set up in Manage Credentials.
Warehouse = drop-down
Select the Snowflake warehouse. The special value, [Environment Default], will use the warehouse defined in the Matillion ETL environment. For more information, read Virtual Warehouses.
Database = drop-down
Select the Snowflake database. The special value, [Environment Default], will use the database defined in the Matillion ETL environment. For more information, read Databases, Tables, and Views.
Schema = drop-down
Select the Snowflake schema. The special value, [Environment Default], will use the schema defined in the Matillion ETL environment. For more information, read Database, Schema, and Share DDL.
Target Table = string
Provide a new table name.
Warning
This table will be recreated on each run of the job and drop any existing table of the same name.
Name = string
A human-readable name for the component.
Connection = drop-down
The connection method. Choose either Direct or Load Balancer. The below parameters will change depending on the connection method.
Host = string
SAP ABAP application server host DNS. This is the host name of your target system. Direct connection only.
System Number = integer
The system number of the SAP ABAP app server. Your system number should have been provided at the point of installation. For ECC, it is usually 10. For S/4H, it is usually 00. Direct connection only.
Message Server Host = string
SAP message server. This is a DNS or IP address. Load balancer only.
Message Server Service = string
An SAP message server port number, for example 3600. To resolve service names sapmsXXX, a lookup in etc/services is performed by the "network" layer of the operating system. If you're using port numbers instead of symbolic service names, no look-ups will be performed and no additional entries are needed. Load balancer only.
Group Server = string
A group of SAP application servers. This parameter is used to identify the application servers used (the "logon group"). This parameter is optional for connecting via a load balancer and is only required if used on your SAP system. The default value is a single SPACE.
Authentication = drop-down
Select the authentication method. Options include Username/Password and SNC. The following parameters will change depending on the connection method.
Username = string
Your SAP username. Only needed if Authentication is Username/Password.
Password = string
Your SAP login password. You can store the password in the component, or use Manage Passwords to create a masked entry for the password. Using Manage Passwords is recommended. Only needed if Authentication is SNC.
SNC Partner Name = string
SNC name of the communication partner server. Must be an LDAP name. For example, p:CD=SID, O=ACompany, C=EN. Find the app server SNC name in the profile parameter snc/identity/as on SAP NetWeaver application server for ABAP. This parameter is required for SNC. The default length is 256 characters. Only needed if Authentication is SNC.
SNC Library Path = string
An SNC library path. This is the path to the SNC library on the client machine (SAP GUI). The default setting is the system-defined library as defined in the environment variable SNC_LIB. Destination service uses the property snc/gssapi_lib on SAP NetWeaver AS for Java. This parameter is optional and only used if Authentication is SNC.
SNC MyName = string
Specify the SNC name. Must be an LDAP name. For example, p:CD=SID, O=ACompany, C=EN. Destination services require it be set in the property snc/identity/as on SAP NetWeaver application servers for Java. The default length is 256 characters. This parameter is optional, but it is recommended to set it to ensure that the correct SNC name is used for the connection if Authentication is SNC. When left blank and SNC is enabled, JCo derives the value.
SNC Quality of Protection = integer
SNC "quality of protection". This parameter is optional. The values available are: 1 for authentication only. 2 for authentication and integrity protection. 3 for authentication, integrity, and privacy protection (encryption). 8 for the global default configuration (equivalent to 3). 9 for maximum protection. Only needed if Authentication is SNC.
Client = integer
SAP client. The default is 100. From SAP help: "A SAP client is defined as a self-contained commercial, organizational, and technical unit within an SAP system. All business data within a client are protected from other clients." Read more about this at Customer Data and System Data.
Language = string
Defaults to en. If not defined, the SAP system returns to JCo the default user language, and JCo will use that default value. Language codes adhere to ISO 639-1 codes.
Connection Options = column editor
Additional connection options for SAP ODP. Add a new row with +; remove a row with -; or click Add All. For each connection option, select the Parameter from the drop-down list and enter its Value as a string. Documentation for each of the available connection options can be found here.
Context = drop down
List of contexts available in the SAP system for the subscriber type SAP_BW. The context acts as a "data provider" for the data source. In SAP, a data provider permits configuration of data for extraction when targeting a specific use.
Search = string
Search string used to identify and reduce the quantity of data sources returned by SAP. For example, in this parameter you may wish to use the string TEXT. The maximum search string length is 30 characters.
Data Source = drop-down
In the context of SAP ODP, a data source is an "extractor". In SAP, an extractor is a data extraction through a specified context. A data source can be the result of aggregation of one or more tables or views in SAP. Data sources are listed where a match with the Search parameter has been ascertained. A semantic character is also provided. Additionally, a contextual description is given where a data source has been provided with description metadata in SAP.
The data sources available show a concatenation of:
- The data source technical name.
- A semantic (the SAP type of data source).
- The SAP data source description, in the language specified at login.
Semantics glossary:
H = Hierarchy F = Transaction Data/Facts P = Master Data/Attributes T = Texts V = View
Data Selection = dual listbox
Columns to include in the extraction. Columns will also have a contextual description where applicable.
Data Source filter = column editor
- Input Column: Select an input column. The available input columns vary depending upon the data source.
- Qualifier:
- Is: Compares the column to the value using the comparator.
- Not: Reverses the effect of the comparison, so "Equals" becomes "Not equals", "Less than" becomes "Greater than or equal to", etc.
- Comparator: Choose a method of comparing the column to the value. Possible comparators include: "Equal to", "Greater than", "Less than", "Greater than or equal to", "Less than or equal to", "Like", "Null". "Equal to" can match exact strings and numeric values, while other comparators, such as "Greater than" and "Less than", will work only with numerics. The "Like" operator allows the wildcard character [%] to be used at the start and end of a string value to match a column. The Null operator matches only null values, ignoring whatever the value is set to. Not all data sources support all comparators, meaning that it is likely that only a subset of the above comparators will be available to choose from.
- Value: The value to be compared.
Load Type = drop-down
Select a load type to manage the quantity of data sent via the network. This is especially useful when dealing with large RFC table parameters. Available options depend on your chosen data source. Possible types are F (full load), D (delta load), and R (recovery, available only if the data source supports deltas). The default is F.
When full load is selected, the whole table is sent back to the caller. When delta is selected, only appended, deleted, and updated table rows are transferred back to the caller.
Max Package Size = integer
Byte integer representing the package size of the data in SAP. The default setting is 5,000,000 bytes.
Max Package Size is based on the size of the package in SAP, measured in compressed bytes. This size may need to be tweaked depending on the content of the package and the size constraint of the cloud data platform.
Delta Initialisation = string
Only used if Load Type is D (delta). To only extract changes (creates, modifications, and deletes) from now onwards, enter X. Leave blank (the default) to also return historical changes.
Recovery Pointer = string
Only available if the data source supports deltas, and the Load Type is R (recovery).
SAP ODP allows you to repeat the extraction of data that may have been lost or corrupted due to an interruption of service, to restore and guarantee the data integrity of the extracted data. A recovery pointer is set by SAP ODP at a specific point-in-time in the delta queue, and you can use this to recover any data change recorded from that point onward. When the recovery pointer is set and the job repeated, all the data changes that happened from that recovery pointer onward are extracted, while all data changes that happened before that recovery pointer are not extracted.
You must obtain the value of the recovery pointer from the Replication Pointers table in SAP. After a data extraction that you believe to have been interrupted, refresh the SAP Replication Pointers table and make a note of the pointer. The pointer is a string of digits in the following format: 20220921082113.000021000. This is a year-month-day-hour-minute-second timestamp, and must be in this exact format; any format other than this will lead to an error. This string should be entered in the Recovery Pointer property to tell the component where in the delta queue to begin the recovery run.
Subscriber Name = string
The subscriber is the consumer of the data. Subscriber name is assigned by the user to allow the identification of which app is consuming the data. The maximum string size is 32 characters.
Subscriber Process = string
The identifier for the extraction process of the subscriber. The maximum string size is 64 characters.
Subscriber Run = string
The run ID of the subscriber. For example, ${dt.now()} The maximum string size is 64 characters. This parameter is optional.
Location = storage location path
Select a storage location. The cloud storage buckets available for selection depend on the credentials you have set up in Manage Credentials.
Target Table = string
Provide a new table name.
Warning
This table will be recreated on each run of the job and drop any existing table of the same name.
Type = drop-down
Choose between using a standard table or an external table.
- Standard: The data will be staged on an S3 bucket before being loaded into a table.
- External: The data will be put into an S3 bucket and referenced by an external table.
Standard Schema = drop-down
Select the table schema. The special value, [Environment Default], will use the schema defined in the environment. For more information on using multiple schemas, read Schemas.
A standard schema is required if the Type property is set to "Standard".
External Schema = drop-down
Select the table schema. The special value, [Environment Default], will use the schema defined in the environment. For more information on using multiple schemas, read Schemas.
An external schema is required if the Type property is set to "External".
Name = string
A human-readable name for the component.
Connection = drop-down
The connection method. Choose either Direct or Load Balancer. The below parameters will change depending on the connection method.
Host = string
SAP ABAP application server host DNS. This is the host name of your target system. Direct connection only.
System Number = integer
The system number of the SAP ABAP app server. Your system number should have been provided at the point of installation. For ECC, it is usually 10. For S/4H, it is usually 00. Direct connection only.
Message Server Host = string
SAP message server. This is a DNS or IP address. Load balancer only.
Message Server Service = string
An SAP message server port number, for example 3600. To resolve service names sapmsXXX, a lookup in etc/services is performed by the "network" layer of the operating system. If you're using port numbers instead of symbolic service names, no look-ups will be performed and no additional entries are needed. Load balancer only.
Group Server = string
A group of SAP application servers. This parameter is used to identify the application servers used (the "logon group"). This parameter is optional for connecting via a load balancer and is only required if used on your SAP system. The default value is a single SPACE.
Authentication = drop-down
Select the authentication method. Options include Username/Password and SNC. The following parameters will change depending on the connection method.
Username = string
Your SAP username. Only needed if Authentication is Username/Password.
Password = string
Your SAP login password. You can store the password in the component, or use Manage Passwords to create a masked entry for the password. Using Manage Passwords is recommended. Only needed if Authentication is SNC.
SNC Partner Name = string
SNC name of the communication partner server. Must be an LDAP name. For example, p:CD=SID, O=ACompany, C=EN. Find the app server SNC name in the profile parameter snc/identity/as on SAP NetWeaver application server for ABAP. This parameter is required for SNC. The default length is 256 characters. Only needed if Authentication is SNC.
SNC Library Path = string
An SNC library path. This is the path to the SNC library on the client machine (SAP GUI). The default setting is the system-defined library as defined in the environment variable SNC_LIB. Destination service uses the property snc/gssapi_lib on SAP NetWeaver AS for Java. This parameter is optional and only used if Authentication is SNC.
SNC MyName = string
Specify the SNC name. Must be an LDAP name. For example, p:CD=SID, O=ACompany, C=EN. Destination services require it be set in the property snc/identity/as on SAP NetWeaver application servers for Java. The default length is 256 characters. This parameter is optional, but it is recommended to set it to ensure that the correct SNC name is used for the connection if Authentication is SNC. When left blank and SNC is enabled, JCo derives the value.
SNC Quality of Protection = integer
SNC "quality of protection". This parameter is optional. The values available are: 1 for authentication only. 2 for authentication and integrity protection. 3 for authentication, integrity, and privacy protection (encryption). 8 for the global default configuration (equivalent to 3). 9 for maximum protection. Only needed if Authentication is SNC.
Client = integer
SAP client. The default is 100. From SAP help: "A SAP client is defined as a self-contained commercial, organizational, and technical unit within an SAP system. All business data within a client are protected from other clients." Read more about this at Customer Data and System Data.
Language = string
Defaults to en. If not defined, the SAP system returns to JCo the default user language, and JCo will use that default value. Language codes adhere to ISO 639-1 codes.
Connection Options = column editor
Additional connection options for SAP ODP. Add a new row with +; remove a row with -; or click Add All. For each connection option, select the Parameter from the drop-down list and enter its Value as a string. Documentation for each of the available connection options can be found here.
Context = drop down
List of contexts available in the SAP system for the subscriber type SAP_BW. The context acts as a "data provider" for the data source. In SAP, a data provider permits configuration of data for extraction when targeting a specific use.
Search = string
Search string used to identify and reduce the quantity of data sources returned by SAP. For example, in this parameter you may wish to use the string TEXT. The maximum search string length is 30 characters.
Data Source = drop-down
In the context of SAP ODP, a data source is an "extractor". In SAP, an extractor is a data extraction through a specified context. A data source can be the result of aggregation of one or more tables or views in SAP. Data sources are listed where a match with the Search parameter has been ascertained. A semantic character is also provided. Additionally, a contextual description is given where a data source has been provided with description metadata in SAP.
The data sources available show a concatenation of:
- The data source technical name.
- A semantic (the SAP type of data source).
- The SAP data source description, in the language specified at login.
Semantics glossary:
H = Hierarchy F = Transaction Data/Facts P = Master Data/Attributes T = Texts V = View
Data Selection = dual listbox
Columns to include in the extraction. Columns will also have a contextual description where applicable.
Data Source filter = column editor
- Input Column: Select an input column. The available input columns vary depending upon the data source.
- Qualifier:
- Is: Compares the column to the value using the comparator.
- Not: Reverses the effect of the comparison, so "Equals" becomes "Not equals", "Less than" becomes "Greater than or equal to", etc.
- Comparator: Choose a method of comparing the column to the value. Possible comparators include: "Equal to", "Greater than", "Less than", "Greater than or equal to", "Less than or equal to", "Like", "Null". "Equal to" can match exact strings and numeric values, while other comparators, such as "Greater than" and "Less than", will work only with numerics. The "Like" operator allows the wildcard character [%] to be used at the start and end of a string value to match a column. The Null operator matches only null values, ignoring whatever the value is set to. Not all data sources support all comparators, meaning that it is likely that only a subset of the above comparators will be available to choose from.
- Value: The value to be compared.
Load Type = drop-down
Select a load type to manage the quantity of data sent via the network. This is especially useful when dealing with large RFC table parameters. Available options depend on your chosen data source. Possible types are F (full load), D (delta load), and R (recovery, available only if the data source supports deltas). The default is F.
When full load is selected, the whole table is sent back to the caller. When delta is selected, only appended, deleted, and updated table rows are transferred back to the caller.
Max Package Size = integer
Byte integer representing the package size of the data in SAP. The default setting is 5,000,000 bytes.
Max Package Size is based on the size of the package in SAP, measured in compressed bytes. This size may need to be tweaked depending on the content of the package and the size constraint of the cloud data platform.
Delta Initialisation = string
Only used if Load Type is D (delta). To only extract changes (creates, modifications, and deletes) from now onwards, enter X. Leave blank (the default) to also return historical changes.
Recovery Pointer = string
Only available if the data source supports deltas, and the Load Type is R (recovery).
SAP ODP allows you to repeat the extraction of data that may have been lost or corrupted due to an interruption of service, to restore and guarantee the data integrity of the extracted data. A recovery pointer is set by SAP ODP at a specific point-in-time in the delta queue, and you can use this to recover any data change recorded from that point onward. When the recovery pointer is set and the job repeated, all the data changes that happened from that recovery pointer onward are extracted, while all data changes that happened before that recovery pointer are not extracted.
You must obtain the value of the recovery pointer from the Replication Pointers table in SAP. After a data extraction that you believe to have been interrupted, refresh the SAP Replication Pointers table and make a note of the pointer. The pointer is a string of digits in the following format: 20220921082113.000021000. This is a year-month-day-hour-minute-second timestamp, and must be in this exact format; any format other than this will lead to an error. This string should be entered in the Recovery Pointer property to tell the component where in the delta queue to begin the recovery run.
Subscriber Name = string
The subscriber is the consumer of the data. Subscriber name is assigned by the user to allow the identification of which app is consuming the data. The maximum string size is 32 characters.
Subscriber Process = string
The identifier for the extraction process of the subscriber. The maximum string size is 64 characters.
Subscriber Run = string
The run ID of the subscriber. For example, ${dt.now()} The maximum string size is 64 characters. This parameter is optional.
Project = drop-down
The target BigQuery project to load data into. The special value, [Environment Default], will use the project defined in the Matillion ETL environment. For more information, read Creating and managing projects.
Dataset = drop-down
The target BigQuery dataset to load data into. The special value, [Environment Default], will use the dataset defined in the Matillion ETL environment. For more information, read Introduction to datasets.
Target Table = string
Provide a new table name.
Only used when the Table Type is "Native".
Warning
This table will be recreated on each run of the job and drop any existing table of the same name.
Cloud Storage Staging Area = string
The URL and path of the target Google Cloud Storage bucket to be used for staging the queried data.
Only used when the Table Type is "Native".
Load Options = multiple drop-downs
- Clean Cloud Storage Files: Destroy staged files on Google Cloud Storage after loading data. Default is On.
- Cloud Storage File Prefix: Give staged file names a prefix of your choice. The default setting is an empty field.
- Recreate Target Table: Choose whether the component recreates its target table before the data load. If Off, the component will use an existing table or create one if it does not exist. Default is On.
- Use Grid Variable: Check this checkbox to use a grid variable. This box is unchecked by default.
Location = string
The URL and path of the target Google Cloud Storage bucket. Only available when the Table Type is "External".
New Target Table = string
A name for the new external table. Only available when the Table Type is "External".
Table Type = drop-down
Select whether the table is Native (by default in BigQuery) or an External table.
| Snowflake | Delta Lake on Databricks | Amazon Redshift | Google BigQuery | Azure Synapse Analytics |
|---|---|---|---|---|
| ✅ | ❌ | ✅ | ✅ | ❌ |