dbt Core
Run dbt commands as part of an orchestration pipeline.
Note
Environment variables passed to this component must have the prefix DBT_.
Note
This component supports the use of pipeline and project variables. For more information, read Variables.
Note
For extracting data from the dbt cloud, use the dbt Cloud Flex connector component.
Properties
Name = string
A human-readable name for the component.
dbt Command = string
A dbt command to execute. A single instance of the component runs a single dbt command.
To learn more about dbt commands, read dbt Command reference.
The dbt run command runs dbt models stored in the Git repository you have synced with. The run command will run all models stored in your repository unless you specify a particular model.
Specify a model by running dbt run --{name_of_dbt_model}.
Git URL = string
URL to the external Git repository to sync to.
Git Branch = string
The Git branch to sync with.
Git Folder Path = string
Path to the folder in your repository that contains the dbt project. Leave empty if the dbt project is at the top of the repository.
Git Username = string
Your username for the service that hosts your repository, such as GitHub.
Git Password = drop-down
Secret definition that denotes your password or personal access token for the service that hosts your repository, such as GitHub.
If you haven't created a secret yet:
- Click Manage. A new browser tab will open at the Secret Definitions tab.
- Click Add secret definition.
-
Configure your secret definition using the table as reference if required.
Property Description Secret definition name A name for the secret definition. Any whitespace added to the start or end of the secret definition name is automatically trimmed. Description An optional, referential description for the project. Max 256 characters. Agent Select the agent that will be used by the pipelines that use this secret. This is only required if you have a Hybrid SaaS solution. To learn how to create an agent, read Create an agent. Secret Name For a Hybrid SaaS solution only, select a named entry created in AWS Secrets Manager. Secret Key For a Hybrid SaaS solution only, select a named secret key tied to your secret name. Use multi-line text input? For a Matillion Full SaaS solution only, toggle "Yes" to enable multi-line text input for secret values that require multi-line input. For example, certificates and SSH keys. Secret value For a Matillion Full SaaS solution only, give a secret value in the field provided. -
Click Create Secret.
- Return to the Git Password drop-down and select your new secret definition from the drop-down list.
Map dbt Environment Variables = column editor
Specify any dbt environment variables and their values.
Note
Environment variables passed to this component must have the prefix DBT_.
Global Configuration in profiles.yml = column editor
Append configuration parameters to the profiles.yml file.
- Key: The parameter name.
- Value: The parameter value.
- Data Type: String, number, or Boolean.
Read dbt's About profiles.yml to learn more.
Profile Configuration in profiles.yml = column editor
Assign additional output parameters or override existing parameters of the profiles.yml file. The Data Productivity Cloud generates the profile.yml file based on the environment you have configured in the Data Productivity Cloud—for instance, if your environment is set up for Snowflake, the Snowflake connection is included in the profile.yml file.
- Key: The parameter name.
- Value: The parameter value.
- Data Type: String, number, or Boolean.
Cloud data warehouse parameter documentation is available below:
| Snowflake | Databricks | Amazon Redshift |
|---|---|---|
| ✅ | ✅ | ✅ |