Reviewing pipeline quality🔗

Public preview

You can use the Review button on the Designer canvas to check whether your pipelines meet defined quality standards. This feature checks your pipeline according to a defined set of rules in order to make sure that your pipelines align with your organization's standards.

The rules used to review your pipeline quality are predefined, but you can configure some according to your needs.

Reviewing a pipeline🔗

Before using the Review feature for the first time, you need to add and configure a rules file, which contains the standards that the Data Productivity Cloud will use to assess your pipeline quality.

To do this:

In the top right of the Designer canvas, click the drop-down icon next to the Review button, then select Manage rules.

Note

The Review button will be grayed out if you haven't configured a rules file yet.
In the rules file, edit the rules to suit your requirements. For more information, read Configuring pipeline quality rules below.
Close the rules file and return to your pipeline.
Click Review.

At the bottom of the canvas, the Review results panel will open. The Review results panel lists any rules that your pipeline does not adhere to. You can choose whether a rule is displayed as an Error or a Warning if the pipeline does not follow the rule.

Reviewing all pipelines in a branch🔗

You can review all orchestration and transformation pipelines within the current branch in a single action.

To review all pipelines:

In the top right of the Designer canvas, click the drop-down icon next to the Review button.
Select Review all pipelines.

The review runs against all eligible pipelines in the branch. Results are displayed in the Review results panel.

Rule violations are listed in a paginated table, with 25 rule violations displayed per page. Each row includes the pipeline name, the rule ID, and a resolution describing the issue and how to address it. A Manage rules link is also available, allowing you to update the pipeline quality rules directly.

Fixing rule violations with Maia🔗

From the Review results panel, you can use Maia to automatically fix selected rule violations.

To fix rule violations using Maia:

In the Review results panel, review the list of rule violations identified across the pipelines in the branch.
Use the checkboxes next to each rule violation to select the issues you want Maia to fix. The maximum selection is 50 rule violations.
Optionally, use the checkbox in the table header to select or deselect all rule violations on the current page.
Click Fix with Maia in the top-right corner of the panel. This will open the Maia chat with a pre-sent message asking Maia to fix the violations.

Reviewing pipelines before committing🔗

You can choose to run a pipeline quality review as part of the commit process.

When committing changes, the Commit changes dialog includes a checkbox to enable pipeline quality review. If the checkbox is selected, the Data Productivity Cloud runs the pipeline quality review before completing the commit.

If the review identifies rule violations with an error enforcement level, the commit is blocked.
If the review passes or only identifies rule violations with a warn enforcement level, the commit proceeds successfully.

This helps ensure critical pipeline quality standards are met before changes are committed, while allowing flexibility for non-blocking warnings.

Configuring pipeline quality rules🔗

The rules that the Data Productivity Cloud uses to review your pipeline quality are stored in a YAML file called pipeline_quality_rules. This file is automatically added to the .matillion/config folder in your project when you click Manage rules for the first time. Do not move the pipeline_quality_rules file out of this folder.

Each rule contains one fixed parameter (id) and two or three editable parameters (enabled, enforcementLevel and config). The table below explains what each of these parameters represents.

You can download a copy of the default rules file here.

Parameter	Value	Description	Editable
`id`	Rule ID	A unique identifier for this rule. This is shown in the Rules results tab.	No
`enabled`	`true` or `false`	Set to `true` to apply this rule to your pipeline, or `false` to skip this rule when reviewing your pipeline.	Yes
`enforcementLevel`	`error` or `warn`	Choose which severity level to display in the Rules results tab if this rule is not followed.	Yes
`config`	Varies depending on the rule	Configure a setting that is specific to this rule, as described below.	Yes

Individual rules🔗

The table below explains what each rule checks. If a rule has a config parameter, the Configuration details column explains how to define the value to check for.

Rule ID	Applies to	Rule	Configuration details
`minimum-components`	All pipelines	Reviewed pipelines must have at least the set number of components.	Set the minimum number of components, e.g. `minComponents: 3`. The default value is `1`.
`maximum-components`	All pipelines	Reviewed pipelines must have no more than the set number of components.	Set the maximum number of components, e.g. `minComponents: 20`. The default value is `10`.
`component-naming-convention`	All pipelines	The name of each component in the reviewed pipeline must adhere to a certain regular expression.	Enter a regular expression, e.g. `regexMatchPattern: "^[a-zA-Z0-9]+$"`.
`unused-components`	All pipelines	Reviewed pipelines must not contain any unused components.	–
`copied-variables`	All pipelines	Copied variables must only be used in concurrent mode iterators.	–
`start-and-end-components`	Orchestration pipelines	Reviewed orchestration pipelines must contain a Start component, and all end points of a pipeline must end in an End Success or End Failure component.	–
`maximum-outputs`	Transformation pipelines	Reviewed transformation pipelines must have no more than the set number of output components, such as Table Output.	Set the maximum number of output components, e.g. `maxOutputs: 2`. The default value is `1`.
`end-output-components`	Transformation pipelines	Each path within a transformation pipeline must end with an output component, such as Table Output.	–