Reviewing pipeline quality

You can use the Review button on the Designer canvas to check whether your pipelines meet defined quality standards. This feature checks your pipeline according to a defined set of rules in order to make sure that your pipelines align with your organization's standards.

The rules used to review your pipeline quality are predefined, but you can configure some according to your needs.

---

Reviewing a pipeline

Before using the Review feature for the first time, you need to add and configure a rules file, which contains the standards that the Data Productivity Cloud will use to assess your pipeline quality.

To do this:

1. In the top right of the Designer canvas, click the drop-down icon next to the Review button, then select Manage rules.

   Note

   The Review button will be grayed out if you haven't configured a rules file yet.

2. In the rules file, edit the rules to suit your requirements. For more information, read Configuring pipeline quality rules below.

3. Close the rules file and return to your pipeline.
4. Click Review.

At the bottom of the canvas, the Rules results tab will open. The Rules results tab lists any rules that your pipeline does not adhere to. You can choose whether a rule is displayed as an Error or a Warning if the pipeline does not follow the rule.

---

Configuring pipeline quality rules

The rules that the Data Productivity Cloud uses to review your pipeline quality are stored in a YAML file called pipeline_quality_rules. This file is automatically added to the .matillion/config folder in your project when you click Manage rules for the first time. Do not move the pipeline_quality_rules file out of this folder.

Each rule contains one fixed parameter (id) and two or three editable parameters (enabled, enforcementLevel and config). The table below explains what each of these parameters represents.

You can download a copy of the default rules file here.

Parameter	Value	Description	Editable
id	Rule ID	A unique identifier for this rule. This is shown in the Rules results tab.	No
enabled	true or false	Set to true to apply this rule to your pipeline, or false to skip this rule when reviewing your pipeline.	Yes
enforcementLevel	error or warn	Choose which severity level to display in the Rules results tab if this rule is not followed.	Yes
config	Varies depending on the rule	Configure a setting that is specific to this rule, as described below.	Yes

Individual rules

The table below explains what each rule checks. If a rule has a config parameter, the Configuration details column explains how to define the value to check for.

Rule ID	Applies to	Rule	Configuration details
minimum-components	All pipelines	Reviewed pipelines must have at least the set number of components.	Set the minimum number of components, e.g. minComponents: 3. The default value is 1.
maximum-components	All pipelines	Reviewed pipelines must have no more than the set number of components.	Set the maximum number of components, e.g. minComponents: 20. The default value is 10.
component-naming-convention	All pipelines	The name of each component in the reviewed pipeline must adhere to a certain regular expression.	Enter a regular expression, e.g. regexMatchPattern: "^[a-zA-Z0-9]+$".
unused-components	All pipelines	Reviewed pipelines must not contain any unused components.	–
copied-variables	All pipelines	Copied variables must only be used in concurrent mode iterators.	–
start-and-end-components	Orchestration pipelines	Reviewed orchestration pipelines must contain a Start component, and all end points of a pipeline must end in an End Success or End Failure component.	–
maximum-outputs	Transformation pipelines	Reviewed transformation pipelines must have no more than the set number of output components, such as Table Output.	Set the maximum number of output components, e.g. maxOutputs: 2. The default value is 1.
end-output-components	Transformation pipelines	Each path within a transformation pipeline must end with an output component, such as Table Output.	–

Got feedback or spotted something we can improve?

We'd love to hear from you. Join the conversation in the Documentation forum!