Pagination

Pagination is an essential feature when working with large datasets from external sources using Custom Connectors. This document provides an in-depth exploration of various pagination options available within a connector configuration, offering insights into how they can be customized to suit your data retrieval needs.

Custom Connectors allow a range of pagination techniques, which can be configured in the Custom Connector dashboard while creating endpoints. Each of these pagination techniques offers a unique way to retrieve data from external sources efficiently. The choice of pagination method depends on the characteristics of the data source and your specific data retrieval requirements.

No Pagination

By default, connectors are often set to No Pagination, which means all data is retrieved without splitting it into pages. This is suitable for small datasets that can be fetched with a single request.

Relative Path

With Relative Path pagination, you navigate to the next page using the relative URI provided in the response. An example response might look like this:

{
  "next": "/orders"
}

Properties:

Base URI: The URI constant to exclude from the relative path.
Next Page URI: The property in the response used to indicate the next page. The drop-down will display the response schema for you to choose a property from.
Page Size Parameter: The query parameter used to specify the page size.
Page Size: The number of records to return per page.

Full Path

Full Path pagination involves navigating to the next page using the complete URI provided in the response. An example response might look like this:

{
  "next": "https://api/orders" 
}

Properties:

Next Page URI: The property in the response used to indicate the next page. The drop-down will display the response schema for you to choose a property from.
Page Size Parameter: The query parameter used to specify the page size.
Page Size: The number of records to return per page.

Page-Based

With Page-Based pagination, you increment a page parameter to move to the next page. An example URL might look like this: ?page=1. Additionally, you may receive information about the total number of pages, as shown below:

?page=1
{
    "totalPages": "10000"
}

Properties:

Page Size Parameter: The query parameter used to specify the page size.
Page Size: The number of records to return per page.
Page Number Parameter: The query parameter used to specify the page number.
Last Page: The property in the response used to indicate the last page. The drop-down will display the response schema for you to choose a property from.

Link Header

Link Header pagination involves using the link header in the response headers to navigate between pages. An example response header might look like this:

Link:
<https://api/data?page=1>; rel="prev",
<https://api/data?page=3>; rel="next"

Properties:

Page Size Parameter: The query parameter used to specify the page size.
Page Size: The number of records to return per page.

Offset

Offset pagination involves paging by incrementing a query parameter. This method is often used with SQL-like databases. An example URL might include parameters like this: ?offset=0&limit=10. You may also receive information about the total record count, as shown below:

{
    "record-count": "10"
}

Properties:

Limit Parameter: The query parameter used for the page limit.
Limit: The number of records to return per page.
Record Count: Select a column from the response to count the total records. The drop-down will display the response schema for you to choose a property from. You can also use a boolean to indicate whether there are any more records, such as has_more = true or false.
Offset Parameter: The query parameter used for the page offset.

Cursor

Cursor pagination uses a cursor parameter to navigate between pages. An example URL might look like this: ?page_size=10&cursor=id001. The response may provide information about the next page and cursor, as shown below:

{
  "next": {
    "page": "2",
    "cursor": "id002"
  }
}

There are two possible ways of using the cursor, and the method used depends on the setting of the Pagination Method property:

Header: Extracts the cursor from a header in the response and sends it as a header in the subsequent request.
Query Parameter: Extracts the cursor from the JSON response and sends it as a query parameter in the subsequent request.

Properties:

Cursor Source Header Name: The name of the header parameter to extract the cursor from. Required if the Pagination Method is Header.
Cursor Header Name: The name of the cursor header to pass via the API request. Required if the Pagination Method is Header.
Page Size Header: The Header used to indicate the page. Required if the Pagination Method is Header.
Page Size: The number of records to return per page. Required if the Pagination Method is Header.
Cursor: The property in the response used for the cursor. The drop-down will display the response schema for you to choose a property from. Required if the Pagination Method is Query Parameter.
More Pages Indicator Key: The property in the response that indicates that there are more pages. Required if the Pagination Method is Query Parameter.
Cursor Parameter Name: The name of the cursor query parameter to pass via the API request. Required if the Pagination Method is Query Parameter.
Page Size Parameter: The query parameter used to specify the page size. Required if the Pagination Method is Query Parameter.
Page Size: The number of records to return per page. Required if the Pagination Method is Query Parameter.

Script

Pages using a custom script. You can use Matillion's scripted Paging, a powerful scripting language that gives you control over pagination, rate limiting, and API request and response management. Read Script paging for details.