Skip to content

Pagination

Pagination is an essential feature when working with large datasets from external sources using Custom Connectors. This document provides an in-depth exploration of various pagination options available within a connector configuration, offering insights into how they can be customized to suit your data retrieval needs.

Custom Connectors offer support for a range of pagination techniques, and they can be configured in the Custom Connector dashboard while creating endpoints.

No Pagination

By default, connectors are often set to No Pagination, which means all data is retrieved without splitting it into pages. This is suitable for small datasets that can be fetched with a single request.

Relative Path

With Relative Path pagination, you navigate to the next page using the relative URI provided in the response. An example response might look like this:

{
  "next": "/orders"
}

Properties:

  • Base URI: The URI constant to exclude from the relative path.
  • Next Page URI: The property in the response used to indicate the next page. You can select from the available column names.
  • Page Size Parameter: The query parameter used to specify the page size.
  • Page Size: The number of records to return per page.

Full Path

Full Path pagination involves navigating to the next page using the complete URI provided in the response. For example:

{
  "next": "https://api/orders" 
}

Properties:

  • Next Page URI: The property in the response used to indicate the next page. You can select a column from the response.
  • Page Size Parameter: The query parameter used to specify the page size.
  • Page Size: The number of records to return per page.

Page-Based

With Page-Based pagination, you increment a page parameter to move to the next page. An example URL might look like this: ?page=1. Additionally, you may receive information about the total number of pages, as shown below:

?page=1
{
  "totalPages": "10000"
}

Properties:

  • Page Size Parameter: Query parameter specifying the page size.
  • Page Size: Number of records per page.
  • Page Number Parameter: Query parameter specifying the page number.
  • Last Page: Property in the response indicating the last page.

Link Header pagination involves using the link header in the response headers to navigate between pages. An example response header might look like this:

Link:
<https://api/data?page=1>; rel="prev",
<https://api/data?page=3>; rel="next"

Properties:

  • Page Size Parameter: The query parameter used to specify the page size.
  • Page Size: The number of records to return per page.

Offset

Offset pagination involves paging by incrementing a query parameter, often used with SQL-like databases. An example URL might look like this: ?offset=0&limit=10. You may also receive information about the total record count, as shown below:

{
  "record-count": "10"
}

Properties:

  • Limit: The number of records to return per page.
  • Record Count: Select a column from the response to count the total records. You can also use a boolean to indicate whether there are any more records, such as has_more = true or false.

Cursor

Cursor pagination uses a cursor parameter to navigate between pages. An example URL might look like this: ?page_size=10&cursor=id001. The response may provide information about the next page and cursor, as shown below:

{
  "next": {
    "page": "2",
    "cursor": "id002"
  }
}

Properties:

  • Cursor: The property in the response for the cursor. You can select a column from the response.
  • Cursor Parameter: The name of the cursor query parameter.
  • Page Size Parameter: The query parameter used to specify the page size.
  • Page Size: The number of records to return per page.

Each of these pagination techniques offers a unique way to retrieve data from external sources efficiently. The choice of pagination method depends on the characteristics of the data source and your specific data retrieval requirements.