tessl/npm-got

Human-friendly and powerful HTTP request library for Node.js

—

Pending

Quality

Pending

Does it follow best practices?

Impact

Pending

No eval scenarios have been run

Securityby

Pending

The risk profile of this skill

Overview

Eval results

Files

Pagination

Name: tessl/npm-got
Author: tessl

Built-in pagination support for APIs that return paginated results with async iterator interface and automatic page traversal.

Capabilities

Pagination Interface

The main pagination interface provides async iteration over paginated API responses.

/**
 * Pagination interface for traversing paginated API responses
 */
interface GotPaginate {
  /**
   * Create async iterator for paginated responses
   * @param url - API endpoint URL
   * @param options - Request options with pagination configuration
   * @returns Async iterator yielding individual items
   */
  <T, R = unknown>(url: string | URL, options?: OptionsWithPagination<T, R>): AsyncIterableIterator<T>;
  
  /**
   * Create async iterator using options-only syntax
   * @param options - Request options including URL and pagination config
   * @returns Async iterator yielding individual items
   */
  <T, R = unknown>(options?: OptionsWithPagination<T, R>): AsyncIterableIterator<T>;
  
  /**
   * Alias for the main pagination function
   */
  each: (<T, R = unknown>(url: string | URL, options?: OptionsWithPagination<T, R>) => AsyncIterableIterator<T>)
    & (<T, R = unknown>(options?: OptionsWithPagination<T, R>) => AsyncIterableIterator<T>);
  
  /**
   * Collect all paginated results into an array
   * @param url - API endpoint URL
   * @param options - Request options with pagination configuration
   * @returns Promise resolving to array of all items
   */
  all: (<T, R = unknown>(url: string | URL, options?: OptionsWithPagination<T, R>) => Promise<T[]>)
    & (<T, R = unknown>(options?: OptionsWithPagination<T, R>) => Promise<T[]>);
}

Pagination Options

Configuration interface for customizing pagination behavior.

/**
 * Options for paginated requests
 */
interface OptionsWithPagination<T = unknown, R = unknown> extends OptionsInit {
  pagination?: PaginationOptions<T, R>;
}

/**
 * Pagination configuration options
 */
interface PaginationOptions<ElementType = unknown, BodyType = unknown> {
  /**
   * Transform function to extract items from response body
   * @param response - HTTP response object
   * @returns Array of items or Promise resolving to array
   */
  transform?: (response: Response<BodyType>) => Promise<ElementType[]> | ElementType[];
  
  /**
   * Filter function for individual items
   * @param item - Current item
   * @param allItems - All items collected so far
   * @param currentItems - Items from current page
   * @returns Whether to include the item
   */
  filter?: (item: ElementType, allItems: ElementType[], currentItems: ElementType[]) => boolean;
  
  /**
   * Function to determine if pagination should continue
   * @param item - Current item being processed
   * @param allItems - All items collected so far
   * @param currentItems - Items from current page
   * @returns Whether to continue paginating
   */
  shouldContinue?: (item: ElementType, allItems: ElementType[], currentItems: ElementType[]) => boolean;
  
  /**
   * Function to generate next page request options
   * @param response - Current response object
   * @param allItems - All items collected so far
   * @param currentItems - Items from current page
   * @returns Options for next request, or false to stop
   */
  paginate?: (response: Response<BodyType>, allItems: ElementType[], currentItems: ElementType[]) => OptionsInit | false;
  
  /**
   * Maximum number of items to collect
   */
  countLimit?: number;
  
  /**
   * Maximum number of requests to make
   */
  requestLimit?: number;
  
  /**
   * Stack all items before yielding (for .all() method)
   * @default false
   */
  stackAllItems?: boolean;
}

Basic Pagination Usage

Simple pagination with async iteration:

import got from "got";

// GitHub commits pagination
const pagination = got.paginate("https://api.github.com/repos/sindresorhus/got/commits", {
  pagination: {
    countLimit: 10
  }
});

console.log("Latest 10 commits:");
for await (const commitData of pagination) {
  console.log(`${commitData.sha.slice(0, 7)}: ${commitData.commit.message}`);
}

Collect all results:

import got from "got";

// Get all commits (up to limit)
const commits = await got.paginate.all("https://api.github.com/repos/sindresorhus/got/commits", {
  pagination: {
    countLimit: 50,
    requestLimit: 5
  }
});

console.log(`Retrieved ${commits.length} commits`);

Advanced Pagination Configuration

Custom transform and pagination logic:

import got from "got";

interface GitHubCommit {
  sha: string;
  commit: {
    message: string;
    author: {
      name: string;
      date: string;
    };
  };
}

interface GitHubResponse {
  items?: GitHubCommit[];
  // GitHub returns array directly for commits endpoint
}

const commits = await got.paginate.all("https://api.github.com/repos/sindresorhus/got/commits", {
  headers: {
    "Authorization": "token ghp_xxxxxxxxxxxxxxxxxxxx",
    "Accept": "application/vnd.github.v3+json"
  },
  pagination: {
    // Transform: GitHub commits API returns array directly
    transform: (response: Response<GitHubCommit[]>) => response.body,
    
    // Custom pagination logic using Link header
    paginate: (response, allItems, currentItems) => {
      const linkHeader = response.headers.link;
      
      if (!linkHeader) return false;
      
      // Parse Link header for next URL
      const nextMatch = linkHeader.match(/<([^>]+)>;\s*rel="next"/);
      if (nextMatch) {
        return {
          url: nextMatch[1],
          headers: response.request.options.headers
        };
      }
      
      return false;
    },
    
    // Filter commits by author
    filter: (commit) => commit.commit.author.name === "Sindre Sorhus",
    
    // Stop early if we find an old commit
    shouldContinue: (commit, allItems) => {
      const commitDate = new Date(commit.commit.author.date);
      const oneMonthAgo = new Date();
      oneMonthAgo.setMonth(oneMonthAgo.getMonth() - 1);
      return commitDate > oneMonthAgo;
    },
    
    countLimit: 100,
    requestLimit: 10
  }
});

console.log(`Found ${commits.length} commits by Sindre in the last month`);

REST API Pagination Patterns

Offset-based pagination:

import got from "got";

interface ApiResponse {
  data: any[];
  meta: {
    total: number;
    page: number;
    per_page: number;
  };
}

const allData = await got.paginate.all("https://api.example.com/users", {
  searchParams: {
    per_page: 50,
    page: 1
  },
  pagination: {
    transform: (response: Response<ApiResponse>) => response.body.data,
    
    paginate: (response, allItems) => {
      const { meta } = response.body as ApiResponse;
      const hasMore = (meta.page * meta.per_page) < meta.total;
      
      if (hasMore) {
        return {
          searchParams: {
            per_page: 50,
            page: meta.page + 1
          }
        };
      }
      
      return false;
    },
    
    countLimit: 1000
  }
});

Cursor-based pagination:

import got from "got";

interface CursorResponse {
  data: any[];
  pagination: {
    next_cursor?: string;
  };
}

const allRecords = await got.paginate.all("https://api.example.com/records", {
  pagination: {
    transform: (response: Response<CursorResponse>) => response.body.data,
    
    paginate: (response) => {
      const { pagination } = response.body as CursorResponse;
      
      if (pagination.next_cursor) {
        return {
          searchParams: {
            cursor: pagination.next_cursor
          }
        };
      }
      
      return false;
    }
  }
});

Link header pagination:

import got, { parseLinkHeader } from "got";

const allIssues = await got.paginate.all("https://api.github.com/repos/owner/repo/issues", {
  headers: {
    "Authorization": "token your_token_here"
  },
  pagination: {
    transform: (response: Response<any[]>) => response.body,
    
    paginate: (response) => {
      const linkHeader = response.headers.link;
      
      if (linkHeader) {
        const links = parseLinkHeader(linkHeader);
        const nextLink = links.find(link => link.parameters.rel === '"next"');
        if (nextLink) {
          return { url: nextLink.reference };
        }
      }
      
      return false;
    }
  }
});

Pagination Utilities

Parse Link Header utility:

/**
 * Parse HTTP Link header for pagination URLs
 * @param header - Link header value
 * @returns Array of parsed link objects
 */
function parseLinkHeader(header: string): LinkHeaderResult[];

interface LinkHeaderResult {
  reference: string;
  parameters: Record<string, string>;
}

Usage:

import got, { parseLinkHeader } from "got";

const response = await got("https://api.github.com/repos/owner/repo/issues");
const linkHeader = response.headers.link;

if (linkHeader) {
  const links = parseLinkHeader(linkHeader);
  const nextLink = links.find(link => link.parameters.rel === '"next"');
  const lastLink = links.find(link => link.parameters.rel === '"last"');
  console.log("Next page:", nextLink?.reference);
  console.log("Last page:", lastLink?.reference);
}

Error Handling in Pagination

import got, { HTTPError, RequestError } from "got";

async function safePagination() {
  try {
    const results = [];
    
    for await (const item of got.paginate("https://api.example.com/data", {
      pagination: {
        countLimit: 100,
        requestLimit: 5
      }
    })) {
      results.push(item);
    }
    
    return results;
  } catch (error) {
    if (error instanceof HTTPError) {
      console.error(`HTTP error: ${error.response.statusCode}`);
      // Return partial results if available
      return [];
    } else if (error instanceof RequestError) {
      console.error(`Request error: ${error.code}`);
      return [];
    } else {
      throw error;
    }
  }
}

Performance Considerations

Concurrent pagination (advanced):

import got from "got";

async function concurrentPagination(baseUrl: string, totalPages: number) {
  const pagePromises = [];
  
  for (let page = 1; page <= totalPages; page++) {
    const promise = got(`${baseUrl}?page=${page}`).json();
    pagePromises.push(promise);
  }
  
  const responses = await Promise.all(pagePromises);
  return responses.flatMap(response => response.data || []);
}

// Use with caution - respects rate limits
const allData = await concurrentPagination("https://api.example.com/users", 10);

Common Pagination Patterns

Pattern	Description	Use Case
Link Header	Uses HTTP Link header with `rel="next"`	GitHub, GitLab APIs
Offset/Limit	Uses `offset` and `limit` parameters	Traditional REST APIs
Page/Per Page	Uses `page` and `per_page` parameters	Many REST APIs
Cursor-based	Uses opaque cursor tokens	Twitter, Facebook APIs
Time-based	Uses timestamp ranges	Analytics APIs