Paginate Any AWS API With AWS SDK v3 Paginators

Photo by Annie Spratt on Unsplash

Almost every AWS list and describe call returns results in pages: 1,000 S3 keys, 50 log groups, 1 MB of DynamoDB items. Code that reads only the first page works in a test account and silently misses data in production. AWS SDK for JavaScript v3 ships a paginator for most of these operations, and they’re the simplest correct way to read everything.

This guide is for TypeScript and Node.js developers. You’ll see how to paginate AWS SDK v3 calls with the generated paginators, how pageSize and startingToken behave, how to write the loop yourself when there’s no paginator, and the mistakes that cause missing pages or loops that never end. The examples were type-checked with strict tsc and run against mocked clients with @aws-sdk/client-s3 and @aws-sdk/client-cloudwatch-logs 3.1141.0 in September 2026. The guide to list all objects in an S3 bucket with AWS SDK v3 goes deeper on ListObjectsV2; this one covers any operation.

How do AWS SDK v3 paginators work?

A paginator is an async generator. You pass a config with your client and the operation’s normal input. On each iteration it copies the current token into the input, calls client.send(), yields the whole response and reads the next token from it. When the response has no next token, the generator finishes. You consume it with for await...of, which the MDN reference for for await…of describes in detail. If you break or return out of the loop, the generator stops and no further request is sent.

Each service names its tokens differently, and the paginator hides that:

The mapping comes from each client’s generated code. Note that pageSize is whatever the operation’s size parameter means: for GetMetricData it’s data points, not metrics.

Which operations have a paginator?

Paginators are generated from each service’s API model, so an operation has one only if AWS marked it as paginated. They’re named paginate plus the operation name and exported from the client package. The quickest check is to type paginate after an import from the client and let your editor list them, or look in the package’s dist-types/pagination folder.

Some paginated-looking operations have none. In @aws-sdk/client-s3 3.1141.0 there’s no paginator for ListObjectVersions, which needs two markers, or for ListMultipartUploads. WorkSpaces’ DescribeWorkspacesConnectionStatus returns a NextToken but has no paginator either. For these, you write the loop yourself, as in the example below.

Prerequisites

• Node.js 18 or later and TypeScript. for await works in any async function and in ES module top-level code.
• The client packages you call, such as @aws-sdk/client-s3 and @aws-sdk/client-cloudwatch-logs, and @smithy/types if you want the Paginator type in your own signatures.
• Credentials the SDK can resolve; see AWS SDK v3 credential providers: fromIni, fromSSO and assume role.

How to paginate with AWS SDK v3, step by step

1. Import the paginator next to the clientimport { S3Client, paginateListObjectsV2 } from "@aws-sdk/client-s3". DynamoDB DocumentClient paginators come from @aws-sdk/lib-dynamodb.
2. Pass the client in the config, the input separatelypaginateListObjectsV2({ client: s3, pageSize: 1000 }, { Bucket }). The client must be an instance of that service’s client class.
3. Loop with for awaitEach page is the full command output, typed, so page.Contents and page.KeyCount are available.
4. Stream or collectHandle items inside the loop when the total can be large; push into an array only when it fits in memory.
5. Save the token if you need to resumeRead the output token from the last page you processed and pass it as startingToken next time.

Example: seven pagination patterns in one module

// paginate.ts: AWS SDK for JavaScript v3 pagination patterns
import {
  ListObjectVersionsCommand,
  S3Client,
  paginateListObjectsV2,
  type ListObjectsV2CommandOutput,
  type ObjectVersion,
} from "@aws-sdk/client-s3";
import {
  CloudWatchLogsClient,
  paginateDescribeLogGroups,
  paginateGetLogEvents,
  type LogGroup,
} from "@aws-sdk/client-cloudwatch-logs";
import type { Paginator } from "@smithy/types";

const s3 = new S3Client({ maxAttempts: 5 }); // every page is a normal request, so retries apply per page
const logs = new CloudWatchLogsClient({});

/** 1. Stream: handle each page as it arrives. Memory stays flat however many objects there are. */
export async function totalBytes(Bucket: string, Prefix?: string): Promise<number> {
  let bytes = 0;
  for await (const page of paginateListObjectsV2({ client: s3, pageSize: 1000 }, { Bucket, Prefix })) {
    for (const obj of page.Contents ?? []) bytes += obj.Size ?? 0;
  }
  return bytes;
}

/** 2. Collect: gather every item from any paginator into one array (only when the full list fits in memory). */
export async function collect<Page, Item>(pages: AsyncIterable<Page>, pick: (page: Page) => Item[] | undefined): Promise<Item[]> {
  const items: Item[] = [];
  for await (const page of pages) items.push(...(pick(page) ?? []));
  return items;
}

export function allLogGroups(): Promise<LogGroup[]> {
  return collect(paginateDescribeLogGroups({ client: logs, pageSize: 50 }, {}), (p) => p.logGroups);
}

/** 3. Stop early: break ends the loop and no further pages are requested. */
export async function firstKeys(Bucket: string, limit: number): Promise<string[]> {
  const keys: string[] = [];
  for await (const page of paginateListObjectsV2({ client: s3 }, { Bucket })) {
    for (const obj of page.Contents ?? []) {
      if (obj.Key) keys.push(obj.Key);
      if (keys.length >= limit) return keys; // returning from inside for await also stops the paginator
    }
  }
  return keys;
}

/** 4. Resume: process one batch of pages per run and hand back the token to start from next time. */
export async function processBatch(maxPages: number, startingToken?: string): Promise<{ names: string[]; nextToken?: string }> {
  const names: string[] = [];
  let pages = 0;
  let nextToken: string | undefined;
  for await (const page of paginateDescribeLogGroups({ client: logs, pageSize: 50, startingToken }, {})) {
    names.push(...(page.logGroups ?? []).map((g) => g.logGroupName ?? ""));
    nextToken = page.nextToken; // undefined on the last page
    if (++pages >= maxPages) break;
  }
  return { names, nextToken }; // CloudWatch Logs tokens expire after 24 hours
}

/** 5. Typed paginator variable: the element type is the full command output. */
export function listPages(Bucket: string): Paginator<ListObjectsV2CommandOutput> {
  return paginateListObjectsV2({ client: s3 }, { Bucket });
}

/** 6. No paginator: ListObjectVersions needs two markers, so loop by hand. */
export async function allVersions(Bucket: string, Prefix?: string): Promise<ObjectVersion[]> {
  const versions: ObjectVersion[] = [];
  let KeyMarker: string | undefined;
  let VersionIdMarker: string | undefined;
  do {
    const res = await s3.send(new ListObjectVersionsCommand({ Bucket, Prefix, KeyMarker, VersionIdMarker }));
    versions.push(...(res.Versions ?? []));
    KeyMarker = res.IsTruncated ? res.NextKeyMarker : undefined;
    VersionIdMarker = res.IsTruncated ? res.NextVersionIdMarker : undefined;
  } while (KeyMarker);
  return versions;
}

/** 7. Token that never ends: GetLogEvents returns a forward token even at the end of the stream. */
export async function readStream(logGroupName: string, logStreamName: string): Promise<string[]> {
  const lines: string[] = [];
  for await (const page of paginateGetLogEvents(
    { client: logs, stopOnSameToken: true }, // without this the loop never finishes
    { logGroupName, logStreamName, startFromHead: true },
  )) {
    for (const e of page.events ?? []) lines.push(e.message ?? "");
  }
  return lines;
}

Pattern 2 is generic: collect(paginateQuery({ client: docClient }, input), (p) => p.Items) works the same way for the DynamoDB queries in query DynamoDB with AWS SDK v3. Pattern 6 is the template for every operation without a paginator: send, collect, copy the next token or markers into the next input, stop when the response says it isn’t truncated.

How do pageSize and startingToken behave?

• pageSize is a maximum, not a promise. Services can return fewer items, including empty pages that still carry a next token. Never treat a short page as the last one; let the paginator decide.
• The input wins. The paginator sets the size parameter only if your input doesn’t. With { pageSize: 100 } and { MaxKeys: 5 }, requests use 5.
• startingToken resumes a listing. It’s the token from the page after the last one you finished. Tokens are opaque and tied to the same input; CloudWatch Logs tokens expire after 24 hours, and other services set their own lifetimes.
• stopOnSameToken handles APIs whose token never disappears. GetLogEvents returns the same forward token you passed in at the end of a stream, and its generated paginator doesn’t stop on that by itself. Pass stopOnSameToken: true, as in pattern 7, and set endTime if events keep arriving while you read.

Collecting vs streaming, and rate limits

Streaming inside the loop keeps memory flat and lets you stop early. Collecting into an array is simpler for small lists such as log groups or Lambda functions, and needed when you sort or join. For account inventories, a middle path works well: stream pages and write each one to disk or a queue as it arrives.

Each page is an ordinary send(), so the client’s retry strategy applies per page: throttled requests are retried with backoff up to maxAttempts. Pages within one paginator are sequential by design, since each needs the previous token. To go faster, run different paginators in parallel, for example one per Region with Promise.all, and keep the count modest for APIs with low rate limits. The guide to configure retry and timeout settings in AWS SDK for JavaScript v3 shows how to raise maxAttempts or switch to adaptive retries for long listings, and monitor AWS service quota usage helps when a quota, not the rate, is the limit.

Permissions needed

A paginator needs exactly the permissions of the operation it calls, once per page. paginateListObjectsV2 needs s3:ListBucket on the bucket ARN; paginateDescribeLogGroups needs logs:DescribeLogGroups; the manual ListObjectVersions loop needs s3:ListBucketVersions. There’s no separate “paginate” permission. To get the list for a whole module, paste it into the IAM policy generator for TypeScript code, or follow the guide to find the IAM actions your AWS SDK for JavaScript code needs.

Troubleshooting and common mistakes

• The second listing returns only the last page. Paginators write the token into the input object you pass. Reusing that object for another paginator starts from the old token: in a test, a reused input returned 1 object instead of 4. Pass a fresh input object each time.
• Invalid client, expected instance of S3Client. The config’s client isn’t an instance of that service’s client class: a different service’s client, a wrapper object, or a copy of the client loaded from a second installed version of the package. For DynamoDB, lib-dynamodb paginators need a DynamoDBDocumentClient.
• A loop that never ends. The API always returns a token, like GetLogEvents. Use stopOnSameToken: true and an end time.
• Only 1,000 results. The code calls send(new ListObjectsV2Command(...)) once instead of paginating. Search the codebase for list and describe calls outside a paginator; the step-by-step guide to migrate a Node.js app from AWS SDK v2 to v3 covers replacing v2’s eachPage loops.
• Tests hang or return the same page forever. A mock that always returns a next token. With aws-sdk-client-mock, use resolvesOnce for each page and a final page without a token, as shown in mock AWS SDK v3 clients in Jest and Vitest.

Limits: what paginators can’t do

• They don’t parallelize pages of one listing. For huge S3 buckets, split the work by prefix and paginate each prefix separately.
• They don’t give you a snapshot. Items created or deleted during a long listing may or may not appear.
• They don’t filter on the server beyond what the operation supports. Filtering inside the loop still pays for every page read.
• They don’t exist for every operation, as shown above, and the pageSize meaning varies by operation.

If you’d rather ask than write the loop, ChatWithCloud generates and runs the SDK calls for you; see how to list AWS resources with natural language from your terminal. Its generated code uses SDK v2, so treat it as a way to get answers, not as code to paste into a v3 project.

Frequently asked questions

Related guides

• List all objects in an S3 bucketListObjectsV2 pagination in depth.
• Query DynamoDB with AWS SDK v3paginateQuery with keys and indexes.
• Configure retries and timeouts in SDK v3Keep long listings alive under throttling.
• Mock AWS SDK v3 in unit testsTest multi-page listings without AWS.
• Migrate from AWS SDK v2 to v3Replace eachPage and manual loops.
• Find the IAM actions your SDK code needsPermissions for every paginated call.