DynamoDB BatchWriteItem With AWS SDK v3: Batch Write Items

Photo by Daniel von Appen on Unsplash

You reach for a batch write when you load a table from a file, copy data between tables, backfill a new attribute or clear out test data. One call writes up to 25 items, which cuts round trips compared with PutCommand in a loop. The catch is that a successful response doesn’t mean everything was written: DynamoDB can hand back part of the batch as unprocessed, and the SDK won’t retry that part for you. If the table has a stream, every item a batch writes also becomes a change record, so a large import can set off a burst of downstream work; the guide to process DynamoDB Streams records in a TypeScript Lambda function shows how to handle it.

This guide is for Node.js and TypeScript developers. You’ll get a module that uses DynamoDB BatchWriteItem with AWS SDK v3 for any number of items, with chunking, duplicate-key handling, retries with backoff and jitter, and consumed capacity totals, plus the limits, IAM policy and errors you’ll meet.

What are the BatchWriteItem limits?

Those limits come from the BatchWriteItem API reference. Deleting an item that doesn’t exist still succeeds and still consumes a write capacity unit, and the response never returns the old items.

BatchWriteCommand, PutCommand or TransactWriteCommand?

Conditional writes, counters and optimistic locking are covered in the guide to DynamoDB UpdateItem with AWS SDK v3: conditions and counters, which also explains when a transaction is worth its cost. When a group of writes has to succeed or fail together, see DynamoDB TransactWriteItems with AWS SDK v3. To send up to 25 SQL-style statements in one call instead, see DynamoDB PartiQL with BatchExecuteStatement in AWS SDK v3.

Prerequisites

• Node.js 18 or later with TypeScript and tsx, and the @aws-sdk/client-dynamodb and @aws-sdk/lib-dynamodb packages.
• A table and its key schema. The module needs the key attribute names to spot duplicates, such as customerId and orderId.
• Credentials the SDK can resolve, as described in AWS SDK v3 credential providers: fromIni, fromSSO and assume role.
• Coming from SDK v2’s DocumentClient.batchWrite? The step-by-step guide to migrate a Node.js app from AWS SDK v2 to v3 covers the client changes, and the free AWS SDK v2 to v3 converter gives you a first draft to review.

How to use DynamoDB BatchWriteItem with AWS SDK v3, step by step

1. Create a DocumentClientDynamoDBDocumentClient.from(new DynamoDBClient({})) lets you pass plain JavaScript objects. Set removeUndefinedValues: true, or an undefined attribute throws before the request is sent.
2. Build write requests{ PutRequest: { Item } } writes a whole item; { DeleteRequest: { Key } } deletes by primary key.
3. Remove duplicate keysTwo requests for the same key make DynamoDB reject the whole call. Keep the last request per key.
4. Split into chunks of 25Also keep an eye on size if items are large: 25 items near 400 KB each would pass 16 MB.
5. Send, then read UnprocessedItemsIt’s in the same shape as RequestItems, so you can send it straight back.
6. Back off before each retryWait a random time that grows with each attempt, and give up after a fixed number of attempts, returning what failed.
7. Run a few chunks in parallelFour concurrent chunks is a sensible start; more only helps if the table has capacity to spare.

Example: batch write any number of items

// batch-write.ts
// Write or delete any number of DynamoDB items with BatchWriteCommand (AWS SDK for JavaScript v3):
// chunks of 25, duplicate keys removed, UnprocessedItems retried with exponential backoff and full jitter,
// a few chunks in flight at once, and consumed capacity added up.
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { BatchWriteCommand, DynamoDBDocumentClient } from "@aws-sdk/lib-dynamodb";

type Item = Record<string, unknown>;
type WriteRequest =
  | { PutRequest: { Item: Item } }
  | { DeleteRequest: { Key: Item } };

const ddb = DynamoDBDocumentClient.from(new DynamoDBClient({}), {
  marshallOptions: { removeUndefinedValues: true }, // undefined attributes would otherwise throw
});

const MAX_BATCH = 25; // BatchWriteItem limit: 25 put or delete requests per call
const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));

interface BatchResult {
  written: number;
  failed: WriteRequest[]; // still unprocessed after every retry
  capacityUnits: number;
  calls: number;
}

/** Remove requests for a key already seen: two requests for one key reject the whole batch. */
function dedupe(requests: WriteRequest[], keyAttrs: string[]): WriteRequest[] {
  const byKey = new Map<string, WriteRequest>();
  for (const r of requests) {
    const source = "PutRequest" in r ? r.PutRequest.Item : r.DeleteRequest.Key;
    byKey.set(JSON.stringify(keyAttrs.map((k) => source[k])), r); // the last request for a key wins
  }
  return [...byKey.values()];
}

async function writeChunk(table: string, chunk: WriteRequest[], maxAttempts: number, result: BatchResult): Promise<void> {
  let pending = chunk;
  for (let attempt = 1; pending.length > 0; attempt++) {
    const res = await ddb.send(new BatchWriteCommand({
      RequestItems: { [table]: pending },
      ReturnConsumedCapacity: "TOTAL",
    }));
    result.calls++;
    result.capacityUnits += (res.ConsumedCapacity ?? []).reduce((sum, c) => sum + (c.CapacityUnits ?? 0), 0);
    const unprocessed = (res.UnprocessedItems?.[table] ?? []) as WriteRequest[];
    result.written += pending.length - unprocessed.length;
    pending = unprocessed;
    if (pending.length === 0) return;
    if (attempt >= maxAttempts) {
      result.failed.push(...pending);
      return;
    }
    // Exponential backoff with full jitter: wait a random time between 0 and 50 ms x 2^attempt, capped at 5 s.
    await sleep(Math.random() * Math.min(5_000, 50 * 2 ** attempt));
  }
}

async function batchWriteAll(
  table: string,
  requests: WriteRequest[],
  options: { keyAttrs: string[]; concurrency?: number; maxAttempts?: number },
): Promise<BatchResult> {
  const { keyAttrs, concurrency = 4, maxAttempts = 8 } = options;
  const unique = dedupe(requests, keyAttrs);
  const chunks: WriteRequest[][] = [];
  for (let i = 0; i < unique.length; i += MAX_BATCH) chunks.push(unique.slice(i, i + MAX_BATCH));

  const result: BatchResult = { written: 0, failed: [], capacityUnits: 0, calls: 0 };
  let next = 0;
  const worker = async () => {
    while (next < chunks.length) {
      const chunk = chunks[next++];
      await writeChunk(table, chunk, maxAttempts, result);
    }
  };
  await Promise.all(Array.from({ length: Math.min(concurrency, chunks.length) }, worker));
  return result;
}

// Example: import 1,000 orders, then delete two of them.
async function main(): Promise<void> {
  const table = process.env.TABLE_NAME ?? "Orders";
  const orders = Array.from({ length: 1_000 }, (_, i) => ({
    customerId: `C#${(i % 50).toString().padStart(3, "0")}`,
    orderId: `O#${i.toString().padStart(5, "0")}`,
    total: Math.round(Math.random() * 20_000) / 100,
    status: "PLACED",
    note: undefined, // dropped by removeUndefinedValues
  }));
  const puts: WriteRequest[] = orders.map((Item) => ({ PutRequest: { Item } }));
  const put = await batchWriteAll(table, puts, { keyAttrs: ["customerId", "orderId"] });
  console.log(`Put ${put.written} items in ${put.calls} calls, ${put.capacityUnits} WCU, ${put.failed.length} failed`);

  const deletes: WriteRequest[] = [
    { DeleteRequest: { Key: { customerId: "C#000", orderId: "O#00000" } } },
    { DeleteRequest: { Key: { customerId: "C#001", orderId: "O#00001" } } },
  ];
  const del = await batchWriteAll(table, deletes, { keyAttrs: ["customerId", "orderId"] });
  console.log(`Deleted ${del.written} items, ${del.failed.length} failed`);
  if (put.failed.length + del.failed.length > 0) process.exitCode = 1;
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});

npm install @aws-sdk/client-dynamodb @aws-sdk/lib-dynamodb
npm install --save-dev tsx typescript @types/node
AWS_PROFILE=dev AWS_REGION=eu-west-1 TABLE_NAME=Orders npx tsx batch-write.ts
# Put 1000 items in 40 calls, 1000 WCU, 0 failed
# Deleted 2 items, 0 failed

The output is illustrative: 1,000 items need 40 chunks, and here no chunk came back with unprocessed items. On a busy provisioned table you’d see more calls than chunks, one per retry. The lib-dynamodb README in the aws-sdk-js-v3 repository lists the other marshalling options, such as convertClassInstanceToMap.

Why do UnprocessedItems need backoff and jitter?

The SDK already retries whole requests that fail with throttling errors, using the retry strategy described in the guide to configure retry and timeout settings in AWS SDK for JavaScript v3. Unprocessed items are different: the call succeeds with HTTP 200, so the SDK sees nothing to retry. DynamoDB returns them when a table or partition runs out of capacity, and the API reference strongly recommends exponential backoff, because an immediate retry hits the same throttled partition.

Jitter matters when several workers retry at once. Without it, they all wait the same time and collide again. The module waits a random time between zero and a cap that doubles each attempt, up to 5 seconds, which spreads retries out. Kinesis PutRecords has the same partial-failure shape, as the guide to write to Kinesis Data Streams with AWS SDK v3 shows.

What does a batch write cost in capacity?

Batching saves round trips, not capacity. Each put or delete consumes the same write capacity as the single-item call: for a standard table, 1 write unit per 1 KB of item size, rounded up, plus writes to every global secondary index the item projects into. A 2.5 KB item costs 3 write units whether you send it alone or in a batch of 25.

Set ReturnConsumedCapacity: "TOTAL", as the module does, to see the real number. On a provisioned table, a big import can exhaust write capacity and push work into UnprocessedItems; raise capacity for the import or slow the writers down. The script to find overprovisioned DynamoDB read and write capacity tells you whether you can lower it again afterwards.

Which IAM permissions does BatchWriteItem need?

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "BatchWriteOrders",
      "Effect": "Allow",
      "Action": "dynamodb:BatchWriteItem",
      "Resource": "arn:aws:dynamodb:eu-west-1:123456789012:table/Orders"
    }
  ]
}

One action covers both puts and deletes in the batch. To restrict which items a caller may write, add a dynamodb:LeadingKeys condition. The guide to find the IAM actions your AWS SDK for JavaScript code needs explains how commands map to actions, and the IAM policy generator for TypeScript code drafts a policy from the module above.

Troubleshooting and common mistakes

• ValidationException: Provided list of item keys contains duplicates. Two requests in one call share a key. Deduplicate per chunk, as dedupe() does.
• ValidationException mentioning “Member must have length less than or equal to 25”. A chunk has more than 25 requests. Check the chunk size and that you didn’t merge tables into one chunk.
• Pass options.removeUndefinedValues=true. An item has an undefined attribute. Set the marshall option or clean the data.
• Items missing after a “successful” run. The code ignored UnprocessedItems. Log failed and exit non-zero, as the example does.
• Items overwritten unexpectedly. A put replaces the whole item. If you meant to change one attribute, use UpdateCommand.
• ResourceNotFoundException. Wrong table name or Region. The client’s Region must match the table’s.

Limits: what BatchWriteItem can’t do

• It can’t update attributes, check conditions or return the previous item.
• It isn’t all-or-nothing. If a partial write would leave your data inconsistent, use TransactWriteCommand instead.
• It doesn’t reduce write capacity or cost, only round trips.
• It writes to one account and Region per client. For reads in bulk, BatchGetCommand has similar rules; for reading back by key, see the guide to query DynamoDB with AWS SDK v3: keys, indexes and pagination.

Before a large overwrite or delete, turn on point-in-time recovery so a bad import can be undone; the script to enable DynamoDB point-in-time recovery on every table does it in bulk. Porting an existing loader from Python? The guide to port a Python boto3 script to Node.js with AWS SDK v3 maps boto3’s batch_writer to the approach above.

Frequently asked questions

Related guides

• Query DynamoDB with AWS SDK v3Read the items back by key and index.
• DynamoDB UpdateItem with AWS SDK v3Conditions, counters and transactions.
• Configure retries and timeouts in SDK v3What the SDK already retries for you.
• Write to Kinesis with PutRecordsAnother batch API with partial failures.
• Enable DynamoDB point-in-time recoveryUndo a bad bulk write.
• Migrate from AWS SDK v2 to v3Move batchWrite calls to the v3 clients.