Write to Kinesis Data Streams With AWS SDK v3 (PutRecords)

Photo by Mark Thompson on Unsplash

The last point is where most Kinesis producers lose data. PutRecords is not all-or-nothing: some records land and others are throttled, and the SDK doesn’t retry those because the HTTP call itself succeeded. This guide is for Node.js and TypeScript developers writing events, logs or clickstream data into Kinesis Data Streams who want a producer that doesn’t silently drop records.

You’ll get a typed module that covers PutRecord for single events, Kinesis PutRecords with AWS SDK v3 for batches, splitting at the request limits, and a retry loop for partial failures. If you’re choosing between messaging services, the guides to send and receive SQS messages with AWS SDK v3 and send events to EventBridge with PutEvents cover the queue and event-bus alternatives.

Prerequisites

• Node.js 18 or later, TypeScript and tsx, with "type": "module" in package.json.
• The @aws-sdk/client-kinesis package and an existing data stream in ACTIVE status.
• Credentials the SDK can resolve; the guide to AWS SDK v3 credential providers covers profiles, SSO and roles.

PutRecord or PutRecords?

The largest record a stream accepts is its MaxRecordSizeInKiB, which DescribeStreamSummary returns. UpdateMaxRecordSize sets it between 1,024 and 10,240 KiB on provisioned streams. Records between 1 and 10 MiB are meant to be occasional; Kinesis handles them with burst capacity.

How to call Kinesis PutRecords with AWS SDK v3, step by step

1. Create one client and reuse itnew KinesisClient({}) at module level, so connections are reused across calls.
2. Encode each recordData is a Uint8Array; TextEncoder turns JSON into bytes. The SDK base64-encodes it on the wire.
3. Choose a partition key per recordRecords with the same key go to the same shard, in order. Use the entity whose events must stay ordered, such as an order ID.
4. Split into batchesNo more than 500 records or 10 MiB per request, counting partition keys.
5. Send and inspect the responseThe Records array in the response lines up with the request, one result per record, success or not.
6. Retry only the failuresCollect entries with an ErrorCode, wait with exponential backoff and jitter, and send them again. Give up after a few attempts and report what’s left.

Example: a producer that retries partial failures

// kinesis-producer.ts
// Write to Kinesis Data Streams with AWS SDK for JavaScript v3: PutRecord for one record,
// PutRecords for batches (split at 500 records / 10 MiB), and a retry loop for the records
// that PutRecords reports as failed inside a successful response.
import { setTimeout as sleep } from "node:timers/promises";
import {
  KinesisClient,
  PutRecordCommand,
  PutRecordsCommand,
  type PutRecordsRequestEntry,
} from "@aws-sdk/client-kinesis";

const kinesis = new KinesisClient({}); // Region from AWS_REGION or the profile
const encoder = new TextEncoder();

const MAX_RECORDS_PER_CALL = 500;
const MAX_BYTES_PER_CALL = 10 * 1024 * 1024; // data + partition keys, per PutRecords request

export interface OutgoingRecord {
  partitionKey: string; // 1-256 characters; same key -> same shard -> ordered within that shard
  data: unknown; // serialized as JSON
}

/** One record, when you need its sequence number or strict per-key ordering. */
export async function putOne(streamArn: string, rec: OutgoingRecord, sequenceNumberForOrdering?: string) {
  const res = await kinesis.send(
    new PutRecordCommand({
      StreamARN: streamArn,
      PartitionKey: rec.partitionKey,
      Data: encoder.encode(JSON.stringify(rec.data)),
      SequenceNumberForOrdering: sequenceNumberForOrdering,
    }),
  );
  return { shardId: res.ShardId, sequenceNumber: res.SequenceNumber };
}

/** Split entries into PutRecords-sized batches: at most 500 records and 10 MiB each. */
function* batches(entries: PutRecordsRequestEntry[]): Generator<PutRecordsRequestEntry[]> {
  let batch: PutRecordsRequestEntry[] = [];
  let bytes = 0;
  for (const e of entries) {
    const size = (e.Data?.byteLength ?? 0) + encoder.encode(e.PartitionKey ?? "").byteLength;
    if (batch.length && (batch.length === MAX_RECORDS_PER_CALL || bytes + size > MAX_BYTES_PER_CALL)) {
      yield batch;
      batch = [];
      bytes = 0;
    }
    batch.push(e);
    bytes += size;
  }
  if (batch.length) yield batch;
}

export interface PutManyResult {
  written: number;
  failed: { partitionKey: string; errorCode: string; message: string }[];
  attempts: number;
}

/** Write many records, re-sending only the failed ones with exponential backoff and jitter. */
export async function putMany(streamArn: string, records: OutgoingRecord[], maxAttempts = 5): Promise<PutManyResult> {
  let pending: PutRecordsRequestEntry[] = records.map((r) => ({
    PartitionKey: r.partitionKey,
    Data: encoder.encode(JSON.stringify(r.data)),
  }));
  let written = 0;
  let attempts = 0;
  let lastErrors = new Map<PutRecordsRequestEntry, { code: string; message: string }>();

  while (pending.length && attempts < maxAttempts) {
    attempts++;
    const retry: PutRecordsRequestEntry[] = [];
    lastErrors = new Map();
    for (const batch of batches(pending)) {
      // A whole-request throttle (ProvisionedThroughputExceededException) is retried by the SDK itself.
      const res = await kinesis.send(new PutRecordsCommand({ StreamARN: streamArn, Records: batch }));
      // The response array lines up with the request array, one result per record.
      (res.Records ?? []).forEach((r, i) => {
        const entry = batch[i];
        if (!entry) return;
        if (r.ErrorCode) {
          retry.push(entry);
          lastErrors.set(entry, { code: r.ErrorCode, message: r.ErrorMessage ?? "" });
        } else {
          written++;
        }
      });
    }
    pending = retry;
    if (pending.length && attempts < maxAttempts) {
      const base = 100 * 2 ** attempts; // 200 ms, 400 ms, 800 ms ...
      await sleep(base / 2 + Math.random() * base); // jitter so producers don't retry in lockstep
    }
  }

  const failed = pending.map((e) => ({
    partitionKey: e.PartitionKey ?? "",
    errorCode: lastErrors.get(e)?.code ?? "Unknown",
    message: lastErrors.get(e)?.message ?? "",
  }));
  return { written, failed, attempts };
}

A caller that checks the stream, writes one event with PutRecord, then 1,200 events with PutRecords in three batches (500, 500 and 200):

// send-events.ts
// Usage: AWS_PROFILE=dev AWS_REGION=us-east-1 npx tsx send-events.ts arn:aws:kinesis:us-east-1:123456789012:stream/orders [count]
import { DescribeStreamSummaryCommand, KinesisClient } from "@aws-sdk/client-kinesis";
import { putMany, putOne, type OutgoingRecord } from "./kinesis-producer.js";

const streamArn = process.argv[2];
const count = Number(process.argv[3] ?? "1200");
if (!streamArn?.startsWith("arn:aws")) throw new Error("pass the stream ARN as the first argument");

// Check the stream before writing: mode, open shards and the largest record it accepts.
const { StreamDescriptionSummary: s } = await new KinesisClient({}).send(new DescribeStreamSummaryCommand({ StreamARN: streamArn }));
console.log(
  `${s?.StreamName}: ${s?.StreamModeDetails?.StreamMode ?? "PROVISIONED"} mode, ${s?.OpenShardCount} open shards, ` +
    `max record ${s?.MaxRecordSizeInKiB ?? "?"} KiB, status ${s?.StreamStatus}`,
);

// One record: returns the shard and sequence number.
const first = await putOne(streamArn, { partitionKey: "order-1000", data: { orderId: 1000, status: "created" } });
console.log(`PutRecord -> ${first.shardId} ${first.sequenceNumber}`);

// Many records, keyed by order ID so each order's events stay on one shard.
const events: OutgoingRecord[] = Array.from({ length: count }, (_, i) => ({
  partitionKey: `order-${1001 + (i % 400)}`,
  data: { orderId: 1001 + (i % 400), status: i < 400 ? "created" : "paid", at: new Date().toISOString() },
}));
const result = await putMany(streamArn, events);
console.log(`PutRecords -> ${result.written} written, ${result.failed.length} failed after ${result.attempts} attempt(s)`);
for (const f of result.failed.slice(0, 5)) console.log(`  ${f.partitionKey}: ${f.errorCode} ${f.message}`);
if (result.failed.length) process.exit(1);

npm install @aws-sdk/client-kinesis
npm install --save-dev tsx typescript @types/node
npm pkg set type=module

AWS_PROFILE=dev AWS_REGION=us-east-1 npx tsx send-events.ts arn:aws:kinesis:us-east-1:123456789012:stream/orders 1200
# orders: PROVISIONED mode, 4 open shards, max record 1024 KiB, status ACTIVE
# PutRecord -> shardId-000000000002 49655213771937519207421850837621376290371526584283185186
# PutRecords -> 1200 written, 0 failed after 2 attempt(s)

Output is illustrative. “2 attempts” means some records were throttled on the first pass and landed on the retry, which is normal during a burst. If failed isn’t empty after the last attempt, send those records to a fallback, such as an SQS queue with a dead-letter queue; the check to find SQS queues without a dead-letter queue makes sure that fallback has one.

Partition keys and hot shards

Kinesis hashes each partition key with MD5 (the algorithm in RFC 1321) into a 128-bit number, and each shard owns a range of those numbers. Keys are Unicode strings up to 256 characters. Every record with the same key lands on the same shard, which gives you per-key ordering and also a ceiling: one shard accepts up to 1 MB per second or 1,000 records per second of writes.

A hot shard is what happens when one key, or a few, carry most of the traffic: a tenant ID where one tenant is huge, a constant like "default", or a timestamp truncated to the second. That shard throttles while the others sit idle. Fixes, in order of preference:

• Use a higher-cardinality key that still keeps related events together, such as order ID instead of customer ID.
• Add a suffix such as tenant-42#3 with a small random number, and accept that ordering then holds only within each suffix.
• Use a random key such as crypto.randomUUID() when order doesn’t matter at all.
• Set ExplicitHashKey to pick the shard yourself, if you manage shard ranges deliberately.

On-demand streams split busy shards automatically, but they don’t isolate a single hot key: a key that exceeds one shard’s limits still throttles. Streams can also use an AUTO record distribution strategy, in which case Kinesis ignores your partition keys and spreads records itself.

On-demand or provisioned: what does it cost?

As of September 2026, the AWS Price List and Kinesis pricing page show these us-east-1 rates:

Worked example for a steady 2,000 records per second of 2 KB each, over a 30-day month (2,592,000 seconds), write side only:

• Provisioned: 4,000 KB/s needs 4 shards (4,000 ÷ 1,024 rounded up). Shards cost 4 × 720 hours × $0.015 = $43.20. Each record is one PUT payload unit, so 2,000 × 2,592,000 = 5,184 million units × $0.014 = $72.58. Total $115.78.
• On-demand Standard: 2,000 × 2 KB × 2,592,000 = 10,368,000,000 KB, or about 9,887.7 GB, × $0.08 = $791.02, plus 720 × $0.04 = $28.80 for the stream. Total $819.82 before read charges.

Steady, predictable traffic is cheaper on provisioned shards; spiky or unknown traffic is simpler on on-demand. You can switch a stream between the two modes twice in 24 hours without interrupting producers.

Permissions needed

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "WriteToOrdersStream",
      "Effect": "Allow",
      "Action": [
        "kinesis:PutRecord",
        "kinesis:PutRecords",
        "kinesis:DescribeStreamSummary"
      ],
      "Resource": "arn:aws:kinesis:us-east-1:123456789012:stream/orders"
    }
  ]
}

If the stream uses server-side encryption with a customer managed KMS key, the producer also needs kms:GenerateDataKey on that key. To check the list against your code, see how to find the IAM actions your AWS SDK JavaScript code calls, or paste the module into the free IAM policy generator for TypeScript.

Troubleshooting and common mistakes

• Records missing downstream, no errors logged. The code ignored FailedRecordCount. Check it on every response and retry the failed entries.
• ProvisionedThroughputExceededException in record results. A shard hit its write limit. Retry with backoff; if it keeps happening on one shard, fix the partition key; if it happens on all shards, add shards or switch to on-demand.
• ProvisionedThroughputExceededException thrown by send(). The whole request was throttled. The SDK retries these automatically; tune attempts with the guide to configure retries and timeouts in AWS SDK v3.
• InvalidArgumentException. More than 500 records, a request over 10 MiB, a record over the stream’s maximum size, or an empty partition key. The batching function guards the first two.
• ResourceNotFoundException. Wrong stream name or ARN, or the client is in another Region than the stream.
• AccessDeniedException or KMS errors. Missing kinesis:PutRecords or KMS permissions; the steps to troubleshoot AWS IAM access denied errors decode the message.

Limits of this approach

The module sends batches one after another from a single process. For very high throughput, run several producers, or aggregate many small records into one Kinesis record so you pay for fewer PUT payload units, which then needs de-aggregation on the consumer side. Retried records arrive after later ones, so if order matters per key, retry that key’s records before sending newer ones, or use PutRecord with SequenceNumberForOrdering. Kinesis doesn’t deduplicate: a retry after a timeout can write a record twice, so consumers should be idempotent. Once Amazon Data Firehose has delivered the stream to S3, you can run an Athena query with AWS SDK v3 and read the results to analyze it.

Watch WriteProvisionedThroughputExceeded in CloudWatch, and publish your own failure counts with the guide to publish custom CloudWatch metrics with SDK v3. Porting a v2 producer? The AWS SDK JavaScript v2 to v3 converter drafts the change, and the guide to migrate a Node.js app from SDK v2 to v3 covers the rest. To inspect a stream without writing code, ChatWithCloud answers questions in plain English using SDK v2 calls on your machine, as shown on how ChatWithCloud runs AWS SDK code.

Frequently asked questions

Related guides

• Send and receive SQS messages with SDK v3A queue instead of a stream.
• Publish an SNS message with SDK v3Fan out to many subscribers.
• Send events with EventBridge PutEventsRule-based event routing.
• AWS SDK v3 retries and timeoutsWhole-request throttling.
• Publish custom CloudWatch metricsCount dropped records.
• Monitor AWS service quota usageShard and stream limits.