Uploading Large Files to IPFS: Chunking, Streaming, and Best Practices

Large files break the naive “read the whole thing into memory and POST it” upload pattern fast. A 2 GB video or dataset export will blow past request timeouts, spike memory on both client and server, and — if you’re base64-encoding the payload — add a third more bytes on the wire than the file actually contains. IPFS itself doesn’t have this problem internally, but your upload client does, and the two need different solutions.

This guide covers how IPFS handles large content under the hood (DAG chunking), and the concrete client-side techniques — streaming, compression, and CAR import — that keep large uploads fast and memory-safe.

Large File Upload Strategies Compared#

The right strategy depends almost entirely on file size and whether the content is already compressed. Small JSON payloads and thumbnails work fine with a direct call. Anything video-sized needs streaming, and anything dataset-sized benefits from CAR import.

How IPFS Chunks Large Content: The DAG#

IPFS never stores a large file as one opaque blob. On ingest, it splits content into fixed-size chunks (256 KB by default) and organizes them into a Merkle DAG — a tree of linked blocks where each parent node just holds the hashes of its children. The file’s CID is the hash of the root node.

This has two practical consequences for you as an API consumer:

1. Chunking is automatic and happens node-side. You never manually split a file into chunks yourself — the pinning service does it as content comes in.
2. Chunking doesn’t fix client-side memory or timeout problems. The DAG structure means IPFS can fetch and verify a file incrementally, but if your upload client still has to buffer the entire file before sending the request, you get none of that benefit. That’s what streaming solves.

Streaming Uploads Instead of Buffering#

For files in the tens-to-hundreds-of-MB range, avoid reading the whole file into memory and base64-encoding it in one pass. Stream it instead.

Node.js — stream a file with constant memory:

import { createReadStream } from 'fs';
import { fetch } from 'undici';

async function streamUpload(filePath, apiKey) {
  const stream = createReadStream(filePath, { highWaterMark: 1024 * 1024 }); // 1 MB chunks

  const response = await fetch('https://api.ipfs.ninja/upload/new', {
    method: 'POST',
    duplex: 'half',
    headers: {
      'Content-Type': 'application/octet-stream',
      'X-Api-Key': apiKey,
    },
    body: stream,
  });

  if (!response.ok) {
    throw new Error(`Upload failed: ${await response.text()}`);
  }

  return response.json();
}

streamUpload('./large-export.zip', 'bws_a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4')
  .then((result) => console.log('CID:', result.cid, 'URL:', result.uris.url));

curl — stream from disk without loading it into a shell variable:

curl -X POST https://api.ipfs.ninja/upload/new \
  -H "X-Api-Key: bws_a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4" \
  -H "Content-Type: application/octet-stream" \
  --data-binary @large-export.zip

Both examples avoid ever holding the full file as a single in-memory buffer. That’s the difference between an upload that scales to a few KB and one that scales to gigabytes.

Skip Base64 for Anything Over a Few MB#

IPFS.NINJA’s JSON upload path ({ content: "<base64>" }) is convenient for small payloads — thumbnails, JSON documents, config blobs — because it’s a single request with no multipart parsing. But base64 encoding inflates payload size by roughly 33%, and it requires the full file in memory before you can encode it at all.

Rule of thumb:

• Under ~5 MB: base64 JSON upload is fine, simplest code path.
• 5 MB – 1 GB: stream raw bytes (see above) instead of base64-encoding.
• Over 1 GB, or uploading many files/a directory as one unit: use CAR import (below) rather than a single streamed request.

Compress Before You Upload — Selectively#

Pre-compressing content before it hits IPFS reduces both transfer time and pinned storage:

import { createReadStream, createWriteStream } from 'fs';
import { createGzip } from 'zlib';
import { pipeline } from 'stream/promises';

async function compressBeforeUpload(inputPath, outputPath) {
  await pipeline(
    createReadStream(inputPath),
    createGzip({ level: 6 }),
    createWriteStream(outputPath)
  );
}

This only pays off for compressible formats — JSON, CSV, logs, uncompressed text, source archives. Skip it for content that’s already compressed (JPEG, MP4, ZIP, most binary formats) — you’ll burn CPU for little or no size reduction, and in some cases gzip output can even be slightly larger than the source.

CAR Import for Very Large Files and Directory Trees#

For files well beyond a few hundred MB, or when you need to pin an entire directory as a single content-addressed unit, build a CAR (Content Addressable aRchive) file client-side and import it directly. A CAR file already contains the DAG structure, so the pinning service doesn’t need to re-chunk the content on ingest — it just verifies and stores the blocks you send.

This is the right tool when:

• A single file exceeds what you’re comfortable streaming in one HTTP request.
• You’re uploading a directory tree (a static site build, a dataset with many files) and want one root CID for the whole tree.
• You’re migrating content that’s already chunked into a DAG elsewhere and want to preserve the exact same CIDs.

See the large uploads API reference and CAR import reference for the full request format.

Metadata-Only Pinning for Content Hosted Elsewhere#

Sometimes you don’t need IPFS.NINJA to store the bytes at all — you need a verifiable, content-addressed reference to something already hosted somewhere else (your own CDN, a data lake, cold storage). In that case, hash the content locally to compute its CID, then use POST /pin to register that CID without re-uploading the full payload:

curl -X POST https://api.ipfs.ninja/pin \
  -H "X-Api-Key: bws_a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4" \
  -H "Content-Type: application/json" \
  -d '{
    "cid": "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi",
    "description": "12GB training dataset, hosted on primary storage, pinned for integrity verification"
  }'

This keeps your primary storage bill on whatever’s cheapest for raw bytes at scale, while giving you the tamper-evidence and portability benefits of a CID.

Handling Timeouts on Large Uploads#

Even with streaming, very large files can exceed a single HTTP request’s timeout window. A few practical mitigations:

1. Increase client timeout explicitly — most HTTP clients default to something far too short (10–30s) for a multi-hundred-MB upload.
2. Retry with resumable logic where possible — for CAR imports, you can often re-send only the blocks that weren’t acknowledged rather than restarting the whole transfer.
3. Split at natural boundaries — for directory uploads, upload independent files/subtrees as separate requests instead of one giant request, then reference them together at the application layer.

Size Limits by Plan#

Every plan enforces a combined storage and file-count limit — check ipfs.ninja/pricing for current numbers before architecting around a specific ceiling, since limits are tied to your plan rather than a fixed per-file cap. If a single upload would push you over your plan’s storage limit, the API returns a structured 402 Payment Required response telling you exactly which dimension (storage vs. file count) you hit — check for that response and surface it to your users rather than treating it as a generic failure.

Ready to start pinning? Create a free account — 50 files, 1 GB storage, 2 GB bandwidth/mo. No credit card required.

About this article: This article was drafted by an AI assistant using IPFS.NINJA’s content generation workflow, then reviewed and approved by Nacho Coll. All code examples were verified against the live IPFS.NINJA API. If you spot an inaccuracy, please open an issue at https://github.com/ipfs-ninja/feedback. Read more about how we use AI in our content and meet the people behind IPFS.NINJA.

For related reading, see how to upload files to IPFS and the IPFS upload API tutorial.