Node.js Streams and Backpressure: Processing Data Without Running Out of Memory

The difference between a Node service that handles a 2 GB file on a 512 MB server and one that crashes trying is almost always streams. The naive approach reads the whole thing into memory, does something, and writes it out — and it works beautifully until the input is bigger than the memory you have. Streams process data in chunks as it flows, so memory stays flat regardless of how large the data gets. They are one of Node's most powerful features and one of its most avoided, usually because backpressure — the mechanism that keeps a fast producer from overwhelming a slow consumer — is misunderstood.

This is part of the Node.js runtime series. Here we cover the stream types, piping, and the backpressure that makes it all safe.

Why streams instead of buffering everything

Consider serving a large file. The obvious version reads it entirely, then sends it:

const data = await fs.readFile("huge-video.mp4") // entire file into memory
res.end(data)

If the file is 2 GB, this allocates 2 GB before sending a single byte, and a handful of concurrent requests exhausts your memory and takes the process down. The streaming version never holds more than a small chunk at a time:

fs.createReadStream("huge-video.mp4").pipe(res)

This reads the file in pieces and writes each piece to the response as it goes, so memory usage stays low and constant no matter the file size, and the client starts receiving data immediately instead of after the whole file is buffered. The principle generalizes: any time you process data that is large, unbounded, or arriving over time, streaming keeps memory flat where buffering grows without limit. This is the same "process it in pieces, do not hold it all" instinct behind message queues at the system level.

The four stream types

Node has four kinds of streams, and once you know them the API stops looking arbitrary.

• Readable streams are sources you read from: a file being read, an incoming HTTP request, process.stdin.
• Writable streams are destinations you write to: a file being written, an HTTP response, process.stdout.
• Duplex streams are both readable and writable, with the two sides independent — a TCP socket, which you both send to and receive from.
• Transform streams are duplex streams where the output is a transformation of the input: compression, encryption, or parsing, where bytes go in one side and modified bytes come out the other.

Streams are built on the EventEmitter pattern — a readable stream emits data events as chunks arrive and an end event when it is done. You can listen to those events directly, but you rarely should, because doing so by hand is where the backpressure bugs come from.

Piping connects streams safely

The idiomatic way to move data from a readable to a writable stream is pipe, which connects them and handles the flow control for you:

readable.pipe(transform).pipe(writable)

This reads from the source, passes chunks through the transform, and writes to the destination, and it manages the pace automatically. You can chain pipes to build a pipeline — read a file, gzip it, write it out — and each stage only receives data as fast as the next stage can accept it. That automatic pacing is backpressure, and it is the reason to prefer pipe (or the modern pipeline helper) over wiring up data events yourself.

Backpressure: the concept that makes streams safe

Here is the problem streams have to solve. A readable stream might produce data far faster than a writable stream can consume it — reading from a fast local disk and writing to a slow network socket, for instance. If the reader just keeps pushing chunks at the writer, the unconsumed chunks pile up in memory, and now you have reinvented the exact out-of-memory problem streams were supposed to prevent, just with extra steps.

Backpressure is the feedback signal that prevents this. When you write to a writable stream, its write() method returns a boolean: true means "keep going," false means "my buffer is full, pause." A properly behaved reader sees that false, stops producing, and waits for the writable stream to emit a drain event signaling it has caught up, then resumes. This back-and-forth keeps the amount of in-flight data bounded no matter how mismatched the two speeds are. The producer is throttled to the consumer's pace, so memory stays flat even when a fast source feeds a slow sink.

The good news is that pipe and pipeline implement this handshake for you. When you pipe a readable into a writable, Node automatically pauses the source when the destination's buffer fills and resumes it on drain. This is the single biggest reason to use piping rather than manual data event handlers: the moment you write your own on("data") loop that calls write() without checking its return value, you have disabled backpressure and reintroduced the memory blow-up. Let piping do it, and backpressure is handled correctly by default.

Use pipeline for error handling

One practical upgrade over raw pipe: the stream.pipeline function. Plain pipe does not forward errors well — if a stage fails mid-flow, you can leak file descriptors or leave streams half-open. pipeline wires the stages together with proper error propagation and cleanup:

const { pipeline } = require("node:stream/promises")

await pipeline(
  fs.createReadStream("input.txt"),
  zlib.createGzip(),
  fs.createWriteStream("output.txt.gz")
)
// resolves when done, rejects (and cleans up) if any stage fails

This gives you the memory safety of streaming, the automatic backpressure of piping, and error handling that does not leak resources — which is what you want in production. It ties directly into the error-handling patterns for the rest of your Node code.

When to reach for streams

The signal is data that is large, unbounded, or time-based. Big file uploads and downloads, processing a multi-gigabyte log or CSV, proxying data between services, real-time feeds, anything where the total size is unknown or could exceed memory — all of these are streams. If you ever find yourself reading an entire large thing into a variable before processing it, that is the moment to ask whether a stream would keep your memory flat instead. Streams have a reputation for being fiddly, but nearly all of that reputation comes from people handling data events manually and getting backpressure wrong. Use pipeline, let it manage the flow, and streams become the straightforward, memory-safe tool they were designed to be — a direct application of keeping the event loop free and memory bounded that defines good Node.