Reading Data In Chunks With Readable Streams

Reading Data In Chunks With Readable Streams

A Readable Stream Of Data From A Local File

In this example, a text file, called longfile.txt, gets read in chunks. The size of the file will impact the number of chunks that gets read.
This uses a native readable stream, createReadStream, to read data from a file one chunk at a time.


// 
// dependencies
// 
const { createReadStream } = require('fs');

// 
// variables
// 
const THE_FILE_TO_READ = './longfile.txt';
const EVENTS = {
  DATA: 'data',
  END: 'end',
}
let howManyChunks = 1;

// 
// functional logic
// 
const readStream = createReadStream(THE_FILE_TO_READ);

function readChunk(data) {
  console.log(`got chunk # ${${howManyChunks}}`, data);
  howManyChunks++;
  console.log('\n');
}

function endReading() {
  console.log(' finished reading');
}

readStream.on(EVENTS.DATA, readChunk);
readStream.on(EVENTS.END, endReading);

Creating a Readable Stream Object From Scratch

Node comes with a readable stream factory function that can be used to create a "lower level" readable stream. The above example is connected to the fs module, but this Readable function is more interpretable to a developers' needs:

const { Readable } = require('stream');

const madeUpData = ['some', 'data', 'to', 'read'];

const createReadStream = () => {
  return new Readable({
    read() {
      console.log('---Readable fn---')
      
      // 
      // note: the "this" is the readable instance
      // 
      if (madeUpData.length === 0) this.push(null);
      else this.push(madeUpData.shift());
    },
  });
};

function readData(data) {
    console.log('got data', data.toString());
}

function endData() {
  console.log('finished reading');
}
const readable = createReadStream();
readable.on('data', readData);
readable.on('end', endData);

that will log...

---Readable read fn---
---Readable read fn---
got data <Buffer 73 6f 6d 65>
---Readable read fn---
got data <Buffer 64 61 74 61>
---Readable read fn---
got data <Buffer 74 6f>
---Readable read fn---
got data <Buffer 72 65 61 64>
finished reading

Creating a Readable stream instance with encoding

Notice the note on the setEncoding property of the Readable function arg object property:

const { Readable } = require('stream');

const madeUpData = ['this','is', 'an', 'array', 'of', 'strings'];

const createReadStream = () => {
  return new Readable({
    encoding: 'utf8',
    read() {
      console.log('---Readable read fn---');
      if (madeUpData.length === 0) this.push(null);
      else this.push(madeUpData.shift());
    },
  });
};

function readData(data) {
  console.log('got data', data);
}

function endData() {
  console.log('finished reading');
}
const readable = createReadStream();
readable.on('data', readData);
readable.on('end', endData);

A Readable Stream Just For JavaScript

objectMode is a property of Readable streams that we as developers can use to build the stream for parsing js objects using the stream + chunk interface.

const { Readable } = require('stream');
const madeUpData = ['this', 'is', 'an', 'array', 'of', 'strings'];

const createReadStream = () => {
  return new Readable({
    objectMode: true,
    read() {
      console.log('---Readable.read()');
      if (madeUpData.length === 0) this.push(null);
      else this.push(madeUpData.pop());
    },
  });
};

const dataFn = (data) => {
  console.log('got data', data);
};
const endFn = () => {
  console.log('finished reading');
};

const readable = createReadStream();
readable.on('data', dataFn);
readable.on('end', endFn);

That will output...

---Readable.read()
---Readable.read()
got data strings
---Readable.read()
got data of
---Readable.read()
got data array
---Readable.read()
got data an
---Readable.read()
got data is
---Readable.read()
got data this
finished reading

A Readable Stream with less code

The Readable constructor function includes a from function that takes some data and uses a different api for devs. This ends up feeling like a more concise bit of code than the above Readable constructor being writting into our own function where a custom implementation of the read() function is also required.

const { Readable } = require('stream');
const madeUpData = [13,26,43,96];

const dataFn = (data) => {
  console.log(`MATH: ${data} * 3 = ${data * 3}`);
};
const endFn = () => {
  console.log('finished reading');
};

const streamObj = Readable.from(madeUpData);
streamObj.on('data',dataFn);
streamObj.on('end', endFn);

Tags:

node

streams

core