Parallel execution of operations — JS: Asynchronous programming

Let's try to make several asynchronous calls at the same time and then make use of the result. Let's try to rewrite our file merging task. Here's the text:

Imagine we need to read the contents of two files and write them to a third one, merging files

From this statement, you can see that we can read both source files simultaneously. Once we've read them, we can write the contents to a new file:

fs.readFile('./first', 'utf-8', (error1, data1) => {
  // ?
});

fs.readFile('./second', 'utf-8', (error2, data2) => {
  // ?
});

Since our code is asynchronous, you can only get the result of each function inside the callbacks. Moreover, we can't know the order in the callbacks run in it because it all depends on which file is read faster.

To track the execution status of these operations, we have to introduce a global state, relative to these operations. We'll use it to track if the task is complete. Moreover, it helps to find out which we'll use to save data in.

We will write the old files to the new ones only when all operations are finished. In addition, we need to separate the data from the first and second files because writing to a new file, unlike reading, must take place in a certain order:

const state = {
  count: 0,
  results: [],
};

fs.readFile('./first', 'utf-8', (error1, data1) => {
  state.count += 1;
  state.results[0] = data1;
});

fs.readFile('./second', 'utf-8', (error2, data2) => {
  state.count += 1;
  state.results[1] = data2;
});

When both operations are complete, the state will be filled with data and count will become 2. We'll tie our code to this condition :

import fs from 'fs';

const state = {
  count: 0,
  results: [],
};

const tryWriteNewFile = () => {
  if (state.count !== 2) {
    return; // guard expression
  }

  fs.writeFile('./new-file', state.results.join(''), (error) => {
    if (error) {
      return;
    }
    console.log('finished!');
  });
};

console.log('first reading was started');
fs.readFile('./first', 'utf-8', (error1, data1) => {
  console.log('first callback');
  if (error1) {
    return;
  }
  state.count += 1;
  state.results[0] = data1;
  tryWriteNewFile();
});

console.log('second reading was started');
fs.readFile('./second', 'utf-8', (error2, data2) => {
  console.log('second callback');
  if (error2) {
    return;
  }
  state.count += 1;
  state.results[1] = data2;
  tryWriteNewFile();
});

// One run
// node index.js
// first reading was started
// second reading was started
// second callback
// first callback
// finished!

// Another run
// node index.js
// first reading was started
// second reading was started
// first callback
// second callback
// finished!

Now we read the files in parallel, and finally see the advantage of simultaneous asynchronous operations. This program executes much faster than the synchronous version! Moreover, the larger the size of the files, the greater the difference.

The reading of the files happens in parallel, but it's worth noting that the work done by JavaScript itself is strictly sequential. Callbacks only start to run when the current call stack is empty. Moreover, it works in the same order in which the asynchronous operations are done. Parallel running does not mean that operations start and end at the same time.

Writing this sort of code every time is tedious, so it's better to use a library called async, which provides a set of ready-made abstractions to work in asynchronous code. It contains dozens of functions for a large number of tasks related to the ordering of asynchronous operations.

Below you can see an example of a solution to our problem using this library:

import { map } from 'async';
import fs from 'fs';

map(['./first', './second'], fs.readFile, (err1, results) => {
  if (err1) {
    return;
  }
  fs.writeFile('./new-file', results.join(''), (err2) => {
    if (err2) {
      return;
    }
    console.log('finished!');
  });
});

It is much better, right? ;) As you'll see later, you can go even further.

Recommended materials

1. async - library for working in asynchronous style