When a Filter Goes Wrong: Adventures in Async/Await

As happens regularly in software development, I had some partially completed code for a Node.js script handed to me containing an enhancement to existing logic that I was to pick up and complete. I went upon my way adding the rest of the required changes so that the code could be tested. That’s when things started to get fishy. It was a data processing job so I didn’t have any idea for how long it would take or how demanding it would be. I started it up, thought it might take a few seconds or minutes and began browsing the web. My MacBook started to sound like a commercial airliner. Websites weren’t opening. Applications were getting slower. Something was very wrong. How could this script be so demanding?

I stopped the job and began my investigation. Nothing seemed out of place. I barely added anything. I ran it again. Same thing. I tried altering the logic I added. No luck. So, I did what you would do, I removed the changes and tested the original. Seemed fine. What could this possibly be? What was wrong with the additional code I was given? There wasn’t much too it. Fetch some data from the database. Manipulate it with a standard filter. Call an async method and await it… Async/await. Is it not waiting? Why wouldn’t it wait? Oh, no… It’s one of those bugs.

The following is a generic recreation of the changes that transpired in the code and my eventual revelation.

Scenario #1: Given an array of objects, filter down to a subset:

const values = [1, 2, 1, 2, 1, 2, 1, 2, 2, 2];
const results = values.filter(value => {
  return value == 1;
});

console.log(results);

Output:

[1, 1, 1, 1]

This works. Very common. Expected. If the value is 1, we keep it, otherwise it’s dropped.

Scenario #2: Given array of objects, filter down to a subset using a function:

const double = function(value) {
  return value * 2;
};

const values = [1, 2, 1, 2, 1, 2, 1, 2, 2, 2];
const results = values.filter(value => {
  let x = double(value);
  return x == 4;
});

console.log(results);

Output:

[2, 2, 2, 2, 2, 2]

This still works as expected and should not be a surprise with a standard function call. If twice the value is 4, we keep it, otherwise it’s dropped.

Scenario #3: Given array of objects, filter down to a subset using an asynchronous function:

const double = function(value) {
  return new Promise(resolve => {
    setTimeout(function() {
      resolve(value * 2);
    }, 5000);
  });
};

const values = [1, 2, 1, 2, 1, 2, 1, 2, 2, 2];
const results = values.filter(value => {
  var x = double(value);
  return x == 4;
});

console.log(results);

Output:

[]

This code shouldn’t work for filtering and didn’t work. If you are familiar with asynchronous code at all you knew that right away. The result of the function call is not going to return the value we want, it’s a Promise and its messing up our condition. The developer that was working on the code knew that as well. If you run this code yourself you will see the output is displayed almost immediately and then after some delay the program completes. This is because the asynchronous operations are still firing even thought the filter method has returned.

Scenario #4: Given array of objects, filter down to a subset using an asynchronous function with async/await applied:

const double = function(value) {
  return new Promise(resolve => {
    setTimeout(function() {
      resolve(value * 2);
    }, 5000);
  });
};

const values = [1, 2, 1, 2, 1, 2, 1, 2, 2, 2];
const results = values.filter(async value => {
  var x = await double(value);
  return x == 4;
});

console.log(results);

Output:

[1, 2, 1, 2, 1, 2, 1, 2, 2, 2];

Huh? That’s the original set of values. The delay is still present. The developer assumed, as I also willingly accepted at the time, that adding async/await was sufficient to wait on the Promise to resolve. It’s not as you can see from the output. It appeared to have a different impact altogether. Nothing I tried would get it to work. The code was being interpreted, but the calls were firing off as fast as they could not waiting for the completion of the last, crippling my machine. The async inside the filter seemed odd to me, but I wasn’t sure why. It was running after all. What followed was a considerable amount of random searching and exponentially more denial but I came to eventually accept that filter does not support async/await. It’s a synchronous function, so even my misguided attempt to add another await outside the filter just confirmed how naive I was. The interpreter finally generated an error.

Being a dead end, I regressed back to a less elegant solution that I describe next. By no means do I love it, but it is at least readable to us mere mortals. Happenstance, this was the last revision to the code as we changed our approach to the job and all this was all removed anyways. There’s likely a clean filter-like solution that uses async/await out there, somewhere, but that is left as an exercise for the reader.

Solution: Given array of objects, filter down to a subset using an asynchronous function with async/await applied inside a for loop:

const double = function(value) {
  return new Promise(resolve => {
    setTimeout(function() {
      resolve(value * 2);
    }, 5000);
  });
};

const values = [1, 2, 1, 2, 1, 2, 1, 2, 2, 2];
const results = [];

const job = async () => {
  for (let value of values) {
    if ((await double(value)) == 4) {
      results.push(value);
    }
  }
};

job().then(() => {
  console.log(results);
});

Output:

[ 2, 2, 2, 2, 2, 2 ]

Great, it finally works. A traditional for loop that builds up a secondary array. It slow and it makes me sad.

Why filter can’t come out of the box able to await passed in async functions, is beyond me, but the limitations are real. The biggest shock to me throughout this whole learning process though was that there was not a single tool that detected the misusage. ESLint didn’t identify it. Visual Studio Code had no input for me. I even installed WebStorm, the self-proclaimed “Smartest JavaScript IDE” while writing this and unfortunately no, nothing helpful there. So there’s only one thing I can proclaim with confidence to you now and forever: Test! Your! Code!