Deep filter js object or array with Lodash extension

This tutorial is outdated, I still need to find some time to update it to the most recent deepdash version.

New sample comments app re-written with React and Redux is ready.

Here is a sandbox with it. And here is GitHub repo

The updated tutorial is coming soon…

Below is an old version of the text with updated versions of Codepens


Lodash method “_.filter” iterates over the top level only. If you want to filter nested data – “_.filterDeep” from the Deepdash will do the job.


Deepdash is an extension for the Lodash library. It mixes several traversal methods right into Lodash object and tries to follow API style of this great library. So if you are familiar with Lodash, you will feel ok with Deepdash too. 


This little tutorial will let you see, how to process tree-like structures easily.

We will build this simple comments app:

See the Pen Deepdash filter deep tutorial by Yuri Gor (@yurigor) on CodePen.0

And here is our: starting point – everything is ready, but filters still don’t work.
Let’s fork it and start to build filters one by one.

Take a look in the code:

  • getData – this function (at the bottom of the script) serves as a sample data source. It returns a tree-like array of comments with nested replies and sub-replies and.. you know, some more replies inside.
  • filterState – the object which accumulates state for all filters we will change.
  • filterComments – the main function in our tutorial. Now it just returns unfiltered data.
  • This Pen already configured to load Lodash and Deepdash libraries. Check Settings / JavaScript to see how.
  • Rest of the code is out of the scope of our tutorial. It’s pretty straightforward, and if you are interested and something is not clear for you – feel free to ask in the comments.

Let’s start coding!

First, we will make to work “Verified” filter. When the user switches it on, we set the field “filterState.verified” of the global “filterState” object to “true”.
All the comments will be re-rendered after that, so our task is to check “filterState” object if it has “verified” flag and to filter data accordingly. Let’s go to the “filterComments” function:

function filterComments(){
  var data = getData();
  var filtrate = data;
  if(!_.isEmpty(filterState)){     
     //will do filtering here
  }  
  
  return data;
} 

Our first check is Lodash’s function “_.isEmpty”. When the filter is off, the app does not set the flag to the “false” value, but removes the corresponding field from the state object with “delete”:

if(event.target.checked)
  filterState[id] = true;
else
  delete filterState[id];

This way we will have an empty “filterState” object in case of all the filters are off. Let’s check explicitly if the “verified” filter is on. If so – let’s filter data:

function filterComments(){
  var data = getData();
  var filtrate = data;
  if(!_.isEmpty(filterState)){
     if(filterState.verified){
      filtrate = _.filterDeep(filtrate,function(value,key){
        return key == 'verified' && value;
      });
    }
    return filtrate;
  }
  return data;
}

As you see, “filterDeep” function has source object and a callback function as arguments. The callback will be called for each node in the tree. It receives current value, field name, path, and parent arguments. Check docs for details.

Right now we are interested in the first two args: “key”, which should be ‘verified’ and “value”- it should equal “true”.
If criteria fulfilled – we return “true”. Else we will return “false”.

Note that in general, it’s a huge difference between returning “false” explicitly and returning “undefined”.
By default, “_.filterDeep” function invokes callback for “leaves” only. A node is considered “leaf” if it has no children.
But you can set “leafsOnly” option to “false” and in this case returning “false” explicitly will prevent iteration over the current node’s children.

Now try to click “Verified” filter…

It moves, but it’s not alive yet. Let’s see what’s happening. Add console.log(filtrate); right before first return statement:

[
  {
    "verified": true,
    "replies": [
      {
        "verified": true,
        "replies": [
          {
            "verified": true,
            "replies": [
              {
                "verified": true
              }
            ]
          }
        ]
      },
      {
        "verified": true
      }
    ]
  },
  {
    "replies": [
      {
        "verified": true,
        "replies": [
          {
            "replies": [
              {
                "verified": true
              }
            ]
          }
        ]
      }
    ]
  },
  {
    "replies": [
      {
        "verified": true
      }
    ]
  },
  {
    "verified": true
  }
]

We have exactly what we’ve asked for: all the “verified” fields with “true” value and all their parents. But we are missing the rest of the data, let’s add it with a second pass.

Now we need to collect all the leaves of each comment we have in the “filtrate” object, both of verified author and for the parent comments, even if it’s not written by a verified author. So let’s check each leaf in the source data object, and if its parent node exists in the previously collected “filtrate” object, then we need it:

function filterComments(){
  var data = getData();
  var filtrate = data;
  if(!_.isEmpty(filterState)){
     if(filterState.verified){
      filtrate = _.filterDeep(filtrate,function(value,key){
        return key == 'verified' && value;
      });
    }
    return _.filterDeep(data,function(value,key,path,depth,parent,parentKey,parentPath){
      return _.has(filtrate,parentPath);
    });
  }
  return data;
}

Let’s try it!

Now we have missing data, but something went wrong with the filter itself. There are unverified users still visible, while we are missing some correct nodes. Let’s debug.

Addconsole.log(_.indexate(data));
and console.log(_.indexate(filtrate));
before return filtered data.
“_.indexate” is another function from Deepdash, it builds flat “index” of the object, where keys are path strings and values are corresponding values. This method by default collect leaves only too.
So here are our indexes:

// data
{
  "[0].name": "Bob",
  "[0].text": "Perfect!",
  "[0].rating": 5,
  "[0].verified": true,
  "[0].replies[0].name": "Alice",
  "[0].replies[0].text": "Agree",
  "[0].replies[0].verified": false,
  "[0].replies[1].name": "admin",
  "[0].replies[1].text": "Thank you!",
  "[0].replies[1].verified": true,
  "[0].replies[1].replies[0].name": "Bob",
  "[0].replies[1].replies[0].text": "Write more!",
  "[0].replies[1].replies[0].verified": true,
  "[0].replies[1].replies[0].replies[0].name": "admin",
  "[0].replies[1].replies[0].replies[0].text": "Ok :)",
  "[0].replies[1].replies[0].replies[0].verified": true,
  "[0].replies[2].name": "Augusta",
  "[0].replies[2].text": "like a brillaint!11",
  "[0].replies[2].verified": true,
  "[1].name": "mr.John",
  "[1].text": "Well done..",
  "[1].rating": 4,
  "[1].verified": false,
  "[1].replies[0].name": "admin",
  "[1].replies[0].text": "Can it be better?",
  "[1].replies[0].verified": true,
  "[1].replies[0].replies[0].name": "mr.John",
  "[1].replies[0].replies[0].text": "May be last three lines can be shorter..",
  "[1].replies[0].replies[0].verified": false,
  "[1].replies[0].replies[0].replies[1].name": "Bob",
  "[1].replies[0].replies[0].replies[1].verified": true,
  "[1].replies[0].replies[0].replies[1].text": "Don't listen to him, it will be unreadable!",
  "[2].name": "Mark",
  "[2].rating": 5,
  "[2].text": "Any way to donate?",
  "[2].verified": false,
  "[2].replies[0].name": "Bill",
  "[2].replies[0].text": "+1",
  "[2].replies[0].verified": false,
  "[2].replies[1].name": "Larry",
  "[2].replies[1].text": "+1",
  "[2].replies[1].verified": true,
  "[3].name": "Regina",
  "[3].rating": 2,
  "[3].text": "Not really like it",
  "[3].verified": true,
  "[3].replies[0].name": "admin",
  "[3].replies[0].text": ":(",
  "[3].replies[0].verified": true
}
// filtrate
{
  "[0].verified": true,
  "[0].replies[0].verified": true,
  "[0].replies[0].replies[0].verified": true,
  "[0].replies[0].replies[0].replies[0].verified": true,
  "[0].replies[1].verified": true,
  "[1].replies[0].verified": true,
  "[1].replies[0].replies[0].replies[0].verified": true,
  "[2].replies[0].verified": true,
  "[3].verified": true
}

What’s wrong? We still see how Alice said “Agree”, but Alice is not the verified user. We can see in the data index: [0].replies[0].verified": false

let’s check what do we have in our filtrate index: [0].replies[0].verified": true. Why?

No, this is not Deepdash bug (but once it almost used to be). To understand what’s happening here, we should know a bit how deep filtering works.


Deepdash iterates over each path of the source object, and if the callback function says “true” it copies the current value to the result object by the exact same path.
This works fine, but result object may start to have sparse arrays if some element was rejected by the callback function. When some array’s element is missing, its place marked as “empty”.
In most cases, developers will not need such sparse arrays as a result, so by default “_.filterDeep” function applies _.condenseDeep function to the result object before return it.
Condensing array means removing empty slots from it. The array will be re-indexed, and values of such condensed array will be placed to the shifted positions with changed indexes.


This is a reason why we have incorrect comments tree – filtrate object contains mutated paths, so its paths do not correspond with source object anymore.

To avoid this situation, we can disable auto-condensing by passing option condense: false

The second issue in our code is Lodash “_.has” function.
It doesn’t consider the possibility of a sparse array and returns “true” for empty array slots. While Deepdash “_.exists” alternative function will work as we need.

function filterComments(){
  var data = getData();
  var filtrate = data;
  if(!_.isEmpty(filterState)){
     if(filterState.verified){
      filtrate = _.filterDeep(filtrate,function(value,key){
        return key == 'verified' && value;
      },{condense:false});
    }
   
    return _.filterDeep(data,function(value,key,path,depth,parent,parentKey,parentPath){
      return _.exists(filtrate,parentPath);
    });
  }
  return data;
}

Here is our working demo:

See the Pen Deepdash filter deep tutorial – verified filter by Yuri Gor (@yurigor) on CodePen.0

Filtering by keyword

Next, let’s implement text filter. You may already have noticed, that entering text into the filter field will highlight matching symbols in the comments.
But we still need to hide comments, where no keywords were found.

Currently, we use “verified” fields as key points in the result “filtrate” object. But now we need to consider another field “text”, and in case of both “verified” and “text” filters are active, we must consider them both at the same time. Semantically “text” is the most important field in the comments data. So let’s limit our callback to respond only to iterations over “text” field, and all the other fields will be checked via “parent” calback’s argument.

function filterComments(){
  var data = getData();
  var filtrate = data;
  if(!_.isEmpty(filterState)){
      filtrate = _.filterDeep(filtrate,function(value,key,path,depth,parent){
        if(key!=='text')
          return;
        var res = true;
        if(filterState.verified){
          res = res && parent.verified;
        }
        return res;
      },{condense:false});
    return _.filterDeep(data,function(value,key,path,depth,parent,parentKey,parentPath){
      return _.exists(filtrate,parentPath);
    });
  }
  return data;
}

Make sure “Verified” filter is still working.
Now it’s easy to add filtering by the second field.
If “filterState.text” exists, we check if the keyword is a substring of “text” field’s value:

function filterComments(){
  var data = getData();
  var filtrate = data;
  if(!_.isEmpty(filterState)){
     
      filtrate = _.filterDeep(filtrate,function(value,key,path,depth,parent){
        if(key!=='text')
          return;
        var res = true;
        if(filterState.verified){
          res = res && parent.verified;
        }
        if(filterState.text){
          res = res && _.includes(value.toLowerCase(),filterState.text);
        }
        return res;
      },{condense:false});
    
    return _.filterDeep(data,function(value,key,path,depth,parent,parentKey,parentPath){
      return _.exists(filtrate,parentPath);
    });
  }
  return data;
}
It works!

The last filter to build – is the filter by stars count. User can exclude comments with specific rating by pressing corresponding star button in the filter panel. Each star button will set “true” in the “filterState.s1..s5″ field.
Since we have rating on top level comments only, we don’t need deep filtering here, just Lodash will be enough.

First, we collect all the pressed star buttons to the array:

  var excludeStars = _.reduce(filterState,function(r,v,k){
    if(k.length==2&&k[0]=='s')r.push(parseInt(k[1]));
    return r;
  },[]);

Then, we remove comments with rating user wants to exclude:

  _.remove(data,function(v){return _.includes(excludeStars,v.rating)});

Finally we have:

function filterComments(){
  var data = getData();
  var excludeStars = _.reduce(filterState,function(r,v,k){
    if(k.length==2&&k[0]=='s')r.push(parseInt(k[1]));
    return r;
  },[]);
  _.remove(data,function(v){return _.includes(excludeStars,v.rating)});
  var filtrate = data;
  if(!_.isEmpty(filterState)){
     
      filtrate = _.filterDeep(filtrate,function(value,key,path,depth,parent){
        if(key!=='text')
          return;
        var res = true;
        if(filterState.verified){
          res = res && parent.verified;
        }
        if(filterState.text){
          res = res && _.includes(value.toLowerCase(),filterState.text);
        }
        return res;
      },{condense:false});
    
    return _.filterDeep(data,function(value,key,path,depth,parent,parentKey,parentPath){
      return _.exists(filtrate,parentPath);
    });
  }
  return data;
}

That’s it. All the filters work as we need. Here is a link to the final codepen

Any feedback or questions are highly appreciated.