How to check for duplicate data sets in JSON payload using JavaScript?

Question

I would like to find duplicate data sets from below payload using combination of 'NAME', 'ID'. If a set exist more than 3 times, I need to return NAME, ID of duplicated data set.

{
"Test": "1",
"value": [
    {
        "NAME": "ABCD",
        "ID": "1234",
        "ACTIVE": "true"
    },
    {
        "NAME": "EFGH",
        "ID": "5678",
        "ACTIVE": "true"
    },
    {
        "NAME": "EFGH",
        "ID": "5678",
        "ACTIVE": "true"
    },
    {
        "NAME": "EFGH",
        "ID": "5678",
        "ACTIVE": "true"
    },
    {
        "NAME": "ABCD",
        "ID": "1234",
        "ACTIVE": "true"
    },
    {
        "NAME": "ABCD",
        "ID": "1234",
        "ACTIVE": "true"
    },
    {
        "NAME": "IJKL",
        "ID": "91011",
        "ACTIVE": "true"
        }
    ]
}

Expected output:

["ABCD:1234", "EFGH:5678"]

It would probably be good to include an example of what you expect the final result to be. — Adam Harte
– Adam Harte, Commented Oct 1, 2021 at 4:32

userDuR · Accepted Answer · 2021-10-02 00:58:02Z

1

Try this. You can improve this further by performance wise, if you work around a bit.

  var data = {
 "Test": "1",
 "value": [
    {
        "NAME": "ABCD",
        "ID": "1234",
        "ACTIVE": "true"
    },
    {
        "NAME": "ABCD",
        "ID": "1234",
        "ACTIVE": "true"
    },
    {
        "NAME": "ABCD",
        "ID": "1234",
        "ACTIVE": "true"
    },
    {
        "NAME": "EFGH",
        "ID": "5678",
        "ACTIVE": "true"
    },
    {
        "NAME": "IJKL",
        "ID": "91011",
        "ACTIVE": "true"
    }
   ]
 };
 var objArray = data.value;
 var duplicates = []; // duplicates will be stored here.
 for(var i=0, iLen = objArray.length; i<iLen;i++){
    var obj = objArray[i];
        var filtered = objArray.filter(function(arrVal) {
        return arrVal.NAME === obj.NAME && arrVal.ID === obj.ID ;
    });
    var dup = obj.NAME + ":" + obj.ID;
    if(filtered.length>=3 && duplicates.indexOf(dup) < 0) {
        duplicates.push(dup);
    }

 }

edited Oct 2, 2021 at 0:58

answered Oct 1, 2021 at 5:11

userDuR

4187 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Rick Over a year ago

thanks for the snippet. i have update the original question, is there is way to achieve it by making changes to original snippet from you?

userDuR Over a year ago

I edited the answer for that. Please check!

fra · Accepted Answer · 2021-10-01 04:41:34Z

0

this questuon was answered multiple times on SO. You create array from the json and compare the elements of the array.

How to remove all duplicates from an array of objects?

answered Oct 1, 2021 at 4:41

fra

1861 silver badge13 bronze badges

3 Comments

Rick Over a year ago

My goal is to find the duplicate set.

fra Over a year ago

yes, so you create array of objects from the json and compare the objects of that array to find the duplicates. There are millions of ways to do that and on that link you find at least 10 of them.

Rick Over a year ago

I am not good with coding. Looking if someone provide me a snippet.

Medo Paw · Accepted Answer · 2021-10-02 01:24:38Z

According to your sample output I believe it's "at least 3 times" rather than "more than 3 times".

Below snippet can product the expected output with the sample data.

const data = {
"Test": "1",
"value": [
    {
        "NAME": "ABCD",
        "ID": "1234",
        "ACTIVE": "true"
    },
    {
        "NAME": "EFGH",
        "ID": "5678",
        "ACTIVE": "true"
    },
    {
        "NAME": "EFGH",
        "ID": "5678",
        "ACTIVE": "true"
    },
    {
        "NAME": "EFGH",
        "ID": "5678",
        "ACTIVE": "true"
    },
    {
        "NAME": "ABCD",
        "ID": "1234",
        "ACTIVE": "true"
    },
    {
        "NAME": "ABCD",
        "ID": "1234",
        "ACTIVE": "true"
    },
    {
        "NAME": "IJKL",
        "ID": "91011",
        "ACTIVE": "true"
        }
    ]
};

const occurrence = {}; // key: count
const result = [];
for (const item of data.value) {
  const key = `${item.NAME}:${item.ID}`;
  occurrence[key] = (occurrence[key] || 0) + 1;
  if (occurrence[key] >= 3) {
    result.push(key);
  }
}

console.log(result);

Collectives™ on Stack Overflow

How to check for duplicate data sets in JSON payload using JavaScript?

3 Answers 3

2 Comments

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest