Node.js Mongodb Callback issue

Question

My app should update if tmx is newer, if older do nothing and if doesn't exist insert the document. If the document is inserted, it works perfectly, else it doesn't update properly or says E11000 dup key. trying to figure out if my callback are wrong or the logic. (I'm new to node.js+mongodb) MongoClient = require('mongodb').MongoClient, assert = require('assert'), url = 'mongodb://localhost:27017/pfc';

    MongoClient.connect(url, function (err, db) {
        run(db);
    });


    function run(db) {
        fs.readFile('log.log', 'utf8', function (err, source) {
            if (err) throw err;
            var dataFile = JSON.parse(source);
            dataFile.forEach(function (item) {
                upsert(db, item, function (err, result) {
                    if (err) console.dir(err);

                });
            });
        })
    }

    function upsert(db, doc, callback) {

    db.collection('flags').findOne({vid: doc.vid}, function (err, item, result) {

        if (item.vid != null) {
            if (!(item.tmx instanceof Date)) {
                item.tmx = new Date(item.tmx)
            }
            if(!(doc.tmx instanceof Date)){
                doc.tmx = new Date(doc.tmx)
            }

            if (item.tmx < doc.tmx) {
                console.dir("Date validation")
                db.collection('flags').updateOne({vid: item.vid}, {
                        $set: {
                            "tmx": doc.tmx
                        }
                    },{upsert:true}, function (err, result) {
                        callback(err, result);

                    }
                )

                callback(err, result);
            }
            else{
                console.dir("older")
                callback(err, result);
            }
        }
        else {
            db.collection('flags').insertOne(doc, function(err, result) {
                callback(err, result);
            });
        }
    })}

Edit: The documents from the 'log.log' file have this structure:

{ vid:2848 tmx: "2015-07-18T23:56:17.000Z" }

{ vid: 2848 tmx: 2015-07-19T00:00:17.000Z }

collection.find({vid: doc.vid},function(err,item){

if(!item) // didnt find in collection, items with vid: 2848 insert doc to collection else if(item) //found an item with vid:2848 if (item.tmx < doc.tmx)//only update if doc.tmx is newer update collection with the most recent document

with @Aaron Dufour help I got rid of the callback problem, thanks :) but now the problem is when I have the collection already populated and go look for newest documents in log.log, it starts from the oldest document till the newest again :(

Your upsert logic is vulnerable to race conditions, which could be causing the issues. Have you tried Mongo's built-in upsert? — Aaron Dufour
– Aaron Dufour, Commented Aug 9, 2015 at 3:25
Can't see how to use it when I have to compare the dates to update — Telmo Ivo
– Telmo Ivo, Commented Aug 9, 2015 at 9:14
since if the item doesn't exist I get item.tmx property of null error, so I need to confirm the item exists so I can compare the date from the file with the date from mongodb — Telmo Ivo
– Telmo Ivo, Commented Aug 9, 2015 at 11:33
yes I have. the file has more than 60000 documents now and I have an average 40 documents with same vid — Telmo Ivo
– Telmo Ivo, Commented Aug 11, 2015 at 14:13

Aaron Dufour · Accepted Answer · 2015-08-09 13:33:32Z

2

Your upsert is vulnerable to race conditions, and run calls it many times in parallel, so that is probably the issue. It is not clear exactly what doc will look like, so you might need slightly more complicated logic, but here's a version that uses Mongo's upsert to make things a bit safer:

function upsert(db, doc, callback) {
  db.collection('flags').update({vid: doc.vid}, {$set: doc}, {upsert: true}, function(err) {
    db.collection('flags').update({vid: doc.vid, tmx: {$lt: doc.tmx}}, {$set: tmx: doc.tmx}, function(err) {
      callback();
    });
  });
}

answered Aug 9, 2015 at 13:33

Aaron Dufour

17.6k1 gold badge51 silver badges73 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Telmo Ivo Over a year ago

m8 you are a genius, saved my life :D

Telmo Ivo Over a year ago

found a little bug, For example, oldest record is from 07/08/2015 00:00:17 and newest is from today. If I run the application again, it will update to the oldest and go till the new again, the ways I tried didn't work. :( How to avoid this?

Aaron Dufour Over a year ago

@TelmoIvo I don't understand the problem you're having. Maybe edit it into the question so you have more room to explain it?

Telmo Ivo Over a year ago

added more details at the end of the question.

Collectives™ on Stack Overflow

Node.js Mongodb Callback issue

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related