node.js retrieve http csv file and load into mongoose

Question

I'm very new to coding in general, so I apologize ahead of time if this question should be rather obvious. Here's what I'm looking to do, and following that I'll post the code I've used so far.

I'm trying to get gzip'd csv rank data from a website and store it into a database, for a clan website that I'm working on developing. Once I get this figured out, I'll need to grab the data once every 5 minutes. The grabbing the csv data I've been able to accomplish, although it stores it into a text file and I need to store it into mongodb.

Here's my code:

var DB        =    require('../modules/db-settings.js');
var http      =    require('http');
var zlib      =    require('zlib');
var fs        =    require('fs');
var mongoose  =    require('mongoose');
var db          =   mongoose.createConnection(DB.host, DB.database, DB.port, {user: DB.user, pass: DB.password});

var request = http.get({ host: 'www.earthempires.com',
                     path: '/ranks_feed?apicode=myapicode',
                     port: 80,
                     headers: { 'accept-encoding': 'gzip' } });
request.on('response', function(response) {
  var output = fs.createWriteStream('./output');

  switch (response.headers['content-encoding']) {
    // or, just use zlib.createUnzip() to handle both cases
    case 'gzip':
      response.pipe(zlib.createGunzip()).pipe(output);
      break;
    default:
      response.pipe(output);
      break;
  }
});

db.on('error', console.error.bind(console, 'connection error:'));
db.once('open', function callback () {
  var rankSchema = new mongoose.Schema({
    serverid: Number,
    resetid: Number,
    rank: Number,
    countryNumber: Number,
    name: String,
    land: Number,
    networth: Number,
    tag: String,
    gov: String,
    gdi: Boolean,
    protection: Boolean,
    vacation: Boolean,
    alive: Boolean,
    deleted: Boolean
  })
});

Here's an example of what the csv will look like(first 5 lines of file):

9,386,1,451,Super Kancheong Style,22586,318793803,LaF,D,1,0,0,1,0
9,386,2,119,Storm of Swords,25365,293053897,LaF,D,1,0,0,1,0
9,386,3,33,eug gave it to mak gangnam style,43501,212637806,LaF,H,1,0,0,1,0
9,386,4,128,Justpickupgirlsdotcom,22628,201606479,LaF,H,1,0,0,1,0
9,386,5,300,One and Done,22100,196130870,LaF,H,1,0,0,1,0

My question is how do I get that csv data into mongodb? I'm not sure on where I should look next or what to do next. I'm sorta just stuck. — thtsigma
– thtsigma, Commented Nov 29, 2012 at 16:04
That's a pretty broad question. You'd parse the CSV into objects that follow rankSchema and then insert them into your collection using the Mongoose APIs. You're going to have to take the time to learn about how to do those steps as there aren't canned solutions. — JohnnyHK
– JohnnyHK, Commented Nov 29, 2012 at 16:13
Alright, well that's sorta what I needed to know. I wasn't sure if what I'm looking for is something that was common or not. I appreciate your kindness and your advice. Thank you. Should I just delete this question? — thtsigma
– thtsigma, Commented Nov 29, 2012 at 16:15
Up to you. If you found it useful someone else might as well. — JohnnyHK
– JohnnyHK, Commented Nov 29, 2012 at 16:27
If you write some code to parse the CSV into a model that you insert using mongoose, feel free to answer your own question so anyone else stumbling here will see how you did it. — Chad
– Chad, Commented Nov 29, 2012 at 16:47

theabraham · Accepted Answer · 2012-12-09 18:30:42Z

Hope it's not too late to help, but here's what I'd do:

Request the CSV formatted data and store it in memory or a file.
Parse the CSV data to convert each row into an object.
For each object, use Model.create() to create your new entry.

First, you need to create a model from your Schema:

var Rank = db.model('Rank', rankSchema);

Then you can parse your block of CSV text (whether you read it from a file or do it directly after your request is up to you.) I created my own bogus data variable since I don't have access to the api, but as long as your data is a newline delimited block of CSV text this should work:

/* Data is just a block of CSV formatted text. This can be read from a file                                                                                                  
   or retrieved right in the response. */                                                                                                                                    
var data = '' +                                                                                                                                                              
    '9,386,1,451,Super Kancheong Style,22586,318793803,LaF,D,1,0,0,1,0\n' +                                                                                                  
    '9,386,2,119,Storm of Swords,25365,293053897,LaF,D,1,0,0,1,0\n' +                                                                                                        
    '9,386,3,33,eug gave it to mak gangnam style,43501,212637806,LaF,H,1,0,0,1,0\n' +                                                                                        
    '9,386,4,128,Justpickupgirlsdotcom,22628,201606479,LaF,H,1,0,0,1,0\n' +                                                                                                  
    '9,386,5,300,One and Done,22100,196130870,LaF,H,1,0,0,1,0\n';                                                                                                            

data = data.split('\n');                                                                                                                                                     

data.forEach(function(line) {                                                                                                                                                
    line = line.split(',');   

    if (line.length != 14)
        return;                                                                                                                                               

    /* Create an object representation of our CSV data. */                                                                                                                   
    var new_rank = {                                                                                                                                                         
        serverid: line[0],                                                                                                                                                   
        resetid: line[1],                                                                                                                                                    
        rank: line[2],                                                                                                                                                       
        countryNumber: line[3],                                                                                                                                              
        name: line[4],                                                                                                                                                       
        land: line[5],                                                                                                                                                       
        networth: line[6],                                                                                                                                                   
        tag: line[7],                                                                                                                                                        
        gov: line[8],                                                                                                                                                        
        gdi: line[9],                                                                                                                                                        
        protection: line[10],                                                                                                                                                
        vacation: line[11],                                                                                                                                                  
        alive: line[12],                                                                                                                                                     
        deleted: line[13]                                                                                                                                                    
    };                                                                                                                                                                       

    /* Store the new entry in MongoDB. */                                                                                                                                    
    Rank.create(new_rank, function(err, rank) {                                                                                                                            
        console.log('Created new rank!', rank);                                                                                                                              
    });                                                                                                                                                                      
});

You could put this in a script and run it every 5-minutes using a cron job. On my Mac, I'd edit my cron file with crontab -e, and I'd setup a job with a line like this:

*/5 * * * * /path/to/node /path/to/script.js > /dev/null

Sorry for just now getting back to this. Its not too late, this is awesome--thanks for your help. I'll accept this answer as its the best i've seen. For some reason, it didn't occur to me to do split the lines myself and foreach through it. I've been looking and messing with a bunch of NPM packages that all needed tinkering with. Accepting the answer, its very much appreciated. The /* Store the new entry in MongoDB. */ is particularly helpful. I know I've seen examples of it in api docs and other pieces, but I wasn't quite able to grasp it. Thank you.

Collectives™ on Stack Overflow

node.js retrieve http csv file and load into mongoose

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related