How can I convert CSV to JSON with data type in python or node.js?

Question

I wanna convert CSV to JSON correct data type

csv file 2nd row is data type. data has over 300 properties example data:

Name	DMG	HP	Human
string	number	number	boolean
knight	100	500	true
archer	50	200	true
dog	-	-	-

if string empty return null

if number empty return 0

if boolean empty return false

my node.js code:

const fs = require('fs')
const papa = require("papaparse")

const results = [];

const options = { header: true, dynamicTyping: true };

fs.createReadStream("characters.csv")
    .pipe(papa.parse(papa.NODE_STREAM_INPUT, options))
    .on("data", (data) => {
        results.push(data);
    }).on("end", () => {
        console.log(results)
    })

output I expecting:

[
    {
        "Name": "knight",
        "DMG": 100,
        "HP": 500,
        "Human": true,
    },
    {
        "Name": "archer",
        "DMG": 50,
        "HP": 200,
        "Human": true,
    },
    {
        "Name": "dog",
        "DMG": 0,
        "HP": 0,
        "Human": false,
    },
]

Does this answer your question? How to convert csv to json in python? — iamjaydev
– iamjaydev, Commented Nov 16, 2022 at 5:41
updated answer to support 300 or 300,000 columns. this should do the trick. Also updated the typecast function to send 0 if the unary produces NaN — JoeCrash
– JoeCrash, Commented Nov 16, 2022 at 6:41

JoeCrash · Accepted Answer · 2022-11-16 06:43:49Z

1

Options options..

in this approach I cached the headerTypes and made a small helper function to return the intended type

define the vars let i = 0, headerTypes = {};

replace your on.data code with this

.on("data", (data) => {
    if(i > 0){
       for(let prop in data){
            if(data.hasOwnProperty(prop)){
                const value = data[prop];
                data[prop] = typecast(headerTypes[prop], value);
            }
        }
    }else{
        //save types from row 0
        headerTypes = data;
    }
    i++;
})

add this helper function

function typecast(type = '', value){
    switch(type){
      case "number":
        return +value || 0; //unary for int or float
      case "boolean":
        return value === !0; //typecast to Boolean will let any filled string like '-' be true, do this instead.
      case "string":
      default:
        return String(value);
    }
}

edited Nov 16, 2022 at 6:43

answered Nov 16, 2022 at 6:30

JoeCrash

5322 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

JoeCrash Over a year ago

now instead of writing these properties out like I did, you can loop through the properties instead

JoeCrash Over a year ago

this should work for multiple rows without having to define them. just add types to typecast as needed.

JameyPlay Over a year ago

ok if it's work I'll be back and accept your answer thanks

JoeCrash · Accepted Answer · 2022-11-16 06:01:10Z

0

Try overriding the properties in the on.data event like so

let i = 0; //define this outside, its only to skip that first row with headers.
if(i > 0){ //skipping first row
    data.DMG = parseInt(data.DMG) || 0; //an int or 0
    data.HP = parseInt(data.HP) || 0; //an int or 0
    data.Human = data.Human === !0; //!0 = not false, will match true string or bool
    results.push(data);
}
i++;//iterate

const fs = require('fs')
const papa = require("papaparse")

const results = [];

const options = { header: true, dynamicTyping: true };
let i = 0;
fs.createReadStream("characters.csv")
    .pipe(papa.parse(papa.NODE_STREAM_INPUT, options))
    .on("data", (data) => {
        //add this here
        if(i > 0){
            data.DMG = parseInt(data.DMG) || 0; //an int or 0
            data.HP = parseInt(data.HP) || 0; //an int or 0
            data.Human = data.Human === !0; //!0 = not false, will match true string or bool
            results.push(data);
        }
        i++;
        //end add this here
    }).on("end", () => {
    console.log(results)
})

answered Nov 16, 2022 at 6:01

JoeCrash

5322 silver badges9 bronze badges

3 Comments

JameyPlay Over a year ago

thanks. but my data have over 300 properties it's could be difficult to write whole 300 props. do you have another way?

JoeCrash Over a year ago

see 2nd answer, I got bored :)

JameyPlay Over a year ago

any idea to directly check type in 2nd row?

Collectives™ on Stack Overflow

How can I convert CSV to JSON with data type in python or node.js?

2 Answers 2

3 Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related