Javascript JSON parse problem with decoded POST request data

Question

I get base64 data as an answer to a POST request.

It's decoded the following way (based on the documentation of the REST API):

let buf = Buffer.from(base64, "base64");
buf = buf.slice(4);
    
let data = gunzipSync(buf).toString()
console.log(data) // -> {"Code":200,"Value":"8e286fdb-aad2-43c6-87b1-1c6c0d21808a","Route":""}
console.log(data.length) // -> 140 -> Seems weird? Shouldn't it be 70?

Problem:

console.log(JSON.parse(data)) -> SyntaxError: Unexpected token  in JSON at position 1

I tried to delete all white characters via replace(/\s/g,''), tried decoding with toString("utf8"), etc. Nothing helps. The only thing that could help is the weird wrong length described above.

Tomalak · Accepted Answer · 2021-02-26 08:02:20Z

1

Your buffer is UTF-16 encoded and contains \0 bytes, like {·"·C·o·d·e·"·=·… (with · representing \0), that's why it's double the expected length. The \0 bytes don't print when you output the buffer with console.log(), that's why the output seems to be correct.

Decode the buffer before JSON-parsing it.

var buffer = Buffer.from(base64, "base64");
var str = buffer.toString('utf16le');

console.log(str)                // -> {"Code":200,"Value":"8e286fdb-aad2-43c6-87b1-1c6c0d21808a","Route":""}
console.log(str.length)         // -> 70

console.log(JSON.parse(str))    // -> { Code: 200, Value: '8e286fdb-aad2-43c6-87b1-1c6c0d21808a', Route: '' }

In general, never work with buffers as if they were strings. Buffers are always encoded in some way, that is their fundamental, defining difference from strings. You must decode them before outputting their contents as text.

edited Feb 26, 2021 at 8:02

answered Feb 26, 2021 at 7:22

Tomalak

340k68 gold badges547 silver badges635 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Samo Over a year ago

Thanks! Didn't think of the utf16 possibility.

Tomalak Over a year ago

@Samo The double length is a dead giveaway. Also \0 is legal in JS strings, but it's not legal in JSON, that's why JSON.parse() fails. That it fails at position 1 tells you the byte order of the UTF-16, in this case the little-endian {·"·C·o·d·e·, i.e. utf16le. The less common big-endian variant ·{·"·C·o·d·e would fail parsing at position 0.

Collectives™ on Stack Overflow

Javascript JSON parse problem with decoded POST request data

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related