Shorten Integer-Array JSON Representation

Question

In my JS-application I'm uploading documents to a server. The documents are stored in Uint8Array's. That means documents are represented as Arrays consisting of Integers.

Before uploading the documents I JSON.stringify the documents. Here you have an example:

  var document = [0,5,5,7,2,234,1,4,2,4]
  JSON.stringify(document)
  => "[0,5,5,7,2,234,1,4,2,4]"

Then I send the JSON-representation to the Server.

My problem is that the JSON-representation has a much bigger file-size than the original Integer-Array. I guess that's because the Array is transformed to a JSON-String. I send much more data to the server then needed. How can I store the data more compressed in JSON?

I thought, the JSON-representation is maybe smaller, if I convert the Array first to Base64 and then to JSON.

What are your ideas? Thanks

What are the contents of the array? Would you have any value in the 0-255 range in about equal proportions, or would you have more ASCII-like values? Base64 will give you a better ratio than the above JSON array representation: one byte will result in an average of 1.33 bytes, while the array representation will result in 3.57 bytes. But if you have mostly ASCII-like data, you could go down to close to 1 byte... — jcaron
– jcaron, Commented Oct 9, 2016 at 17:00
There is this lz-string compression library for JS which presumably compresses the string to 25% of it's original size. Might be useful if you have large JSON data. — Redu
– Redu, Commented Oct 9, 2016 at 18:33

jcaron · Accepted Answer · 2016-10-09 17:04:38Z

1

The integer array JSON representation results in an average 3.57 ratio between input and output:

10 values are represented by 2 bytes (one digit, one comma)
90 values by 3 bytes (2 digits, one comma)
156 values by 4 bytes

On the other hand, base64 will result in an average 1.333... ratio (3 bytes are encoded as 4).

If you have mostly ASCII-like characters in your array (i.e. in the range 32-126), you would probably be better off just sending them as strings (with a few characters escaped), but not if you have random 8-bit data.

You could use some kind of base94 representation to get a better ratio over base64, but is it really worth the cost?

Also note that if you may also consider compression of the data.

answered Oct 9, 2016 at 17:04

jcaron

17.8k6 gold badges36 silver badges47 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Shorten Integer-Array JSON Representation

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related