How to deal with massive arrays in javascript

Question

I'm trying figure out what the best strategy is for dealing with large datasets in JavaScript. I have files with four hours of data read at a 100ms interval.

This turns into a one dimensional array, however I can't get the thing to load all at once. Should I try splitting it into smaller arrays and is there an optimal array size?

Any advice would be appreciated, this is the first time I've had to deal with this type of problem. Some of the arrays are only 500kb but they still won't load as single module exports. Does AJAX work better for some reason? If so why?

Is this in the browser or in Nodejs? Also, please include your code. — Sven
– Sven, Commented Aug 5, 2017 at 20:17
you should definitely use batch size to get data over network. — Amitpal Rawat
– Amitpal Rawat, Commented Aug 5, 2017 at 20:21
The front end is a d3 line graph that pops data off the array and transitions the path update in a translation to make it look like the data is being generated in real time. In this sense, the length of the array can be essentially infinite, so I think working towards a streamable interface is the long term solution I need. I'm mainly a UI designer so this is a first for me. — MFave
– MFave, Commented Aug 6, 2017 at 1:21

guest271314 · Accepted Answer · 2017-08-05 22:19:27Z

1

You can use fetch() and .getReader() property of Response, which returns a ReadableStream, to read the data as a stream.

fetch("/path/to/server")
.then(response => response.body.getReader())
.then(reader => {
  const processStream = ({done, value}) => {
          if (done) {
            return reader.closed;
          }
          // do stuff with `value`:`Uint8Array` chunk of data
          return reader.read().then(processStream);
        }
  return reader.read().then(processStream);                
})
.then(() => "done reading stream")
.catch(err => console.error(err));

edited Aug 5, 2017 at 22:19

answered Aug 5, 2017 at 20:27

guest271314

1

Sign up to request clarification or add additional context in comments.

1 Comment

guest271314 Over a year ago

@MFave You can also pipe the ReadableStream to a WritableStream instance using .pipeTo(), see Receiving data via stdin and storing as a variable/array

Collectives™ on Stack Overflow

How to deal with massive arrays in javascript

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related